Geek at Work Blog

An LLM endpoint looks like a URL. In production, it is a policy surface. The first version of an AI feature calls a provider SDK directly, with an API key in an environment variable. That works until a second model, provider, team, fallback requirement, or finance question arrives. Then every application reimplements the same routing, retries, quotas, caching, and logging — each one differently.

The missing component is not another model server. It is the gateway in front of the models. This article covers that layer: routing, cache correctness, failure handling, and token-level cost control. The GPU and serving layers below it are covered in the previous article on inference infrastructure; here we stay above the model endpoint.

Serving machine learning models is no longer a research problem — it is an infrastructure problem. Model inference workloads have their own shape: bursty traffic, expensive GPU-bound nodes, multi-gigabyte images and model weights, and autoscaling that reacts to tokens per second rather than CPU. In this guide we walk through what a production inference stack actually looks like — first as self-managed infrastructure on Kubernetes, then as managed cloud services from AWS, Google Cloud, Azure, and the GPU-native clouds — and compare the two paths on operations, scaling, latency, and cost from the perspective of the person who has to keep both running at 3 a.m.

DevOps, SecOps, and Infrastructure as Code (IaC) are foundational to building, securing, and scaling cloud-native environments. DevOps unifies software development and IT operations through practices and tools that automate and standardize software delivery. It promotes collaboration, continuous integration, and continuous deployment (CI/CD), enabling teams to deliver high-quality applications faster and more reliably. DevOps best practices include integrating automation at every stage, leveraging CI/CD pipelines, and using microservices and container orchestration for scalable solutions [1].

In the fast-paced world of software development, delivering value quickly and reliably is paramount. Effective management of deployment environments is a crucial component of achieving this. Gone are the days of “works on my machine” and manual, error-prone deployments. Modern DevOps principles demand automated, repeatable, and consistent deployments across a variety of environments. This article delves into the complexities of managing these environments, exploring strategies, tools, and best practices to ensure smooth, scalable, and predictable software delivery. We will cover not just the what but also the why behind each approach, helping you make informed decisions tailored to your specific needs.

In the ever-evolving landscape of modern application development and data management, organizations often face critical decisions about which database system best aligns with their needs—whether it’s scaling to meet growing demands, accommodating complex queries, or leveraging advanced features for innovation. For many companies that have historically relied on MySQL, a moment arrives when they recognize the limitations of their current infrastructure and seek a more robust solution. This is where PostgreSQL emerges as a compelling alternative.

In today’s rapidly evolving digital landscape, businesses are increasingly turning to cloud computing to enhance flexibility, scalability, and efficiency. AWS, as a leader in cloud infrastructure, offers unparalleled opportunities for innovation but also presents challenges in managing costs effectively. While the cloud provides immense power, it can quickly become a significant expense if not optimized properly. The key to success lies in striking the perfect balance between cost efficiency and performance—ensuring that you maximize your investment without compromising on the quality of service.

The proliferation of cloud infrastructure necessitates robust methods for detecting and managing drift—the divergence between intended configurations (defined via IaC) and actual resource states. This guide delves into causes, consequences, and mitigation strategies using tools like Terraform, AWS CloudFormation, Azure Resource Manager, and open-source utilities.

In the fast-paced world of software development, maintaining high code quality is crucial for building robust applications. Automated code quality checks ensure consistency, reduce errors, and enhance maintainability. Bitbucket Pipelines, Atlassian’s CI/CD tool, seamlessly integrates into your workflow, allowing you to automate these checks efficiently.

Shift-Left Security has emerged as a critical strategy in modern software development, especially within DevOps environments. By integrating security testing early into the development lifecycle, teams can identify and mitigate vulnerabilities more effectively. This article delves into the concept, benefits, practices, tools, challenges, and case studies related to Shift-Left Security.

# What is Shift-Left Security?

Shift-Left Security emphasizes incorporating security measures at every stage of the Software Development Life Cycle (SDLC). Unlike traditional approaches where security testing occurs post-development, this method shifts the focus earlier, ensuring potential threats are addressed during the design and development phases. This proactive approach aligns with DevOps principles, fostering collaboration between security, development, and operations teams to enhance overall application security.

Continuous Configuration Automation (CCA) is a critical practice in modern DevOps workflows, enabling organizations to manage and maintain consistent infrastructure configurations across development, testing, and production environments. Unlike traditional manual configuration methods prone to errors and inconsistencies, CCA leverages automation tools to ensure that systems are always in the desired state. This systematic approach reduces downtime, accelerates deployment cycles, and enhances security by enforcing compliance standards.