AI Inference Infrastructure from a DevOps Perspective: Self-Managed Kubernetes vs Managed Cloud

Mon, 26 Oct 2026 00:00:00 Z

Serving machine learning models is no longer a research problem — it is an infrastructure problem. Model inference workloads have their own shape: bursty traffic, expensive GPU-bound nodes, multi-gigabyte images and model weights, and autoscaling that reacts to tokens per second rather than CPU. In this guide we walk through what a production inference stack actually looks like — first as self-managed infrastructure on Kubernetes, then as managed cloud services from AWS, Google Cloud, Azure, and the GPU-native clouds — and compare the two paths on operations, scaling, latency, and cost from the perspective of the person who has to keep both running at 3 a.m.

# Why Inference Infrastructure Is Different

A model inference service looks like a normal web API but behaves very differently under the hood:

Hardware constraints. GPUs are expensive, scarce, and can’t be live-migrated. A pod that pins a GPU shapes how you schedule, drain, upgrade, and preempt. Driver/CUDA/kernel version alignment is a real compatibility matrix you must manage.
Big, cold artifacts. Images with CUDA runtimes and downloaded model weights easily reach 5–50 GB. Cold starts are measured in minutes, not seconds, which breaks naive scale-to-zero designs.
Custom scaling signals. HPA on CPU is nearly useless: vLLM saturates GPU KV-cache utilization way before CPU looks busy. You need scaling on model-specific metrics (queue depth, requests per replica, time to first token).
Different SLOs. For LLM serving you don’t just care about p99 latency of a response; you care about time to first token (TTFT) and time per output token (TPOT), because streaming changes how users perceive latency.
Cost concentration. A single A100/H100 node can cost several thousand dollars a month. Inference is where most of your ML cloud bill lives, and idle-GPU waste is the default state unless you engineer against it.

These constraints drive the real architectural decision: where does the GPU fleet and the serving logic live — in your cluster, or in a provider’s?

# The Anatomy of an Inference Request

Before choosing infrastructure, it helps to understand what a single request actually does, because every infra decision downstream is reacting to this behavior.

For a typical LLM request on vLLM:

Request admission. The request arrives over HTTP/gRPC and enters a waiting queue.
Weight residency check. The model weights must already be in GPU memory — loading 16 GB from disk takes tens of seconds to minutes, so this must happen at startup, per replica.
Batching. The runtime waits a few milliseconds to coalesce concurrent requests into a batch. Modern serving engines do continuous batching: new requests join a running batch at each decode step rather than waiting for the whole batch to finish, keeping GPU utilization flat under load.
Prefill and decode. The prompt is processed (prefill, compute-bound), then tokens are generated one at a time (decode, memory-bandwidth-bound). The KV cache — the stored attention state of the context — grows with prompt + output length and is the real memory consumer, not the weights.
Streaming response. Tokens stream out over SSE/WebSocket as they’re generated.

Why this matters for DevOps:

KV cache means memory, not compute, is often the bottleneck → you tune max-model-len, quantization, and PagedAttention-style cache management. OOM at the KV cache is the most common production failure of a self-managed LLM deployment.
Continuous batching means throughput per GPU is highly dependent on concurrency → autoscaling to more replicas can decrease cost efficiency if it breaks batching. You often want fewer, busier replicas rather than many idle ones.
Weights at startup means every scale-up event pays a multi-minute loading cost → your autoscaler needs long cooldowns and pre-warmed node pools to handle bursts.

Keep this lifecycle in mind; it explains the Kubernetes specifics below.

# Option 1: Self-Managed Inference on Kubernetes

The typical self-managed stack looks like this:

Layer	Common choice
Serving runtime	vLLM (LLMs), Triton with TensorRT/TensorRT-LLM, SGLang, KServe, BentoML
GPU enablement	NVIDIA GPU Operator (drivers, device plugin, DCGM monitoring, MIG)
Cluster/fleet	EKS / GKE / AKS with GPU node groups, or bare metal
Request routing	Ingress or service mesh, load balancing aware of in-flight requests
Autoscaling	HPA + KEDA on custom metrics; Ray Autoscaler for LLM serving
Storage/weights	Model artifacts in object storage, PVC or init-container download, local NVMe cache
Observability	DCGM exporter, Prometheus, Grafana, LLM-specific dashboards

## Serving Runtimes: Know What You’re Deploying

Choosing a runtime is the first real decision, and it’s a DevOps-relevant one because each has different operational characteristics:

vLLM — the current default for serving open-weight LLMs via an OpenAI-compatible API. Best-in-class continuous batching with PagedAttention. Operationally, you manage GPU memory limits carefully (the gpu_memory_utilization flag) and version-bump aggressively: the project moves fast and performance improves release to release.
Triton Inference Server — NVIDIA’s general-purpose server, targeting traditional ML (XGBoost, ONNX, vision models) and TensorRT-LLM for NVIDIA-tuned LLM serving. Best when you have a mixed zoo of model types behind one server, or you want NVIDIA’s engines. More config-heavy (model repository structure, config.pbtxt), but predictable.
SGLang — newer engine with advanced scheduling (RadixAttention, prefix caching), often outperforms vLLM on multi-turn conversation traffic where prompts share prefixes. Watch this space if chat workloads dominate.
KServe — not a runtime but a serving framework on Kubernetes: gives you InferenceService CRDs, canary traffic splits among model revisions, scale-from-zero via queue-proxy, and pluggable backends (it can run vLLM or Triton underneath).

The practical pattern: vLLM (or SGLang) for OpenAI-compatible LLM endpoints, Triton for everything else, KServe if you want model-level reconciliation and canarying as Kubernetes-native objects rather than rolling your own.

## GPU Operator and Node Management

Running GPUs in Kubernetes without the operator means hand-installing drivers on node images, which breaks under upgrades. The NVIDIA GPU Operator packages the whole stack — drivers, container toolkit, device plugin, DCGM exporter for metrics, and MIG manager — as controllable Kubernetes objects. Node pool definition typically looks like:

apiVersion: eks.amazonaws.com/v1alpha2
kind: NodeGroup # EKS managed node group; GKE/AKS equivalents apply
metadata:
 name: gpu-inference
spec:
 instanceType: g6e.12xlarge # L40S-class cards
 labels:
 node-type: gpu-inference
 taints:
 - key: nvidia.com/gpu
 value: "dedicated"
 effect: NoSchedule
 scaling:
 minSize: 2
 maxSize: 10

Things to get right here:

Taint GPU nodes. Non-GPU pods will happily schedule onto your most expensive nodes unless you taint. Every real cluster has horror stories of a misconfigured logging DaemonSet costing thousands per month.
Set up Cluster Autoscaler / Karpenter for GPU instance types. GPU capacity is scarce in many regions. Configure least-waste priority (e.g., prefer 8-GPU nodes only for 8-GPU jobs), consolidation carefully — consolidations that reschedule GPU pods cause visible service blips from weight reloading.
MIG (Multi-Instance GPU) for small models. On A100/H100 you can partition a GPU into slices (e.g., 7×10GB slices). If your models fit in 10 GB, MIG turns one node into seven serving replicas and slashes cost. The GPU Operator’s MIG manager configures this; the device plugin then exposes nvidia.com/mig-1g.10gb as a schedulable resource.

## Weights: The Storage Side of Inference

Model loading is a first-class ops concern. The naive path —Pod starts, downloads 15 GB from Hugging Face on the fly — gives you multi-minute readiness, wasted bandwidth (every replica downloads), and flaky readiness on network hiccups.

Better patterns, in increasing order of investment:

Local NVMe cache on the node. An init container (or hostPath-backed downloader) checks for the model in a node-local directory and downloads only if absent. After the first scale-up on a node, subsequent pods start in seconds.
PVC with ReadOnlyMany. A pre-populated PVC (or a Filestore/EFS/GCFileSystem mount) shared across replicas avoids redundant downloads. Watch network filesystem throughput — for big models, it can become a bottleneck slower than downloading from object storage.
Pre-baked serving images or snapshots. Bake weights into the image (big images, ~15–40 GB pull) or AMI/disk snapshot (fast). This is the “warm node pool” approach: autoscaler launches nodes from a snapshot with weights and even model server config already in place.

The vLLM deployment example from earlier, with the cache volume pattern:

apiVersion: apps/v1
kind: Deployment
metadata:
 name: llm-inference
spec:
 replicas: 2
 selector:
 matchLabels: { app: llm-inference }
 template:
 metadata:
 labels: { app: llm-inference }
 spec:
 nodeSelector:
 node-type: gpu-inference
 tolerations:
 - key: nvidia.com/gpu
 operator: Exists
 effect: NoSchedule
 initContainers:
 - name: fetch-model
 image: amazon/aws-cli:latest
 command: ["sh", "-c", "aws s3 sync s3://our-models/llama-8b /models/llama-8b --no-progress"]
 volumeMounts:
 - name: model-cache
 mountPath: /models
 containers:
 - name: vllm
 image: vllm/vllm-openai:v0.6.4
 args:
 - --model
 - /models/llama-8b
 - --max-model-len
 - "8192"
 - --gpu-memory-utilization
 - "0.92"
 resources:
 limits:
 nvidia.com/gpu: "1"
 memory: 64Gi
 requests:
 cpu: "8"
 readinessProbe:
 httpGet: { path: /health, port: 8000 }
 initialDelaySeconds: 10
 periodSeconds: 5
 failureThreshold: 60 # allow minutes for first load
 volumeMounts:
 - name: model-cache
 mountPath: /models
 volumes:
 - name: model-cache
 emptyDir:
 sizeLimit: 40Gi

Notice the long failureThreshold on the readiness probe: cold starts are legitimately minutes. If you set a normal 30-second threshold, every scale-up triggers a crash loop of your new replica and the autoscaler fights itself.

## Autoscaling on the Right Signal

Standard HPA on CPU is the wrong tool, for two reasons: GPU inference barely uses host CPU, and scale-up latency (node launch + image pull + weight load) is minutes, so you need to scale early and hold, not react and thrash.

The standard approach is KEDA with Prometheus metrics from the serving runtime. vLLM exposes:

vllm:num_requests_waiting — queue depth
vllm:num_requests_running — in-flight per replica
vllm:time_to_first_token_seconds / vllm:e2e_request_latency_seconds — latency quantiles
vllm:gpu_cache_usage_perc — KV cache saturation

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
 name: llm-scaledobject
spec:
 scaleTargetRef:
 name: llm-inference
 minReplicaCount: 2 # warm floor: never below your p-cold-start tolerance
 maxReplicaCount: 12
 cooldownPeriod: 600 # long enough to avoid thrash — readiness is slow
 triggers:
 - type: prometheus
 metadata:
 address: http://prometheus.monitoring:9090
 query: sum(vllm:num_requests_waiting)
 threshold: "10" # target: max ~10 queued requests per replica

Key engineering points:

Warm floor over scale-to-zero. With minute-scale cold starts, scale-to-zero only works for workloads where users tolerate minutes of latency (background jobs, internal batch tools). Otherwise keep a warm floor and treat scale-down as a cost lever, not a default.
Long cooldowns. Because readiness is slow, aggressive scale-down after a spike makes you pay the full cold-start cost repeatedly. Ten minutes of cooldown is more normal than datacenter default.
Node-level pre-scaling. Pod autoscaling doesn’t help if the autoscaler needs to launch a GPU node to satisfy it. Pre-provision GPU capacity or over-provision GPU nodes with fake low-priority “placeholder” pods (the classic cluster-autoscaler trick) if burst latency is contractual.
Batching-aware reasoning. A single replica can often handle 20–50 concurrent LLM requests with good TPOT. If you scale on requests-per-replica with a target like 30 or 50, you keep the batching advantage instead of shattering load across underfilled replicas.

## Load Balancing and Traffic Shape

Standard round-robin across pods is not ideal for LLM serving: replicas have different queue lengths already. Consider a routing layer aware of in-flight requests (e.g., a gateway or a serving framework’s own router that balances by num_requests_running), and sticky routing by conversation/session for multi-turn traffic so prefix caches stay warm. If you’re doing streaming (SSE/WebSocket), confirm your ingress layer has long/idle timeouts tuned — common sources of mysterious 502s mid-stream.

## Spot GPUs and Interruption Engineering

Spot/preemptible GPU capacity is frequently 50–80% cheaper and, because inference is stateless (unlike training checkpoints), somewhat friendlier to interruptions. Engineering it well means:

Disruption-tolerant deployments: reasonable PodDisruptionBudgets, including spot-node taints and tolerating instance termination notices (via 2-minute termination handlers draining pods).
Fast weight loading (the storage patterns above) so a rescheduled replica is serving reasonably quickly after a handoff.
Multiple capacity shapes: don’t demand only p5.48xlarge; deploy with fallback smaller instance types and run several smaller replicas.
Some teams run spot for the bulk of capacity and on-demand as a warm internal minimum, using cluster autoscaler “prefer-spot” strategies.

## Observability for Inference

The metrics stack is standard Prometheus/Grafana, but the content is GPU- and LLM-specific:

DCGM exporter — GPU utilization, memory usage, temperature, power, ECC errors. High temperature and throttling directly affect latency and hardware life.
Serving engine metrics — queue depths (waiting requests), running requests, TTFT and end-to-end latency quantiles, KV cache usage.
Gateway metrics — request rate, streaming connection counts and durations (long-lived streaming requires rounding and thinking about connection limits and timeouts differently than normal HTTP).
Cost instrumentation — tokens served per hour per GPU, dollars per million tokens. This is your north-star SLO for inference economics: track it per model version, because quantization and version changes move it substantially.

Alerting on GPU utilization is misleading: a well-batched GPU should run at ~100% memory bandwidth/utilization under load. Alert instead on queue depth and latency SLOs, treating GPU utilization as a capacity-planning graph.

# Option 2: Managed Cloud Inference Services

Every major provider has a managed inference product — you bring a model artifact, config, and get an endpoint:

Provider	Service	Notes
AWS	SageMaker Endpoints (incl. async & serverless), Bedrock, EC2 with Deep Learning AMIs	SageMaker handles serving infra; Bedrock offers hosted foundation models; asynchronous endpoints handle large payloads
Google Cloud	Vertex AI (Endpoints, Model Garden), GKE Autopilot with GPUs	Tight integration with BigQuery/Colab and Google’s managed Nvidia/TPU fleet
Azure	Azure ML online endpoints, Azure OpenAI	Azure OpenAI is often the shortest path for enterprise LLM use; Azure ML for custom models
GPU-native clouds	CoreWeave, RunPod, Lambda, Modal, Anyscale	Kubernetes-native (CoreWeave) or simpler per-GPU rental; strong price-performance and quick cold-start UX

The mental model: you give up governing the whole cluster below the model. The platform provides the serving runtime (or a container you supply), GPU instance selection, autoscaling targets, network isolation (VPC endpoints), encryption at rest/in transit, audit trails, and per-invocation pricing modes.

A typical managed setup is declarative, roughly conceptually:

# Conceptually — each provider uses its own DSL/SDK
endpoint:
 name: llama-8b-prod
 model: s3://our-models/llama-8b-finetuned.tar.gz
 container: 763104351884.dkr.ecr.us-east-1.amazonaws.com/djl-inference:vllm
 instance_type: ml.g6e.4xlarge
 autoscaling:
 target_metric: ModelInvocationsPerInstance
 target_value: 50
 min: 1
 max: 8
 data_capture: enabled # for drift monitoring / replay

What you gain

Time to first production call measured in hours: no GPU operator, no driver pain, no cluster provisioning, no MIG experiments.
Built-in autoscaling on provided inference metrics, handled by the provider’s control plane, with sensible scale-in defaults and warm pool semantics engineered upstream.
Compliance and network posture mostly handled: private VPC endpoints, KMS-encrypted model artifacts, IAM auth, audit logs — what would take weeks of K8s policy engineering.
Pricing discipline tools: serverless variants with per-invocation billing and scale-to-zero handled for you (SageMaker serverless endpoints, Vertex AI predictive); async endpoints for long-running jobs with S3 payloads.
Traffic management and monitoring included: canary weights on endpoint versions, dashboards, and drift/capture hooks.

What you pay for

Per-instance markup — typically 1.3–2× raw EC2-equivalent GPU-hour cost, in exchange for the managed control plane.
Limited runtime knobs. Batching behavior, continuous batching parameters, KV cache policy, and GPU partitioning are whatever the platform exposes; if the platform’s defaults aren’t optimal for your traffic shape, you absorb it, or you use custom containers and hope your tuning isn’t broken by an endpoint update.
Platform-shaped architectures. Your request path routes through the provider’s frontend, so p50 latency can add tens of milliseconds; some platforms add queue-before-GPU even for warm endpoints.
Lock-in to control plane conventions. IAM roles, autoscaling metric names, SDK/DSL, endpoint state model — nontrivial to port to another provider later (though most lock-in is at this config layer, not the model itself).
Cost concentration at sustained load. Running one managed endpoint 24/7 is close to always paying the premium. Managed serverless endpoints scale smoothly but pass through latency; if your traffic is sustained rather than bursty, the raw-GPU time-based pricing of EC2 (plus your own ops) often wins on cost per million tokens.

# Comparing the Two Paths

Dimension	Self-managed K8s	Managed cloud
Time to first production call	Days to weeks	Hours
Cost at sustained 24/7 load	Lower (spot + reserved + fractioning + MIG)	Higher per-GPU-hour
Cost at spiky/bursty traffic	Harder (idle GPUs, warm floors)	Easier (serverless variants)
Cold-start behavior	You engineer it (snapshots, caches, warm pools)	Platform-managed warm pools
Ops burden	High — GPU operator, drivers, serving tuning, spot interrupts	Low — endpoint ops only
Model-level optimization	Full (quantization, batching, CUDA graphs, speculative decoding)	Limited to what the platform exposes
Scaling signal	Custom metrics, high tuning effort	Platform-provided targets
Vendor lock-in	Low — cluster and serving code are portable	Significant — endpoint, IAM, SDK layer
Latency overhead	Direct pod networking	Provider frontend hop
Observability	DIY (Prometheus, DCGM, Grafana)	Included dashboards
Token-economics visibility	Complete — you measure everything	Provider dashboards only

# A Pragmatic Decision Framework

Most teams don’t need a binary answer — they need a routing rule. Patterns that work in practice:

Start managed, migrate what deserves it. Deploy on SageMaker/Vertex/Azure ML endpoints first — validate the model has product fit before investing in its infrastructure. When a model’s sustained traffic, special GPU configuration, or per-token economics justify it, lift the serving code to your cluster.
Keep self-managed for tuning-sensitive workloads. Continuous batching behavior, quantization (AWQ, GPTQ, FP8/FP4), speculative decoding, prefix caching, and KV-cache offloads matter enormously for cost per million tokens. If cost per token is your north-star metric, you want a runtime you control: your own vLLM (or SGLang) on GPUs you own is usually the frontier.
Use serverless managed endpoints for hard-to-forecast, low-QPS workloads. Internal tools, demos, admin features, geographically rare traffic — scale-to-zero in the cloud avoids keeping a GPU warm for three requests a day, and the latency is acceptable when users expect internal tools to be slower.
Match the traffic shape to the pricing model. Sustained 24/7 load → self-managed with spot/reserved and MIG fractioning. Bursty, spiky, unpredictable → managed serverless. Long-running batch inference jobs → async endpoints or spot-backed batch jobs on your own clusters.
Don’t fight the GPU supply shortage alone. If you can’t get capacity in-region from your primary cloud, the GPU-native clouds (CoreWeave is literally K8s under the hood; RunPod/Lambda prioritize availability) are often a supplement — treat them in your multi-cloud strategy as capacity, with the same serving images and IaC.
Beware common pitfalls either way: under-provisioning GPU memory for KV cache, forgetting to request/raise GPU service quotas in the target region, model weight download times dominating autoscaling behavior, GPU utilization metrics firing “capacity” alarms when the GPU is simply efficiently batched, and long-lived streaming connections leaking through ingress misconfigurations.

# Conclusion

Inference infrastructure is infrastructure engineering, more than ML engineering. Self-managed Kubernetes gives you control, best unit economics at scale, and deep optimization headroom (quantization, batching, fractioning) — at the price of being responsible for some of the hardest workloads to keep running: GPUs, huge deterministic artifacts, custom autoscaling, interruption engineering. Managed cloud services trade cost per GPU-hour and flexibility for operational simplicity, compliance posture, and speed to production.

Most mature ML platform teams end up with both: managed endpoints for early-stage and spiky models, and a tuned self-managed GPU fleet on Kubernetes for the workloads where cost per token and tail latency are competitive differentiators. The DevOps answer isn’t “Kubernetes or managed” — it’s knowing what your traffic looks like at the token level and using both paths where the tradeoffs serve you.

# Further Reading

vLLM documentation — serving runtime, continuous batching, and Prometheus metrics
NVIDIA GPU Operator — drivers, device plugin, DCGM, and MIG management on Kubernetes
KEDA documentation — event-driven autoscaling on custom metrics
KServe — model serving framework with traffic splitting on Kubernetes
SGLang — RadixAttention-based serving engine for prefix-heavy workloads
SageMaker Inference — endpoint types, autoscaling, and serverless inference
Vertex AI Endpoints and Azure ML online endpoints — managed inference reference

Geek at Work Blog