The Inference Gateway: Routing, Caching, and Cost Control for Production LLM Traffic

An LLM endpoint looks like a URL. In production, it is a policy surface. The first version of an AI feature calls a provider SDK directly, with an API key in an environment variable. That works until a second model, provider, team, fallback requirement, or finance question arrives. Then every application reimplements the same routing, retries, quotas, caching, and logging — each one differently.

The missing component is not another model server. It is the gateway in front of the models. This article covers that layer: routing, cache correctness, failure handling, and token-level cost control. The GPU and serving layers below it are covered in the previous article on inference infrastructure; here we stay above the model endpoint.

# Two Gateways, One Traffic Path

The industry uses two names for adjacent things, and conflating them causes most of the confusion:

• An AI gateway is the entry point for LLM traffic. It owns identity, keys, virtual model names, provider routing, quotas, caching, guardrails, and cost attribution. It fronts managed model APIs and, often, self-hosted models.
• An infrastructure inference gateway is a Kubernetes data-plane component. It routes requests to pods in an InferencePool using inference-aware signals: queue depth, KV-cache utilization, prefix-cache state, and active LoRA adapters. It does not manage provider keys or budgets.

They solve different halves of the same problem:

The split is logical, not necessarily physical. Agent Router (formerly Envoy AI Gateway) spans both: it routes across providers and supports InferencePool backends. LiteLLM, Kong AI Gateway, and similar products occupy the AI gateway layer; the Kubernetes Gateway API Inference Extension defines the inference layer.

The AI gateway fronts provider APIs and the inference gateway for your own pools.

If you have one model, one team, and no cost pressure, skip the gateway; it adds a hop and a failure domain. Once you need centralized credentials, per-team quotas, provider fallback, or a defensible answer to “who spent this money?”, it earns its place.

# Routing That Understands Inference

The first job of the gateway is to stop applications from naming models. A client asks for chat-default, not provider-x/model-y-2026-09-14; the gateway maps the alias to a concrete deployment without a client release. Version the alias (for example, chat-default-2026-10) so rollback is a routing change, not an incident.

Routing policy should use attributes the application actually knows:

• Tenant and tier. Free users can get a smaller model; paid users the primary one.
• Workload class. Interactive chat is latency-sensitive; batch summarization is not.
• Context size. A 200k-token request should not land on a 32k-window model or on a replica with a full KV cache.
• Region and data residency. An EU tenant should not silently route to a US deployment.
• Capabilities. Tool calling, JSON mode, vision, and structured output are not interchangeable across models.
• Cost ceiling. A team budget can exclude frontier models.

Inside a self-hosted pool, round-robin is blind to request cost: ten long-context requests can load a replica more than forty short ones. Least-connections is better but still treats requests as equal. The Kubernetes inference extension adds an Endpoint Picker (EPP) that reads model-server metrics and chooses an endpoint per request. vLLM exposes most of these signals:

• vllm:num_requests_running — current load per replica
• vllm:kv_cache_usage_perc — how close the replica is to memory pressure
• vllm:prefix_cache_queries and vllm:prefix_cache_hits — whether the replica has a warm prefix cache
• active LoRA adapters — whether the replica can serve the requested fine-tune

Prefix-cache affinity is the highest-leverage example. A multi-turn conversation resends the same system prompt and history every turn. If the next request lands on a replica that already holds that prefix in KV cache, prefill is largely skipped and time to first token drops. Strict session affinity has a failure mode, though: one popular conversation can pin itself to one replica while others idle. The useful policy is affinity with a load guard — prefer the warm replica until queue or KV-cache pressure crosses a threshold, then move.

An InferencePool can declare what happens when the endpoint picker is unavailable. With failureMode: FailOpen, the gateway picks an endpoint itself rather than failing the request. That favors availability, but requests may land on a cold or overloaded replica. Fail-closed protects latency SLOs and hurts uptime; make the trade-off explicit.

# Caching Without Lying to Users

Caching saves the most money and causes the most subtle incidents. Three caches get mixed together in vendor decks:

Exact response caching is the safest. The key must include the model version, every sampling parameter that affects output, tool definitions, response format, and tenant. Miss one and you will serve an answer generated under different conditions. A TTL and an invalidation path for model or prompt releases are mandatory. Hit rates are low for chat and high for support macros, classification, and test suites; cache the workloads where it works.

Provider prompt caching is not a response cache. The gateway passes the request through, and the provider reuses prefill state for a stable prefix. Savings show up as cache-read tokens and lower time to first token. The rule: keep the prefix byte-stable. Do not inject a timestamp, request ID, or per-user greeting into the system prompt if you want a hit. Keep sessions on the same provider and region when the cache is regional. Account for cache-read and cache-write tokens separately; they are priced differently from ordinary input.

Semantic caching is the dangerous one. It embeds the prompt, finds the nearest cached prompt above a threshold, and returns the stored response. For single-shot FAQ it can work. For multi-turn or agentic traffic it is a correctness bug: each turn is the previous request plus a small delta, so consecutive turns are nearly identical in embedding space. LiteLLM’s documentation warns that semantic caching “will replay stale responses” on multi-turn or agentic traffic and recommends exact-match caching instead. The embedding call also runs inline, so a slow embedding endpoint becomes request latency.

Tenant isolation is the other trap. Kong’s AI Semantic Cache documentation warns that the policy does not use request headers to separate cached responses and recommends configuring it per AI consumer rather than globally. LiteLLM’s semantic_cache_scope: end_user setting exists for the same reason: by default, every end user behind one virtual key shares a semantic bucket. A cached answer for one customer can be served to another who asks something similar. A global semantic cache is a data leak with a hit-rate dashboard.

The rule: cache hit rate is a cost metric, not a quality metric. A 40% hit rate with wrong answers is worse than no cache. Gate semantic caching behind an allowlist of intents, scope it by tenant and end user, keep TTLs short, support a bypass header, and evaluate cached answers like generated ones. When in doubt, use exact-match caching and leave semantic caching off.

# Failure Handling and Quotas

A gateway makes reliability policy explicit. SDK defaults are optimistic: retry a few times, wait, hope the provider recovers. At scale, those defaults compose badly.

Fallback is a product decision. A chain usually goes: same model in another region or provider, then a smaller model from the same family, then a static degraded response. If a paid user silently gets the small model, that is a product change. Return the actual model in response metadata (for example, an x-model-used header) and decide per feature whether a downgrade is acceptable. Test fallbacks for tool-call shape, JSON validity, context window, and refusal behavior.

Retries need one owner. If the SDK makes three attempts and the gateway makes three, one request can become nine upstream calls. Pick one layer — usually the gateway — and set the other to zero. Retry only transient failures: 429, 5xx, connection errors, and timeouts before any output. Never retry 400, 401, 403, context-length errors, or content-policy rejections. Add jitter and a retry budget; a retry storm during a provider incident turns a partial outage into a full one.

Timeouts should be split by phase. Connect, time to first token, idle between streamed chunks, and total are different controls. A single 30-second timeout kills legitimate long generations; a missing idle timeout holds connections open after a client disappears. Set total timeout above the upstream p99 but below client patience, and make sure ingress and load balancers do not time out streams earlier than the gateway.

Rate limits should use tokens and concurrency, not just requests. A 200k-token prompt is not equivalent to a 200-token prompt, but both count as one request per minute. Reserve estimated input plus max_tokens before dispatch, reconcile against actual usage, and release the difference. A 32k-token prompt with max_tokens: 8000 reserves 40k tokens; at a 200k-token-per-minute team quota, only five such requests can be in flight. Pair this with concurrency limits so a burst cannot exhaust provider quotas or gateway memory, and return 429 with Retry-After instead of queueing without bound.

Gateway quotas do not replace provider quotas. Accepting traffic you cannot deliver moves the failure downstream; keep both aligned and leave retry headroom.

# Observability and Unit Economics

The gateway sees every request, including quota rejections, so emit one structured event per request:

• request ID, tenant, application, and virtual model
• resolved provider, model, and region
• input, output, cached-read, cached-write, and reasoning tokens
• time to first token, time per output token, and total latency
• status, error type, retry count, and fallback flag
• cache status and estimated cost

Use OpenTelemetry GenAI conventions. Spans carry gen_ai.provider.name, gen_ai.request.model, gen_ai.response.model, gen_ai.operation.name, and usage attributes such as gen_ai.usage.input_tokens, gen_ai.usage.output_tokens, and gen_ai.usage.cache_read.input_tokens (a subset of input; do not double count). Server-side latency uses gen_ai.server.time_to_first_token and gen_ai.server.time_per_output_token; token counters live under gen_ai.client.inference.usage.*. They are still Development, so pin a version and map them to your schema.

Do not log prompts and responses by default. The GenAI content attributes (gen_ai.input.messages, gen_ai.output.messages) are opt-in and can contain personal data. If you must capture content, sample it, redact it, set a short retention, and make access auditable.

Cost accounting starts with a versioned price table: input, output, cache read, cache write, and reasoning prices per provider and model. Calculate cost from actual provider usage, not token estimates. Reconcile monthly against the invoice; a drifting price table is worse than none. Show cache hits as avoided upstream cost, not zero cost — the gateway still did work, and semantic-cache embeddings are not free.

Track p95 TTFT, p95 TPOT, error rate by provider, fallback rate, 429 rate, queue depth, tokens per request, and cost per successful task. “Cost per million tokens” is an input metric; “cost per successful task” is the business metric, because it captures retries, fallbacks, and wasted generations. Break both down by tenant and model; alert on SLO burn and fallback spikes, not raw spend.

# A Reference Architecture

A workable production topology separates policy from serving:

                         +----------------------+
  apps / agents  ----->  |  AI gateway          |
                         |  auth, aliases,      |
                         |  quotas, cache,      |
                         |  fallback, telemetry |
                         +----------+-----------+
                                    |
                 +------------------+------------------+
                 |                                     |
                 v                                     v
       managed model APIs                  inference gateway
       (provider A, provider B)            (Gateway API + EPP)
                                                       |
                                              +--------+--------+
                                              |                 |
                                              v                 v
                                         vLLM pool A       vLLM pool B
                                         base model        LoRA adapters

A minimal LiteLLM configuration (model IDs are placeholders):

model_list:
  - model_name: chat-default
    litellm_params:
      model: openai/<primary-model>
      api_key: os.environ/PRIMARY_API_KEY
  - model_name: chat-default
    litellm_params:
      model: anthropic/<secondary-model>
      api_key: os.environ/SECONDARY_API_KEY
  - model_name: chat-default
    litellm_params:
      model: openai/llama-3.3-70b-instruct   # served by vLLM
      api_base: http://inference-gateway.inference.svc/v1
      api_key: os.environ/INTERNAL_GATEWAY_KEY
  - model_name: chat-fallback
    litellm_params:
      model: openai/<smaller-model>
      api_key: os.environ/PRIMARY_API_KEY

router_settings:
  routing_strategy: simple-shuffle
  num_retries: 2
  timeout: 30
  allowed_fails: 3
  cooldown_time: 30

litellm_settings:
  fallbacks: [{"chat-default": ["chat-fallback"]}]
  cache: true
  cache_params:
    type: redis

On the Kubernetes side, the inference gateway routes to an InferencePool rather than a Service. An abridged example:

apiVersion: inference.networking.k8s.io/v1
kind: InferencePool
metadata:
  name: llama-70b
spec:
  selector:
    matchLabels:
      app: vllm-llama-70b
  targetPorts:
    - number: 8000
  endpointPickerRef:
    name: llama-70b-epp
    port:
      number: 9002
    failureMode: FailOpen
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: llama-route
spec:
  parentRefs:
    - group: gateway.networking.k8s.io
      kind: Gateway
      name: inference-gateway
  rules:
    - matches:
        - path:
            type: PathPrefix
            value: /v1/chat/completions
      backendRefs:
        - group: inference.networking.k8s.io
          kind: InferencePool
          name: llama-70b

The AI gateway treats the inference gateway as another OpenAI-compatible backend. You can change serving runtimes, GPU pools, or model versions without changing application code, and add a managed provider without changing Kubernetes manifests.

Treat the gateway like production infrastructure: at least two replicas across zones, a pod disruption budget, no synchronous calls to external services in the policy chain, Redis local to the region, and no response buffering so streaming actually streams. Test at twice your expected peak; a gateway that is cheap at idle can become the first-launch bottleneck. Decide whether a break-glass path direct to a provider exists. It bypasses quotas and audit, so document it as an emergency procedure, not a habit.

# Anti-Patterns and a Checklist

Predictable failures:

• Gateway as a business-logic layer. Prompt construction, RAG, and agent orchestration belong in the application; the gateway routes, enforces, and observes.
• Retry multiplication. SDK retries plus gateway retries plus provider retries turn one failure into a load spike. One owner, one budget.
• Silent downgrade. Falling back to a smaller model without telling the caller changes quality. Surface the actual model.
• Semantic cache by default. It is a correctness risk on multi-turn and agentic traffic, and a tenant-isolation risk when scoped globally.
• Prompt logging everywhere. It creates a PII archive with no owner. Log metadata; opt in to content.
• Routing on stale metrics. If the scrape interval outlasts the burst, the endpoint picker steers with yesterday’s traffic.
• Unmonitored single point of failure. The gateway sees all traffic, so its own availability, latency, and error budget need dashboards too.

A short checklist before calling the layer production-ready:

1. Apps use virtual model names; aliases are versioned and reversible.
2. Per-team quotas exist in tokens and concurrency, not just requests per minute.
3. The fallback chain is tested for tool calling, JSON, context length, and refusal behavior.
4. Exactly one layer owns retries, with jitter and a budget.
5. Timeouts are split into connect, first-token, idle, and total.
6. Caches are scoped by tenant, model version, and parameters; semantic caching is opt-in.
7. Every request emits tokens, TTFT, TPOT, status, fallback flag, and cost.
8. Content is not logged by default; the gateway has its own SLO and capacity test.

# Conclusion

The model is the product; the gateway is how you keep it running. It turns a model name into policy, a provider incident into a fallback, a cache hit into either a saving or a correctness bug, and a token into a number finance can reconcile.

You do not need every feature on day one. Start with one virtual model, one fallback, one token-based quota, and one dashboard for cost per successful task. Add inference-aware routing when you run your own pools, and caching only where you can prove it is safe. It is the difference between a demo and a feature that survives its first real month.

# Further Reading

• Agent Router (formerly Envoy AI Gateway) — provider routing and InferencePool support
• Gateway API Inference Extension — InferencePool and Endpoint Picker
• LiteLLM Router and caching — aliases, fallbacks, cooldowns, caches
• Kong AI Gateway — policies, semantic caching, consumer scoping
• OpenTelemetry GenAI conventions — spans, token metrics, content capture
• vLLM metrics — queue, KV cache, prefix cache, TTFT, TPOT
• AI inference infrastructure from a DevOps perspective — the GPU and serving layer below the gateway