Beyond a Forward Pass · Part 4 of 5 · LLM inference · July 2026
Part 4: Prefix Caching & Routing
Part 4 of 5. Shared stems are shared compute — if you route and cache like a classic stateless service, you throw that win away.
Series · Beyond a Forward PassPart 4 of 5Prefix cacheJul 28, 2026
Shared prefixes are shared work
Two requests that start with the same tokens share the same early K/V. Recomputing that
prefix is pure waste. Multi-turn chat is the obvious case: turn 5 includes turns 1–4 as a
prefix. But production traffic is full of quieter sharing:
Identical system prompts across thousands of users
“Once upon a time…” style common stems (less common in enterprise, common in consumer)
“When scaling regular model serving, you replicate the model and route round-robin or to
the least busy replica… With LLMs, because there can be so much shared computation… you
typically want to route the query to a replica that has the relevant stuff cached —
prefix-aware routing.”
— Robert Nishihara
Automatic prefix caching (engine level)
Modern engines implement a global (or worker-local) store of KV blocks keyed by token
content hashes:
Tokenize request.
Walk the longest prefix whose blocks are already in the cache (often a radix tree).
Resume prefill only for the uncached suffix.
Insert new blocks into the cache with reference counts.
SGLang’s RadixAttention is a clear articulation of the radix-tree approach:
each edge is a token sequence chunk; nodes point at KV blocks. vLLM’s automatic prefix
caching and TensorRT-LLM / other stacks have analogous features under different names.
Win condition: high prefix hit rate turns expensive prefill into near-zero
incremental work. Multi-turn chat and agent loops benefit first.
From caching to routing
Cache locality dies if a load balancer ignores it. Classic strategies:
Strategy
Behavior
Prefix reuse
Round-robin
Even spread
Accidental only
Least connections / power of two choices
Load-aware
Still cache-blind
Sticky session by user/conversation ID
Same replica for a chat
Strong for multi-turn
Prefix-aware / cache-aware routing
Send to worker with longest matching prefix or hottest system prompt
Best for shared system prompts at scale
At cluster scale you often combine: consistent hash on conversation ID for multi-turn,
plus a control-plane map of which workers hold which popular prefixes (system prompt
IDs), plus fallback to least-loaded when cache miss or hot-spot overload.
A concrete production-shaped design: Ray Serve’s
PrefixCacheAffinityRouter tracks a character-level prefix tree of routed
content and sends each request to the replica with the longest common prefix, falling back
to power-of-two choices under weak match or imbalance. Reported effect: hit rate stays
usable as replica count grows, with large TTFT cuts (on the order of ~60% on a published
32B workload) versus random or plain power-of-two routing that erodes cache locality.
Note the approximation: the router sees request text, not the engine’s exact block-hash
events — good enough when prefixes are long and stable.
Engine-side, two indexing styles dominate: radix trees (SGLang
RadixAttention — LRU nodes with refcount zero; prioritize waiters by matched prefix length;
multi-call programs reported up to ~5× e2e) and block hashes (vLLM
automatic prefix caching hashes each full block from parent hash + tokens + optional
extras like LoRA/multimodal salts; only complete blocks are shared; reclaim via LRU free
queue).
Hot-spot danger
If 80% of traffic shares one system prompt and you route all of it to the one worker that
cached it first, you create a celebrity hot spot. Mature designs:
Replicate hot prefixes across N workers
Route among the N with load balancing
Track marginal gain of another replica of a prefix vs another cold worker
Cache hierarchy (what serious systems grow into)
L1: GPU KV blocks currently mapped for running sequences
L2: GPU-resident idle prefix blocks (evictable)
L3: CPU pinned memory / NVMe for cold prefixes (optional)
Cross-node: transfer KV from prefill pool to decode pool (Part 2) or between replicas
Each level trades hit latency vs capacity. The policy layer is where products differ more
than model weights do.
Application-level moves (you control these)
Stable system prompts: do not randomize whitespace or dates in the shared prefix.
Put dynamic content after static content: tools, policies, persona first; user-specific data last.
Session affinity: pass a conversation key to the gateway.
Prompt caching products: Anthropic/OpenAI-style prompt caching APIs are the hosted version of the same idea — bill savings track hit rates.
Gotcha: any change in the shared prefix invalidates the cache key. “Helpful
” vs “Helpful” (trailing space) is a miss. Version your system prompts deliberately.
How to know it’s working
Export prefix cache hit rate and saved prefill tokens.
Compare TTFT with cold vs warm system prompt.
Watch per-replica traffic entropy — collapse to one replica means hot-spot routing.
For multi-tenant, namespace cache keys by tenant if prompts can leak via side channels (security review, not just perf).
Next: when one replica is not one replica — tensor/pipeline sharding and Mixture of Experts
routing inside the model itself.