Beyond a Forward Pass · Part 2 of 5 · LLM inference · July 2026
Part 2: Prefill, Decode & Disaggregation
Part 2 of 5. Two compute stages with opposite bottlenecks — and why serious deployments sometimes run them on different GPU pools.
Series · Beyond a Forward PassPart 2 of 5Prefill / decodeJul 28, 2026
Two phases, not one forward pass
Every generation request has two regimes that look nothing like each other on a GPU:
Prefill (prompt processing)
Ingest all prompt tokens at once
Build initial KV cache for the prompt
Matrix–matrix heavy (GEMMs)
Usually compute-bound
Determines TTFT
Decode (token generation)
Emit one (or a few) new tokens per step
Append K/V for the new token
Matrix–vector-ish per sequence
Usually memory-bandwidth bound
Determines TPOT / streaming feel
“The prefill stage is often compute bound; the decode stage is often GPU memory bandwidth
bound… If you schedule them on the same GPUs, they can interrupt each other… A common
technique is prefill–decode disaggregation — run them on different pools of compute and
move the data in between.”
— Robert Nishihara
Roofline intuition (why the bottleneck flips)
GPUs deliver enormous FLOPs, but HBM bandwidth is finite. Arithmetic intensity =
FLOPs / bytes moved. Prefill of a long prompt has high intensity: many tokens share the
same weight loads and participate in large matmuls. Decode of batch size B at sequence
position t mostly reloads model weights (and streams KV) to produce B new tokens —
bytes moved stay huge, FLOPs per byte stay low.
Rule of thumb many practitioners use: if you are not saturating compute during prefill,
your kernels or batch shape are wrong; if you are not memory-bound during decode, you
are probably not looking at a real decode-heavy production mix.
Implication: optimizing only “the model kernel” is not enough. Scheduling
must treat prefill and decode as different resource customers that can starve each other.
Interference when co-located
On a single GPU pool doing both:
A long RAG prefill can delay every decoding stream’s next token (TPOT spikes).
A full decode batch can delay TTFT for new arrivals.
Latency becomes unpredictable — bad for SLOs, worse for multi-tenant SaaS.
Continuous batching (Part 1) improves utilization but does not by itself solve phase
interference. Chunked prefill helps; isolation helps more at scale.
Prefill–decode disaggregation
Disaggregation means: run prefill on one set of GPUs (or nodes), decode
on another, and transfer the KV cache (and control metadata) between them.
What moves across the boundary
The request identity and sampling state
The KV tensors for the prefilled prompt (dominant payload)
Sometimes multi-modal encoder outputs if the architecture separates them
Why it can win
Hardware fit: prefill likes high FLOPs; decode likes high HBM bandwidth and capacity for many concurrent KVs.
Scaling curves: prefill scales with prompt tokens; decode scales with concurrent users × generation length.
Predictability: decode pools are not hit by bursty 100k-token prefills.
Why it can lose
KV transfer latency and network cost (NVLink / RDMA / PCIe topology matters).
More moving parts: two autoscalers, two failure domains, consistency of tokenizer/model revision.
Short-prompt, short-output chat may not pay back the transfer tax.
Systems research and production stacks treat this as a first-class topology choice:
DistServe co-optimizes per-phase parallelism and GPU counts for TTFT vs
TPOT goodput, places stages by interconnect so KV transfer stays small relative
to latency when bandwidth is high (example they give: OPT-66B, 512-token KV ≈ 1.13 GB;
~90 Gbps at 10 rps is feasible on InfiniBand or intra-node NVLink). Reported up to
~7.4× more requests or ~12.6× tighter SLOs at >90% attainment vs coupled baselines.
Splitwise provisions separate prompt/token pools (and optional mixed
pools), may put decode on heterogeneous or power-capped GPUs, overlaps layer-wise KV
transfer on InfiniBand (residual ~5–8 ms in their reports), and chooses
workload-dependent P:D ratios — large cluster wins on the order of ~1.4× throughput at
lower cost, or higher under fixed power budgets.
Failure modes to design for: bandwidth-sensitive placement, bursty KV
pressure on decode memory, FCFS convoy effects where one long prefill blocks short ones,
and fault coupling when many prefill workers map to one decode instance.
Co-located scheduling tricks (when you don’t disaggregate)
Technique
Idea
Tradeoff
Chunked prefill
Split long prefills into token chunks; interleave with decode
Smoother TPOT; slightly higher total prefill time
Priority / QoS classes
Interactive vs batch queues
Operational complexity; fairness debates
Separate model replicas by workload
Some replicas only long-context RAG, some only chat
Capacity fragmentation
Length prediction / bucketing
Group similar prefill sizes
Prediction error hurts
A minimal cost model
Let \(L_{\text{in}}\) be prompt tokens, \(L_{\text{out}}\) generated tokens, B concurrent
sequences. Roughly:
Prefill work scales roughly as
\[ O(L_{\text{in}}^{2}) \text{ attention} + O(L_{\text{in}}) \text{ MLP} \]
(exact constants depend on architecture; FlashAttention changes constants, not the story).
Decode work scales roughly as
\[ O\!\left(L_{\text{out}} \cdot \big(L_{\text{in}} + L_{\text{out}}/2\big)\right) \]
attention over growing context, but in practice wall-clock is dominated by weight bandwidth × steps.
For chat with short answers, decode steps dominate user-perceived latency. For long-document
summarization or agent traces with huge system prompts, prefill dominates cost and TTFT.
Your product mix should drive whether you invest in disaggregation or in faster prefill kernels.
What to measure in your stack
Split latency dashboards into TTFT and TPOT — never only e2e.
Correlate TTFT spikes with prefill token volume and batch composition.
If TPOT degrades when RAG traffic rises, you have phase interference; try chunked prefill before a full redesign.
If you already run multi-node, prototype disaggregation only when KV transfer << prefill time on your fabric.
Next: the structure that prefill builds and decode reads every step — the KV cache — and
the OS-inspired trick that made large continuous batches practical.