Part 1: Continuous Batching

Why this series exists

A vision model takes an image, runs one forward pass, and says “cat.” An LLM generates one token at a time, with variable-length inputs and outputs, multi-turn context, and memory that grows as the conversation grows. That is why specialized engines — vLLM, SGLang, TensorRT-LLM — exist.

This series takes each of the five differences he walked through and goes to foundation depth: what breaks without the technique, how it is implemented, and what you should measure when you run or buy inference.

The core problem: variable-length work

Autoregressive decoding is a loop. Given prompt tokens \(x_1,\ldots,x_n\), the model produces \(y_1\), then \(y_2\) conditioned on \(x\) and \(y_1\), and so on until EOS or max_tokens. You cannot know \(T_{\text{out}}\) at request start. That single fact destroys classic batching assumptions.

Batching still matters: GPU kernels amortize weight loads across sequences. Loading 14 GB of BF16 weights once to process 32 tokens is far better than loading them 32 times. But the batch composition must change while generation is in flight.

Static (request-level) batching — and why it wastes the GPU

Static batching collects up to B requests, runs them until every sequence in the batch hits EOS or max length, then starts the next batch. While short requests sit idle waiting for the long tail of the batch, their KV slots and GPU slots are still occupied. White space after early EOS is pure underutilization.

How bad is it? It depends on variance of output lengths. Chat and agent workloads have heavy-tailed generation lengths. Anyscale’s 2023 continuous-batching benchmarks (OPT-13B, A100-40GB) showed naive static batching collapsing toward ~80 tok/s as variance rose, while continuous batching + memory optimizations stayed an order of magnitude higher.

Continuous batching = iteration-level scheduling

The fix was formalized in Orca (OSDI ’22): schedule at the iteration (token step), not the request. At every decode step:

1. Run one forward pass for all active sequences in the current batch.
2. Any sequence that just emitted EOS is evicted; its resources free.
3. If the waiting queue has work and memory allows, admit new sequences.
4. Repeat.

This is also called in-flight batching or dynamic batching (careful: “dynamic” sometimes means request-level batch size selection, which is weaker). Industry engines implement the Orca idea with refinements:

How it is implemented in practice

The scheduler loop

Production schedulers maintain (at least) a waiting queue and a running set. A simplified control loop looks like:

while True:
    # 1) Free finished sequences; reclaim KV blocks
    for seq in running:
        if seq.finished: free_kv(seq); running.remove(seq)

    # 2) Admit new work if memory + policy allow
    while waiting and can_allocate(waiting[0]):
        seq = waiting.pop(0)
        allocate_kv(seq)
        if needs_prefill(seq):
            prefill_queue.append(seq)
        else:
            running.append(seq)

    # 3) Build this iteration's batch
    batch = select_batch(prefill_queue, running)  # tokens, not just sequences

    # 4) One model forward (possibly chunked prefill + decode mixed carefully)
    outputs = model.forward(batch)
    append_tokens(outputs)

Why prefill complicates the batch

Prefill of a long prompt is a large, compute-heavy GEMM-heavy step. Decode of many sequences is memory-bandwidth heavy (see Part 2). Naively mixing a huge prefill with latency-sensitive decode can spike TTFT for everyone else. Engines therefore use policies such as:

• Chunked prefill — break a long prefill into chunks interleaved with decode steps.
• waiting_served_ratio (TGI-style) — bound how aggressively prefills jump the line.
• Prefill/decode disaggregation — put the stages on different GPU pools (Part 2).

Memory is the real admission control

Continuous batching only helps if you can fit more concurrent sequences. That is why PagedAttention (Part 3) is not a side topic: without non-contiguous, just-in-time KV allocation, iteration-level admission still fails on fragmentation and over-reservation. Anyscale’s results: continuous batching alone ~8× vs naive static; continuous batching + vLLM memory management up to ~23× on high-variance workloads (OPT-13B, A100-40GB — the peak multiple is not “batching alone”).

Admission control that avoids mid-generation deadlock

Orca’s original design reserves KV capacity up to each request’s max_tokens on admit so a sequence cannot get stuck halfway through generation with no free KV left. Finished requests free those slots. At matched latency they reported up to 36.9× throughput vs FasterTransformer on GPT-3 175B-class serving — a different experiment than Anyscale’s 23×, same family of ideas.

TensorRT-LLM’s name for the same loop is in-flight batching (IFB). Useful knobs in that stack: max_batch_size, max_num_tokens (packed token budget per step; default often 8192), optional chunked prefill, and metrics that split waiting vs scheduled/active requests. Thinking in “tokens per step,” not only “sequences per batch,” is how packed continuous batchers stay safe under mixed prefill+decode.

Metrics you actually care about

What to do with this knowledge

• If you are still doing static batch inference for online chat, switch to an engine with iteration-level scheduling.
• For offline batch jobs (eval, synthetic data), static or large microbatches can still be fine — length variance is often lower and latency is not user-facing.
• Load-test with realistic length distributions (exponential/heavy-tailed outputs), not fixed 128-token generations.
• Watch GPU memory, not just utilization % — continuous batching’s ceiling is KV capacity.

Next: the two compute stages inside every request — prefill vs decode — and why modern clusters sometimes put them on different machines.

Sources & further reading

• Orca: A Distributed Serving System for Transformer-Based Generative Models (OSDI ’22) — iteration-level scheduling, max_tokens KV reservation
• Anyscale — Continuous batching enables 23× throughput
• TensorRT-LLM — Paged Attention, IFB, and request scheduling
• vLLM blog — Easy, Fast, and Cheap LLM Serving with PagedAttention
• NVIDIA — Mastering LLM Techniques: Inference Optimization
• Primary framing: Robert Nishihara & Linda Vivah, LLM inference vs regular inference (NYC walk / LinkedIn + X reposts)