Three engines, three memory models

What an inference engine actually is

Every request is the same loop:

1. Tokenize the prompt.
2. Prefill. The whole prompt is known, so the transformer can run it in parallel and write a key/value tensor per layer into a cache.
3. Decode. Sample one token, append it, attend against the cache, repeat. Decode is memory-bound. A 70B model in FP16 is about 140 GB of weights. One token means reading those weights once. Batching is how you amortize the read.
4. Sample. Logits to a token ID. This is a tiny kernel that becomes the bottleneck if you do it naively on the CPU at batch 256.

The engine is the software that owns the KV cache, the batch, the sampler, the kernels, and the HTTP socket. Hugging Face Transformers does the math. These three projects do the serving.

They disagree about memory.

Memory: the actual fork in the road

vLLM: PagedAttention, then a hash index

vLLM's 2023 SOSP paper (arXiv:2309.06180) is the reason the rest of the field talks about "pages." Naive serving allocates a contiguous KV buffer of max_seq_len per request. Short requests waste the tail. The allocator fragments. You cannot pack 40 requests into a GPU that has room for 40 average lengths, because the allocator reserved 40 maxima.

PagedAttention copies the OS virtual-memory trick. KV is carved into fixed-size blocks (vLLM default: 16 tokens). Each sequence holds a block table: logical token positions to physical blocks in a global pool. Blocks are allocated as the sequence grows and freed when it finishes. Reference counts let two sequences share a block.

The original CUDA kernel walked that table inside attention. vLLM's own design docs now mark that kernel writeup as historical. Current vLLM (V1 is the only engine; V0 was removed in the 0.28 line) feeds paged KV into FlashAttention, FlashInfer, TRTLLM-GEN, FlashMLA, or Triton, depending on GPU and model. The idea survived. The original kernel is no longer the point.

On top of the pages, vLLM V1 added automatic prefix caching as a hash index, not a tree. Each full block is keyed by:

hash(parent_hash, block_tokens, extra)

extra covers LoRA IDs, image hashes, and an optional cache_salt so two tenants cannot timing-attack each other's prefixes. Default hash is SHA-256. Only full blocks are cached. Partial tails are not. Eviction is LRU over a free queue wired as a doubly-linked list on the block objects themselves, so moving a block to the tail is O(1) and there is no Python deque wrapper in the hot path.

That is the vLLM bet: treat GPU memory like an OS treats RAM, then hash the pages so shared prefixes are a lookup, not a tree walk.

SGLang: RadixAttention, then three cache tiers

SGLang also pages KV. The differentiator is the index.

RadixAttention keeps a radix tree keyed by token IDs over the whole pool. A path from root to a node is a prefix. Two requests that share a system prompt share the nodes. The scheduler matches waiting requests against the tree before it admits them. Longest-prefix-match (--schedule-policy lpm) is a first-class policy, not a blog post. Eviction on the tree is LRU/LFU/SLRU.

Default --page-size is 1 (token-level). vLLM's default is 16. Token-level pages make prefix sharing finer and the metadata fatter. You can raise the page size. The default tells you what they optimized first: reuse, not allocator simplicity.

In August 2026 they collapsed the hybrid-model mess into Unified Radix Cache. Full attention, sliding-window attention, and Mamba/GDN recurrent state do not reuse the same amount of a prefix. Old SGLang grew a cache class per combination. The new tree is one topology with composable components (FULL, MAMBA, copy-on-write on shared GDN checkpoints). That is the patch you need once Qwen3.8-style hybrids (GQA layers interleaved with gated delta-net layers) are the models people actually serve.

HiCache extends the tree off the GPU. GPU is L1, host DRAM is L2, and L3 is someone else's store: Mooncake, DeepSeek 3FS, NIXL, AIBrix. L1/L2 are per instance. L3 is cluster-wide. Prefill/decode disaggregation hands KV over GPU-direct RDMA instead of recomputing it. This is the production-scale answer to "the prefix does not fit in HBM."

vLLM has the same problem and a growing connector zoo (LMCache, Mooncake, NIXL, simple offload). SGLang's answer is in the tree. vLLM's answer is in the block hash plus pluggable transfer. Both work. They feel different when you read the code.

llama.cpp: a unified KV pool, slots, and a paged future

llama.cpp does not ship PagedAttention as the default. --ctx-size is still the total KV token budget across sequences, not "context per user" (discussion #4130). Continuous batching is default-on (-cb). Attention isolation is a KQ_mask. As long as there are free cells, llama_decode accepts another llama_batch.

The layout is no longer only "one big shared buffer." --kv-unified is a flag (on by default when slot count is auto). Maintainers have been moving toward independent per-sequence caches; both shapes exist depending on flags and build. --kv-unified-per-slot (merged June 2026, bartowski/ngxson) caps per-slot context so you can say "4 slots × 4096" instead of doing the arithmetic in your head.

llama-server still papers over occupancy with slots (-np). Idle-slot VRAM behavior has been a recurring footgun; LLAMA_KV_KEEP_ONLY_ACTIVE=1 exists because people expected idle slots to evict. The unified-when-on layout is elegant at n=1 and n=4. It over-allocates at n=100 the same way pre-vLLM servers did: it reserves capacity as if peak occupancy were typical occupancy.

Paged KV is not vapor. PR #22569 (opt-in --kv-paged, default off) reports the number you would expect: on an A10G, unified OOMs at 26 concurrent Llama-3 8B sequences; paged admits 247 and 2.5× aggregate throughput, with a small loss at the concurrency unified could already handle. Phase 2 (CoW / prefix cache) is not done. Treat paged llama.cpp as incoming infrastructure, not the 2026 default.

KV dtype is a llama.cpp superpower the Python engines copy more slowly. -ctk / -ctv independently set K and V to f16, bf16, q8_0, q4_0, q5_0, iq4_nl, and friends. Quantizing the cache is often a bigger VRAM win than one more weight bit. Combined with --flash-attn on (now the auto default) and --kv-offload, this is how a 12 GB card pretends to be a 24 GB card.

Prefix reuse in llama.cpp today is slot-shaped: save/restore (--slot-save-path), checkpoints (--ctx-checkpoints), and prompt caching. It is not a radix tree. A homelab with one user and a 100k-token system prompt uses slot save/restore and gets the win. A 200-tenant agent farm does not.

Scheduling and continuous batching

vLLM V1 splits processes. Default vllm serve -tp 4 is 1 API server + 1 engine core + 4 GPU workers = 6 processes, talking ZMQ. The engine core runs a busy loop: schedule, dispatch, sample. Chunked prefill is normal. New requests join an in-flight batch at token boundaries. Preemption is FCFS with the later request sacrificed first. Data-parallel MoE adds a coordinator process. This is an operating-system-shaped server. It wants CPU headroom, shared memory, and a container.

SGLang spent, in Lianmin Zheng's words, more time on CPU overhead than on GPU kernels. The v0.4 line shipped a "zero-overhead batch scheduler": overlap CPU schedule with GPU execute so the GPU never waits on Python. Cache-aware admission (LPM) is the unique lever. --schedule-policy is fcfs | lpm | dfs-weight | lof | priority | routing-key. Priority preemption is real. Prefill delayer exists to stop DP ranks from issuing tiny prefills that desynchronize the batch. This is a scheduler written by people who stared at traces of agent loops, not chatbot QPS.

llama.cpp continuous-batches inside one process. -b is the logical batch, -ub the physical micro-batch. The HTTP server (tools/server, cpp-httplib + nlohmann/json) is one binary. No ZMQ, no Ray, no tokenizer subprocess. --cont-batching is on. The limit is the unified pool and the slot count, not Python. At 1–8 users this is the right shape. At 100 users you are fighting the allocator PR #22569 exists to replace.

NVIDIA Dynamo is not an engine. The 28 August 2026 NVIDIA post is the industry admitting the layer above the engine: route to a warm prefix, split prefill from decode, autoscale each pool. It wraps vLLM, SGLang, and TensorRT-LLM. llama.cpp is not in that diagram. That is information.

Sampling stacks

This is where llama.cpp still wins on taste, and where the Python engines win on batch.

llama.cpp sampler chain (default order): penalties, DRY, top-n-sigma, top-k, typical-p, top-p, min-p, XTC, temperature. Mirostat, dynamic temperature, logit bias, GBNF grammars, JSON Schema, --backend-sampling (experimental, on-device). The chain is a list you can reorder with --samplers. Homelab people care about this because a 27B GGUF with DRY + min-p is a different model than the same GGUF with temperature 0.8 and top-p 0.95. Constrained decoding is GBNF, which is old, local, and good enough for "emit this JSON."

vLLM sampling lives in vllm/v1/sample/ as fused CUDA ops (_apply_penalties, _sample). Per-request sampling params in a continuous batch is the hard part; a naive PyTorch sampler becomes the CPU bottleneck. Structured output is xgrammar or guidance. Logits processors are a documented extension point. Beam search still exists. The design goal is: sampling must not stall a 256-wide decode.

SGLang made structured output a product. Compressed finite-state machines update logit masks in microseconds (LMSYS 2024 blog: up to 3× faster JSON decoding). Jump-forward decoding skips tokens the grammar has already determined. xgrammar is in the runtime, not an afterthought. If your API is "return this schema, every time," this is the engine that treated that as a kernel problem.

None of the three will save you from a bad temperature. llama.cpp will let you experiment. vLLM/SGLang will let you do it at batch.

Speculative decoding

Speculative decoding is "a cheap model proposes k tokens, the expensive model verifies them in one forward." Accept 3, skip 3 decode steps.

llama.cpp can mix a draft implementation with a draftless one on the same server; draftless wins if both fire. --spec-default turns on ngram-mod. This is not the "limited" column I would have written in 2024. The remaining gap is not algorithm names. It is that llama.cpp still verifies those drafts against a slot/unified cache, not a paged radix tree.

Two warnings from this month, because speculative decoding is where silent correctness bugs hide:

• llama.cpp issue #27852 (28 August): ngram-cache begin() was a no-op, so a slot reused the previous request's n-grams. Acceptance 86% → 11%. Faster to turn spec off. This is the slot-state class of bug. Python engines have analogous ones; they get named CVE-shaped GitHub issues instead of LocalLLaMA threads.
• r/LocalLLaMA (27 August): Blackwell recipes that actually move the needle on vLLM right now are NVFP4 KV + MTP or DFlash2 + NVFP4, not "enable speculative decoding" as a boolean.

SGLang's own GLM-5.3 day-0 post (28 August) quoted 537.6 tok/s/user NVFP4 and 413 tok/s/user FP8 at batch=1, TP8, 8× B300, on multi-turn agentic traffic. That is a first-party number on flagship silicon with the kernel work they shipped for GLM-5.2. Treat it as "the ceiling they will show you," not "what your 4090 does."

Quantization

Three different religions.

llama.cpp / GGUF. Integer quant is the product. Q2_K through Q8_0, IQ quants, Unsloth dynamic UD-Q*_K_XL, MXFP4 and NVFP4 landing in ggml. Weights stay quantized through the GEMM. The convert script is Python (convert_hf_to_gguf.py); runtime is C. You can mix: Q6_K weights, q4_0 K-cache, q4_0 V-cache. This is how low-VRAM boxes exist.

vLLM. A zoo, because production checkpoints are a zoo: FP8, MXFP8/MXFP4, NVFP4, INT8, INT4, GPTQ, AWQ, GGUF, compressed-tensors, ModelOpt, TorchAO. Kernels from CUTLASS, TRTLLM-GEN, CuTeDSL. GGUF is supported so you can bring a llama.cpp file to a vLLM server; it is not the happy path. The happy path on Blackwell is NVFP4 weights + NVFP4 KV.

SGLang. Same neighborhood: AWQ, GPTQ, FP8, MXFP8, NVFP4, GGUF, ModelOpt, bitsandbytes, plus mlx_q4 / mlx_q8 for the MLX path. --kv-cache-dtype includes fp8_e4m3, fp8_e5m2, nvfp4, fp4_mx_block16. --quantize-and-serve exists for ModelOpt prototyping. Day-0 posts from LMSYS this year almost always ship an NVFP4 checkpoint next to the recipe.

If your weights are GGUF, start with llama.cpp. If your weights are a Hugging Face safetensors FP8/NVFP4 drop, start with vLLM or SGLang. Converting back and forth to "try the other engine" is how people lose a weekend and then quote a benchmark that measured the converter.

Codebases: languages, shape, hackability

GitHub linguist bytes, 29 August 2026:

llama.cpp is a CMake project. ggml/ is the tensor library. src/ is llama. tools/server/ is the HTTP surface. gguf-py/ is the one Python you need to convert. Backends live under ggml/src/ggml-*.cpp and friends: CUDA, Metal, HIP, Vulkan, SYCL, CPU SIMD (AVX/NEON/RVV), CANN, MUSA, WebGPU, OpenCL, RPC. You can read a kernel without a conda env. You cannot read the whole tree in an afternoon anymore (10k commits, multimodal, a Svelte web UI, TypeScript), but the inference path is still "C structs and a graph." Hackable. The failure mode is backend #ifdef soup.

vLLM is vllm/ (Python) + csrc/ (CUDA/C++) + rust/ + a multiprocessing executor. V1 code that matters: vllm/v1/engine/core.py, vllm/v1/core/sched/, vllm/v1/worker/gpu/. Adding a model is a nn.Module with a keyword-only VllmConfig constructor so sharding and quant happen during init (a 405B model cannot be materialized then sliced). The issue tracker is a firehose (7k open). The contributor base is the largest. The failure mode is "I needed one flag and I got a distributed systems project."

SGLang is python/sglang/srt/ (runtime), sgl-kernel/, sgl-model-gateway/, rust/ (experimental radix tree core), python/sglang/srt/mem_cache/ (the layer cake: allocation → hybrid_cache → allocator → pool → pool_host → storage). The frontend language (the original "Structured Generation Language") still exists; most production users hit the OpenAI HTTP server and never write an SG program. The runtime is the product. Hackable if you live in Python/CUDA. The failure mode is velocity: 5k open issues on 33k stars is a higher ratio than vLLM. Day-0 model support is a feature and a treadmill.

Maintainability, as I would bet my own time:

• Change a sampler: llama.cpp.
• Add a Hugging Face architecture that landed this morning: vLLM, then SGLang, then llama.cpp (GGUF convert + kernel work).
• Add a prefix-cache eviction heuristic: SGLang (it is a tree policy).
• Ship a binary to a friend's Mac without Python: llama.cpp (llama.app installer, May 2026).
• Survive a 200-GPU MoE deploy: vLLM or SGLang, with a team. Not llama.cpp.

An arXiv preprint this month (2608.13884) scraped 33,228 PRs from vLLM and SGLang to talk about agentic coding's effect on those repos. That paper existing is the tell: both Python engines are now too big for any one human to hold in their head, and they are being extended by agents. llama.cpp's maintainer list is still a page of names you can learn.

Kernels and hardware

NVIDIA CUDA

All three run. vLLM and SGLang are CUDA-first serving stacks: CUDA graphs (vLLM default is full-and-piecewise as of 0.28), torch.compile, FlashAttention 2/3/4, FlashInfer, TRTLLM-GEN. vLLM's auto-priority is FlashInfer then FA then Triton on Blackwell (SM 10.x), and FA then FlashInfer then Triton on Ampere/Hopper. FlashInfer is not in the pre-built pip wheel; it is a separate install or a Docker image. SGLang's default attention backend is FlashInfer on non-Hopper, FA3 on Hopper, with a matrix of which backend supports FP8 KV, FP4 KV, speculative topk>1, sliding window, multimodal.

llama.cpp CUDA is custom ggml kernels, not FlashInfer. Flash attention is a flag (-fa), not the entire architecture. It is fast enough for local. It is not the stack SemiAnalysis is PRing against InferenceX.

NVIDIA Dynamo (this month's messaging) sits around vLLM/SGLang/TRT-LLM. If your 2026 plan is "one replica," ignore Dynamo. If your 2027 plan is "prefill pool + decode pool + prefix-aware router," you are picking vLLM or SGLang now so you can plug them in later.

AMD ROCm

vLLM: HIP graphs, AITER, growing but historically the support matrix lagged CUDA (SemiAnalysis AgentX called this out). Community forks still appear when INT8 on MI100-class cards is the job (r/LocalLLaMA, 26 August, 4× MI100).

SGLang: treated AMD as a first-class day-0 target for DeepSeek and GLM. MI300X / 325X / 355X show up in the GLM-5.3 cookbook next to B300. Microsoft Azure serving DeepSeek-R1 on AMD with SGLang is the citation people use. If your cluster is Instinct, SGLang is the engine whose README sounds like it has on-call.

llama.cpp: HIP backend. It works. It is not where ROCm 10's claimed 3.3× inference lift (AMD, this week) will land first. Vulkan is the escape hatch when HIP is being HIP.

Apple Silicon

llama.cpp Metal is a first-class citizen. ARM NEON, Accelerate, Metal enabled by default on macOS. Unified memory is the whole point of a Studio. This is why my llama-swap config is llama-server -ngl 99 --flash-attn on and not a Docker image.

vLLM's in-tree macOS build is experimental CPU. GPU on Apple Silicon is the out-of-tree vLLM-Metal plugin (MLX/Metal). Not the happy path.

SGLang has a documented MLX/Metal hardware page and mlx_q4/mlx_q8 quant options. Single-Mac, --tp-size 1. Newer. Fine for experiments. Not what I would point a family member at. Neither Python engine documents Metal tensor/pipeline/expert parallel.

If the box is a Mac, llama.cpp. If you insist on Python, MLX. vLLM/SGLang on Metal is a science project.

CPU and everything else

llama.cpp CPU SIMD is the reason GGUF exists on a laptop with no dGPU. AVX, AVX2, AVX512, AMX, NEON, RVV. --n-cpu-moe / --n-cpu-ffn keep expert or FFN weights on host and run the hot path on GPU. Hybrid is a flag, not a paper.

vLLM and SGLang both have CPU backends (Intel Xeon shows up in SGLang's README). They are "we should not crash" backends, not "replace llama.cpp on a Threadripper" backends.

llama.cpp's other backends are the long tail nobody else will do: Vulkan, SYCL (Intel GPUs), CANN (Ascend), MUSA, WebGPU, OpenCL Adreno, RPC (split a model across a Mac and a 3060, which is a real YouTube genre this month), zDNN, Hexagon-in-progress. This is ggml's actual ideology, restated by Gerganov at 100k stars: the stack has to run on every device or it is vendor-locked.

Multi-GPU

llama.cpp --split-mode layer is "put layer N on GPU 0 and layer N+1 on GPU 1." It is pipeline-ish. It is not Megatron tensor parallel. --split-mode row splits weight rows. --tensor-split 3,1 is a ratio, not a process group. For two consumer cards in one box, this is enough. For 8× H100 MoE, it is not.

vLLM shards during module init so a 405B never materializes 810 GB on one rank. That constructor constraint is the whole distributed-memory story.

Concurrency: 1 user vs 100

n = 1, local, you are typing. llama.cpp. Lowest TTFT tax from Python, CUDA context, and NCCL. Metal or CUDA, GGUF, flash-attn on, cache quantized if VRAM is tight. Speculative n-gram if you like living dangerously (reset your slots). llama-swap in front so the 31B is not resident while you are in a meeting.

n = 10, shared homelab, mixed models. Still llama.cpp, still llama-swap. One big GGUF at a time. TTL unloads it. A vLLM container in llama-swap is valid for a model that only exists as FP8 safetensors, with the cost of a 30–90 s cold start and a Python stack that wants --shm-size. Do not run three vLLM replicas on a 24 GB card. The card holds one.

n = 100, one model, OpenAI-compatible API, NVIDIA. vLLM or SGLang. Continuous batching + paged KV is the whole game. Pick vLLM if you want the default, the broadest model list, TPU/Intel plugins, and the larger hiring pool. Pick SGLang if the workload is multi-turn agents with a fat shared prefix, structured JSON, or a model whose day-0 kernels landed in sgl-kernel this week (Kimi K3, GLM-5.3, DeepSeek V4). The throughput gap between them on a vanilla Llama-70B is often single-digit percent. The gap on a prefix-heavy agent with Radix LPM vs hash-prefix-on-full-blocks is not.

n = 100, you thought llama.cpp slots would do it. Unified KV will OOM or livelock in the way PR #22569 measured (26 sequences vs 247). -np 100 with a huge -c just allocates a 100-wide contiguous tax. Wait for paged, or put a real engine on the GPU node and keep llama.cpp on the laptops.

OpenAI-compatible APIs, all three:

• vLLM: vllm serve, /v1/chat/completions, Anthropic Messages, gRPC, tool parsers, reasoning parsers.
• SGLang: python -m sglang.launch_server (default port 30000, default host 127.0.0.1), OpenAI surface, gRPC, --enable-http2.
• llama.cpp: llama serve / llama-server. OpenAI chat completions, Anthropic Messages, embeddings, rerank, infill, /completion, GBNF, tools. The API is good. The scheduler behind it is the slot machine.

llama-swap is not an engine. It is a Go proxy that starts engines. That is the correct architecture for a homelab that wants Gemma at lunch and a 35B MoE at night. It is the wrong architecture for 100 concurrent users of one model: you wanted a batcher, you built a process babysitter.

What the last 30 days actually look like

I ran the social-listening pass (Reddit, HN, YouTube, GitHub, Digg, plus native X search) over 30 July – 29 August 2026. X auth is not wired into that engine on this machine; the X lines below are from the X search tools, not from cookie scraping.

Popularity, bluntly. llama.cpp still has the GitHub crown and the YouTube "just install this" genre (SchizoDev, 83k views: "everything uses llama.cpp at the back end"). vLLM has the production crown and the homelab-with-a-real-GPU crown (r/LocalLLaMA is full of vLLM recipes this month: 2×3090 + DFlash2 at 218 tok/s, 280 upvotes; Blackwell NVFP4 KV; Intel Arc Pro B70 XPU). SGLang has the frontier-model crown and a quieter LocalLLaMA footprint. 33k stars vs 90k is not "unpopular"; it is "the people who need it already have it, and they are on Slack, not Reddit."

Adoption signals that are real.

• Amazon documented production vLLM for Rufus, including multi-node Trainium across tens of thousands of chips for Prime Day traffic. That is a named, dated, first-party writeup, which is rarer than a README logo wall.
• Inferact (vLLM core) $150M seed at ~$800M, a16z + Lightspeed. First vLLM Conference happened this week; AMD co-hosted the rooftop. Simon Mo in the PyTorch Con promo: "easiest to use and most efficient."
• RadixArk (SGLang core) $100M seed at ~$400M, Accel. Ying Sheng left xAI to run it. LMSYS still hosts the project. Day-0 posts this month: Kimi K3, GLM-5.3, MiniMax H3 diffusion. The README lists xAI, NVIDIA, Cursor, the hyperscalers, and "400,000 GPUs." Treat the GPU count as a self-report, not a census.
• ggml / llama.cpp team joined Hugging Face in February 2026 (Gerganov's own post). A 1,346-upvote r/LocalLLaMA thread this week asked whether an NVIDIA acquisition of Hugging Face would swallow llama.cpp's copyright. That is a community fear, not a closed deal. The underlying fact is: the most portable inference engine is now employed by a model hub, and the model hub is in play. llama.cpp's README still does not list named hyperscale deployments the way Amazon/vLLM does. That is also information.

Engineering pulse.

• vLLM 0.28.0 (26 August): V0 engine gone. Full-and-piecewise CUDA graphs default.
• SGLang 0.5.18 (22 August): 710 PRs in a release notes roundup; AMD AITER fused paths for DeepSeek V4.
• llama.cpp: tagged v0.2.0 (first non-b* tag in a while) plus b10683 the day I pulled. llama.app (12 August HN: 364 points) is the UX bid: one installer, one llama binary, OpenAI server, existing GGUF cache reused.

The quotable split. r/LocalLLM, 29 August, Qwen3.8-Flash-Next on an RTX PRO 6000: "llama.cpp isn't ready for agentic work, vLLM is ~4x faster at long context." That matches the architecture. Agentic work is prefix reuse + batch + tool JSON. llama.cpp can do tools. It cannot yet page 200 parallel prefixes. vLLM can. SGLang was built for that sentence.

On X, NVIDIA spent the week explaining Dynamo as the layer around the engines. That is the production tell: the argument is no longer "which engine" for a single box. It is "which engine does your orchestrator speak." llama.cpp is not on that list. llama.cpp is how the rest of us still have a box.

Decision guide

Two anti-picks:

• Do not put vLLM on a Mac Studio and then blog about how vLLM is slow. You measured PyTorch+CUDA-absence.
• Do not put llama.cpp -np 64 on an H100 and then blog about how llama.cpp cannot serve. You measured the unified allocator.

What I actually run

Mac Studio, llama-swap, llama-server from Homebrew, -ngl 99 --flash-attn on --host 127.0.0.1. Models are GGUF: Gemma 4 31B-class dense, Gemma 4 MoE, a 35B-A3B thinking MoE, a vision pair with mmproj. Context 32k or 128k depending on the weights. --parallel 2 on the coding models. llama-swap TTL 1800 s. The OpenAI SDK talks to localhost:8080 and does not know or care which GGUF is up.

That is the correct engine for this machine. If I put a 4× GPU Linux box on the tailnet for a family API, I would run vLLM for the always-on instruct model and keep llama-swap on the Studio for the weird GGUFs. If that API became an agent with a 20k-token tool preamble shared across sessions, I would try SGLang on the same box before I bought another GPU.

The engines are not competitors. They are answers to "who owns the KV cache." vLLM pages it. SGLang trees it. llama.cpp shares a pool and is teaching that pool to page. Pick the data structure that matches the occupancy you actually have.

Numbers: GitHub API and linguist byte counts pulled 29 August 2026. Engine versions: vLLM 0.28.0, SGLang 0.5.18, llama.cpp b10683. Community window: 30 July – 29 August 2026. Speculative-decoding rows for llama.cpp checked against docs/speculative.md after a verifier pass; I had understated EAGLE3/DFlash/DSpark there on first draft. I did not re-benchmark these engines for this post; tok/s figures are attributed to their authors. No inspected primary source publishes a same-hardware, same-model, August 2026 three-way bench of 1 user vs ~100 concurrent requests.