Cooking Qwen 3.8 27B: what Mia and 0xBakeer actually toggle

1. What a cooked recipe actually is

The 27B weights are flour. The recipe is which oven (SGLang vs vLLM vs MLX), what temperature (NVFP4 / FP8 / BF16), which intern drafts the next sentence (MTP / DSpark / DFlash2), how many guesses before the senior checks (k / block size), how much of the fridge is reserved for leftovers (KV + GDN pool), and how you time the dish.

What they actually ship, in order of usefulness:

1. A pin of a working image and checkpoint so you do not wander Hugging Face.
2. A flag stack with a reason next to each flag: cookbook said X, we measured −3%, pin Y.
3. A sequential test order so you do not change five things at once.
4. Workload-split numbers. Speculation is not one tok/s.
5. A concurrency ladder, because the k that wins at one stream can lose 43% of fleet throughput at eight.
6. Gotchas that cost a full model load: wrong parser name, prefix cache silently off, SSE under-counting.

2. Mia — SGLang, NVFP4, three engines

Repo: MiaAI-Lab/Qwen3.8-27B-SGLang-DGX-Spark. Hardware: NVIDIA DGX Spark, GB10, 128 GB unified, ~273 GB/s. Engine: cookbook-pinned lmsysorg/sglang:qwen38-27b, plus a derived image for DFlash2 because that merged upstream after every published tag.

Three swap-in scripts, same NVFP4 27B, only the drafter changes:

Flags they pin after measuring

They also pin FlashInfer, torch.compile + decode graphs, prefill graphs off, thinking parser qwen3, tool parser qwen3_coder. NGRAM spec rejected (~30% under MTP). Accept threshold left at 1.0.

Sequence, reconstructed

1. Start from the SGLang cookbook DGX Spark + NVFP4 + DSpark cell. Do not invent a flag stack from Twitter.
2. Pin the image digest. If DFlash2 is not in that tag, build a derived image. Do not mix nightlies mid-bench.
3. Tier A, kernel path: they report zero net gain once the cookbook path was correct.
4. Tier B, config/host: GDN dtype, radix, mem fraction, chunk, cpuset, pool formula. One change. Re-baseline in-session. The box drifts ~1.5 tok/s under power-cap.
5. Drafter A/B on two probes that disagree: LRUCache code and a long essay. Code deltas under 15% are noise. The essay is the discriminator.
6. Sweep block / EAGLE tuple. Pin the winner. Offer the runner-up as an env override, not a second default.
7. Count from completion_tokens. DFlash2 batches ~3.75 tokens per SSE event at a fixed ~8 events/s.
8. Concurrency ladder last, after mem-fraction is crash-safe.
9. Crash-learn: DFlash2 + mem 0.95 dequantized the whole NVFP4 lm_head (~2.5 GB) at CUDA-graph capture and hard-rebooted GB10. Fix: in-place lm_head.quant_method.apply.

3. 0xBakeer — vLLM, FP8, measured k

Repo: 0xBakeer/Qwen3.8-27B-FP8-on-a-single-DGX-Spark. Same Spark, different oven: official vLLM 0.27.1 aarch64, target Qwen/Qwen3.8-27B-FP8. This is the cleaner lab notebook. Speed only. No quality eval. Read LIMITATIONS.md before quoting.

k=15 is a trap on mixed workloads: best MTP on edits, worst overall on fresh generation. No single k wins both latency and fleet throughput.

The findings that cost a weekend if you skip NOTES.md

1. Prefix caching is off by default on hybrid models. vLLM computes default = supported and not is_hybrid. Qwen3.8-27B is hybrid. You must pass --enable-prefix-caching. Worth 14–22× on long shared prefixes. Costs ~22% of KV capacity.
2. k must match workload predictability. Edit/refactor accept 98%+. Fresh gen at MTP k=15 is 28.9% accept and slower than k=3.
3. Drafter architecture beats draft depth. Cost per draft token: MTP 0.153, DSpark 0.046. DSpark accepts fewer tokens per pass and is still 46% faster.
4. k rank reverses under concurrency. k=14 is the fastest one-stream config and loses 43% of aggregate at c=8 versus k=7.
5. A k tuned on one quant does not transfer. Same k=14 costs ~5% at c=8 on 4-bit.
6. The 4-bit advantage vanishes under concurrency. DSpark k=7, FP8 vs NVFP4: +27% at c=1, +10% at c=8, +0.2% at c=16. Single-stream decode is bandwidth-bound. A batch makes one weight read serve many sequences and the byte count stops mattering.
7. Adaptive verification cannot run on this model. GDNAttentionBackend does not support it. Architectural, any GPU. Loud failure at KV init, six minutes into startup.
8. Thinking is on by default at xhigh. Benches that omit enable_thinking: false and a max_tokens cap are measuring a reasoning novel.
9. Parser names do not follow a pattern. qwen3 for reasoning, qwen3_xml for tools on vLLM. A typo costs a full load.

DFlash2 later, same device: generative 31.72 vs DSpark 19.12 vs MTP 17.99 vs stock 7.94 (accept 4.61). That win is single-stream. At c≥2, 4-bit plus free in-checkpoint MTP overtakes on their curve. On vLLM, FP8 is the only Qwen3.8-27B build that can serve DFlash2, because the selector wants an unquantized LM head.

4. The playbook

Collapsing both cooks plus the SGLang cookbook into something you can run on any oven:

1. Phase 0, define the dish. One model, one box, one engine. Two probes that fight: copy-friendly (code edit) and fresh prose (essay). Utility checklist before speed: parsers, context you will actually use, thinking off for decode benches.
2. Phase 1, honest serial. Speculation off. Tokens/sec ≈ bandwidth / bytes per step. ~16.5 GB NVFP4 weights: Spark AR ~13–16 t/s, M4 Max (~546 GB/s) AR ~33 t/s. Anything sustainably above that is speculation or a counting bug.
3. Phase 2, native spec, small. In-checkpoint MTP at a small tuple. Record accept length. If accept is ~1.0, stop. Do not go deeper.
4. Phase 3, hybrid footguns. Prefix cache often supported but off. GDN state does not trim() like KV on partial reject. YaRN can leak into the draft config.
5. Phase 4, drafter architecture before draft depth. Cheaper guesses beat more guesses.
6. Phase 5, memory geometry. Mem-fraction, KV dtype, GDN pool = concurrency × slots. Too high: capture OOM. Too low: k refuses to start.
7. Phase 6, host/kernel last. CPU pinning, graphs, attention backend. These are 2–7% moves. Drafter + k are 2–4×. People invert this order because kernel tweets are more fun.
8. Phase 7, concurrency ladder. 1, 2, 4, 8, 16. Publish aggregate and per-stream. Re-choose k. Re-choose quant.
9. Phase 8, count correctly, freeze. completion_tokens. n≥5 on the discriminator. Pin image digest + flag file.

5. 64 layers: ~400M each? Why only 16 keep KV?

From the Qwen3.8-27B card: dense 27B, hidden 5120, 64 layers, vocab padded to 248,320, FFN intermediate 17,408, native 262,144 context, YaRN to 1M, native MTP, native vision. Layout: 16 × (3 × GDN→FFN then 1 × gated attention→FFN). That is 48 Gated DeltaNet layers and 16 full-attention layers. 3:1.

Is it ~400M parameters per layer?

As a back-of-envelope after embeddings, yes. Architecturally, no. Layers are not equal, and most of a layer is not attention.

• Naive divide: 27e9 / 64 ≈ 422M if every parameter lived in a layer. They don’t.
• Embedding: 248,320 × 5,120 ≈ 1.27B. The LM head is the same shape. ~2.5B sits outside the stack. Remainder ≈ 24.5B / 64 ≈ 383M per layer average.
• FFN alone, three SwiGLU matrices 17,408 × 5,120: ~267M per layer. About two thirds of each floor is the cafeteria.

3 × 17,408 × 5,120 = 267,386,880

Full-attention GQA (24 query heads / 4 KV heads, head dim 256) is about 73M plus gates. GDN uses a different split (48 value heads, 16 QK heads, head dim 128) and a recurrent state update. Different math, similar width, so layers stay similarly sized.

Why this architecture?

Classic transformers do full attention every layer. Attention cost and KV memory grow with sequence length. At 262K–1M that becomes the product. Linear / recurrent mixers (Mamba, DeltaNet, Gated DeltaNet) keep a fixed-size state. You fold the new token into a matrix of constant size. The weakness: the notepad forgets sharp token-to-token pointers. The 3:1 hybrid, also used in Qwen3-Next / 3.5 / 3.6, spends cheap long-range glue on 75% of depth and real attention every fourth layer so retrieval and copying still work. That is how a 27B dense model advertises 262K native context on a 128 GB box.

Why only 16 layers keep KV cache?

Only the 16 gated-attention layers do attention that needs keys and values from previous tokens. The 48 GDN layers have a recurrent state of fixed size. Nothing there grows with t.

16 layers × 4 KV heads × 2 (K and V) × 256 dim × 2 bytes
= 65,536 bytes ≈ 64 KB per token (BF16)

FP8 KV (Mia): half of that ≈ 32.8 KB per token
1M sequence ≈ 33 GB

If all 64 layers were full attention, multiply by 4: ~131 KB/token FP8, ~131 GB for 1M tokens of KV alone. Mia’s GDN pool is extra and does not grow with sequence length (~78 MB/slot × concurrency × 4; default 10 concurrent → 40 slots ≈ 3.1 GB).

6. The three drafters

Autoregressive decode, no tricks: the 27B reads all its weights once per token. That is the 15–33 t/s floor depending on box and quant.

Speculative decoding: a cheap intern proposes the next k tokens. The senior (the 27B) checks them in one weight-read. If the intern got five right, you paid one senior pass for five tokens. The senior’s distribution still wins. Speculation cannot force a token. It can change text across configs because batch geometry changes floating-point order. 0xBakeer measured 8/20 greedy prompts byte-identical across tuned vs stock. Deterministic within a config, not across configs.

7. CUDA graphs, FlashInfer, FA3, the GDN pool

The Spark recipe is four NVIDIA serving technologies plus a GDN state machine. They are not synonyms, and they are not sitting in MLX under different names.

Two runtimes, two contracts

SGLang is a CUDA-first serving engine: paged KV, radix prefix cache, overlap scheduler, speculative runners (EAGLE, DSpark, DFLASH), and a zoo of NVIDIA kernels. The Spark “recipe” is a cell in the SGLang cookbook that Mia started from and then pinned.

MLX is Apple’s array library. The GPU language is Metal. Graphs are lazy evaluation plus optional mx.compile, not CUDA graph capture. Speculation is whatever mlx_vlm / mlx-dspark implemented in Python plus those shaders.

SGLang has an experimental SGLANG_USE_MLX=1 backend. It is a partial port of the scheduler onto MLX arrays. It does not give you FlashInfer, CUDA graphs, or Mia’s DFlash2 runner. Running it on a Mac is SGLang Python overhead without the CUDA kernel zoo. Wrong runtime, incomplete port, slower.

CUDA graphs: recording the dance

A GPU is fast. Telling it what to do from the CPU is not, when each decode step is a few milliseconds of math. Normally: CPU launches kernel, GPU runs, CPU launches next. For 64 layers that is dozens of launches per token. Launch overhead can rival the math when the math is skinny.

CUDA graph capture records a sequence of launches once, then replays the whole sequence with one submit. Pointers and shapes must be fixed. That is why batch size is capped (--cuda-graph-max-bs-decode 4), why changing k can force a recapture, and why Mia’s DFlash2 crash happened at graph capture when a 2.5 GB dense head suddenly materialized.

Analogy: teaching a kitchen line by shouting each step versus filming the service once and pressing play. Replay is faster until someone orders a dish that is not in the film.

FlashInfer FP4 GEMM: the multiply kitchen

GEMM is the matrix multiply behind almost every linear layer. FlashInfer is the kernel library behind SGLang and vLLM: attention, GEMM, MoE, FP8/FP4, CUDA-graph compatible workspaces. On Blackwell it includes NVFP4 GEMM that runs on Tensor Cores with weights and activations in 4-bit (W4A4), accumulate in FP32.

NVFP4 the codec (portable): E2M1 4-bit values, 16-wide E4M3 FP8 scale, optional FP32 tensor scale. MLX can store this packing.

NVFP4 the Tensor Core datapath (Spark): the matrix unit swallows FP4×FP4. That is W4A4.

NVFP4 on MLX Metal: a shader looks up the E2M1 nibble, multiplies the E4M3 scale, and dots against fp16/bf16 activations. That is W4A16. You saved weight bandwidth. You did not get Blackwell FP4 MMA.

FA3: FlashAttention 3

Naive attention builds a huge score matrix, softmax, times values. That matrix does not fit in fast memory at long context. FlashAttention tiles the work so softmax is done online in SRAM. FA2 improved tiling. FA3 (Hopper-oriented, now a serving backend) overlaps math and memory better, unifies prefill and decode so the whole model including attention can sit inside one CUDA graph, and is a selectable SGLang/vLLM attention backend.

FlashInfer vs FA3: overlapping kitchens. FlashInfer is the broader serving library (paged KV, decode split-K, FP4 GEMM). FA3 is a specific attention algorithm that often wins throughput and graph-friendliness on NVIDIA. Mia pins FlashInfer on the Spark NVFP4 cell. MLX attention is neither.

The GDN pool: extra_buffer, lazy, ReplaySSM

Each of the 48 GDN layers holds a recurrent state per request. Speculation mutates state for tokens that might be rejected. Prefix cache wants to store a GDN snapshot in the radix tree without freezing the live request. Strategies:

• no_buffer: tight, fights with cache/spec.
• extra_buffer: a second slot so the running request keeps mutating while a snapshot goes to the tree.
• extra_buffer_lazy: allocate that extra lazily. Mia’s DSpark/MTP pin. DFlash2 asserts and rejects it.

Mia’s formula: max-mamba-cache-size = MAX_CONCURRENT_REQUESTS × 4. The spec verify window is a separate engine buffer. If you also multiply by draft tokens you over-provision ~2×. ReplaySSM (Tri Dao, 2026; used on SGLang’s DSpark path) stores a few hundred bytes of raw inputs per draft step and, after the sampler picks the accepted length, folds the accepted prefix from a checkpoint. Bit-identical to running those tokens for real. That is what lets DSpark compose with radix prefix cache on mutable GDN state.

One paragraph

Mia’s 56.6 / 227.6 is NVFP4 weights consumed as W4A4 on Tensor Cores, FlashInfer FP4 GEMMs, FA3-class attention option, decode CUDA graphs so the CPU is not the bottleneck, an FP8 KV cache so 16 attention layers stay cheap, a GDN pool and ReplaySSM so speculation and prefix cache do not fight the 48 notepads, and a DFlash2 runner fused into that graph. A Mac has the same weight codec and can run the same DFlash2 tensors in software. It does not have that kernel orchestra. SGLang-on-Mac does not import the orchestra. It imports a scheduler onto Metal without the instruments.

8. How they count, and how people lie by accident

• completion_tokens / engine counters are the only number that is a token.
• SSE events: DFlash2 on Mia’s image emits ~8 events/s, ~3.75 tokens each. Event-counting reads ~9 tok/s when the server is doing ~30.
• Three clocks: wall-time including prefill, net-decode after first token, and a synthetic structural concurrency fixture. Mix them and you can “beat” anyone.
• Aggregate vs per-stream: 227.6 at 16 streams is 28.2 per client. A single-user chat never sees 227.
• Thinking on, uncapped: you are not measuring decode.

9. Field take: what copies onto a Mac

I ran this research because the Spark recipes look copy-pasteable onto an M4 Max 128 GB, which has roughly 2× the memory bandwidth of GB10. That bandwidth argument only sets the serial floor. Mia’s 56.6 is already ~3.4× Spark autoregressive. Doubling that on Metal is not a bandwidth identity.

Steal their lab order, not their kernel list. Serve MLX if you are on this silicon. Compare two scoreboards honestly: single-stream speculation (the Mac can play) versus 16-way graph batching (Spark can play). Keep hard prose next to any structural fixture. Speculation that only wins on LRUCache is not a chat recipe.

10. Sources

• MiaAI-Lab — Qwen3.8-27B SGLang DGX Spark (flags, measured table, DFlash2 ladder, counting note)
• 0xBakeer — Qwen3.8-27B FP8 on a single DGX Spark (README, NOTES.md, RESULTS.md)
• Hugging Face — Qwen/Qwen3.8-27B (64-layer layout, GDN vs gated attention, FFN 17408)
• SGLang cookbook — Qwen3.8-27B (Spark cell, GDN radix, ReplaySSM, DSpark)
• FlashInfer (FP4 GEMM, CUDA-graph workspaces)
• z-lab / incoai — Qwen3.8-27B-DFlash2
• inco.ai — DFlash2 write-up
• Kubesimplify — reproducing 0xBakeer on Spark
• Related on this site: Qwen 3.8 27B vs DeepSeek V4 Flash, KV cache, prefix caching