Part 5: Sharding & Mixture of Experts

Classic scale-out: replicate the model

For “regular” models, horizontal scale is boring in a good way: copy the weights to N independent replicas; load-balance requests. Each replica is a full model. That still exists for LLMs — and for dense midsize models it is often the right answer.

But frontier LLMs break the “one GPU, one replica” story in two directions:

1. The model is bigger than one GPU → weight sharding (tensor / pipeline / expert parallel).
2. The model is sparse MoE → only a subset of experts activate per token; routing becomes dynamic inside the forward pass.

Parallelism taxonomy (dense models)

Inference cares about different costs than training: you may accept less perfect FLOPs efficiency to hit latency SLOs. TP within a node (NVLink) is common; PP across nodes needs careful microbatching or you bubble. Engines (vLLM, TensorRT-LLM, Megatron-derived stacks) expose these as deployment-time parallel configs.

Mixture of Experts — sparse capacity

An MoE layer replaces a single MLP with E expert MLPs plus a router (gating network). For each token, the router selects top-k experts (often k=1 or 2). FLOPs scale with activated experts, not E. Parameter count can be huge (hundreds of B) while active parameters stay closer to a dense 10–30B-class forward.

Why serving MoE is a systems problem

• Expert placement: experts sharded across GPUs; tokens must travel to the GPUs that hold their experts (all-to-all).
• Load imbalance: popular experts overheat; capacity factors and aux losses during training try to prevent this, but inference still sees skew.
• Batching interaction: continuous batching now also means dynamic expert traffic matrices every step.
• Memory: all experts’ weights must live somewhere even if cold — capacity planning is about resident experts + KV, not only active FLOPs.

A common pattern Nishihara described: replicate attention (or keep it densely parallelized) while sharding experts. Tokens fan out to expert GPUs, compute, and fan in. From the outside it looks like one logical model; inside it is a routed fabric.

EP vs TP for experts (and hybrids)

• Expert parallel (EP): each device hosts a subset of full experts; tokens all-to-all to the ranks that own their top-k experts (dispatch + combine).
• Tensor parallel (TP) on MoE: each expert’s matrices are sharded so every GPU sees all tokens’ hidden states — simpler routing, worse scale for huge sparse models.
• Hybrid: stacks such as TensorRT-LLM allow \[ \mathrm{moe\_tp\_size} \times \mathrm{moe\_ep\_size} = \mathrm{tensor\_parallel\_size} \]

In vLLM-style deployments, enabling EP often sets \[ \mathrm{EP\_SIZE} = \mathrm{TP\_SIZE} \times \mathrm{DP\_SIZE} \] Expert layers then shard across EP ranks while attention uses DP (replicate when TP=1) or TP. Multi-node EP uses specialized all-to-all backends (high-throughput paths for prefill, low-latency for decode). SGLang reports order-of-magnitude decode wins for large sparse MoEs when PD disaggregation is combined with EP versus pure wide TP baselines (config-specific; e.g. multi-× vs TP16 on DeepSeek-V3-class stacks).

Expert Parallelism Load Balancer (EPLB)

Hot experts overheat; cold experts waste memory bandwidth elsewhere. Production EP systems (vLLM, SGLang, TensorRT-LLM Wide-EP) run EPLB: re-place or replicate hot experts from activation statistics. That costs redundant-expert memory (on the order of ~GB per redundant expert per rank for DeepSeek-V3-class models) to buy balanced all-to-all and stable latency.

Models you already use that force this

Modern open weights are full of MoE: Mixtral-class, DeepSeek-V3-class, Qwen-MoE, Llama MoE variants, coding models like Laguna-class sparse designs. If you serve them:

• Prefer engines with first-class MoE kernels and expert parallel (EP) support.
• Benchmark with real token routing distributions — synthetic uniform tokens lie.
• Watch all-to-all time in profiles; fabric choice (NVLink domain vs Ethernet) decides feasibility.

Putting all five together

The five differences are one system:

1. Continuous batching keeps the GPU full under variable lengths.
2. Prefill/decode awareness (and maybe disaggregation) matches phase to hardware.
3. Paged KV makes memory allocatable at iteration granularity.
4. Prefix cache + routing amortizes shared stems across the fleet.
5. Sharding / MoE makes models that do not fit — or should not fully activate — serveable.

That is why “just run model.generate” is a demo, and vLLM/SGLang/TRT-LLM + an orchestration layer (Ray, k8s, Dynamo-style) is a product.

Operator checklist for MoE + sharding

• Map parallel strategy: TP size within node, PP/EP across nodes, replica count for HA.
• Cap concurrent sequences using KV + expert workspace, not weight size alone.
• Alert on expert imbalance and all-to-all latency, not only GPU util %.
• Version routing kernels with model weights — MoE bugs show up as quality cliffs, not crashes.

Sources & further reading

• Megatron-LM / tensor & pipeline parallelism literature
• TensorRT-LLM expert parallelism
• SGLang expert parallelism
• NVIDIA — model parallel techniques
• Robert Nishihara on MoE sharding vs simple replication (NYC walk transcript)
• Anyscale — K8s + Ray + PyTorch + vLLM stack