Serving MoE models with expert parallelism on 8×H200

Why MoE serving is its own problem

Mixture-of-experts models have become the default shape for large open-weight releases: DeepSeek-V3 and R1, Qwen3-235B-A22B, Llama 4 Maverick, gpt-oss-120b. They share a property that breaks intuitions built on dense models. Total parameters decide how much memory you need. Active parameters decide how much compute each token costs. The gap between the two is often 10-30×.

That gap is why MoE models are cheap to run per token and awkward to deploy. You must hold every expert in memory even though each token touches only a handful, and the way you split those experts across GPUs determines whether you get the efficiency the architecture promises. This guide covers the memory math, the parallelism choice, batching, and routing imbalance, with vLLM and SGLang as the reference engines.

The models, by the numbers

Figures below come from the public model cards and papers; they are rounded.

Memory math for 8×H200

Eight H200s give you 8 × 141 GB = 1,128 GB of HBM. Plan to lose 5-10% of that to CUDA context, NCCL buffers, CUDA graphs, and activation workspace, so budget roughly 1,030 GB of usable memory.

Weights. Bytes per parameter times total parameters:

• DeepSeek-V3 in its native FP8: ~671 GB. Fits, with roughly 350 GB left for KV cache.
• Qwen3-235B in BF16: ~470 GB. In FP8: ~235 GB. Both fit; FP8 leaves far more room for context.
• Llama 4 Maverick in FP8: ~400 GB. Fits comfortably.
• gpt-oss-120b in its native MXFP4: ~60-65 GB. Fits on a single 80 GB GPU; eight H200s are for throughput, not capacity.

KV cache. This is where the architecture matters as much as the parameter count.

Qwen3-235B uses grouped-query attention with 4 KV heads of dimension 128 across 94 layers. In BF16 that is 2 (K and V) × 4 × 128 × 2 bytes × 94 layers ≈ 188 KB per token. A 32k-token request needs about 6 GB; 50 such requests in flight need about 300 GB.

DeepSeek-V3 uses multi-head latent attention (MLA), which caches a compressed 512-dimension latent plus a 64-dimension rotary key per layer. In BF16 that is (512 + 64) × 2 bytes × 61 layers ≈ 70 KB per token, so a 32k request is about 2.3 GB. MLA is the reason a 671B model can serve meaningful concurrency on a single node.

A sizing rule we use: usable memory minus weights, divided by KV bytes per token, gives total tokens in flight. Divide by your average sequence length (prompt plus output) to get a ceiling on concurrent requests. If that ceiling is below the batch size you need for throughput, quantize the weights or the KV cache before adding nodes.

Tensor parallel vs expert parallel

Tensor parallelism (TP) splits every weight matrix across GPUs. For dense layers and attention, it works well on a single NVLink node. For MoE layers, it splits every expert eight ways. DeepSeek-V3's routed experts have an intermediate size of 2,048; split across 8 GPUs, each GPU holds a 256-column slice of every expert. Those GEMMs are tiny, inefficient, and each one ends in an all-reduce.

Expert parallelism (EP) gives each GPU whole experts instead. With 256 routed experts on 8 GPUs, each GPU owns 32. Tokens are dispatched to the GPUs that hold their selected experts through an all-to-all, processed with full-size expert GEMMs, and sent back with a second all-to-all. Inside one node, NVLink at 900 GB/s per GPU handles that traffic well. Across nodes, it becomes the dominant cost, which is why multi-node EP depends on optimized all-to-all kernels like DeepEP and a good InfiniBand fabric.

Attention needs its own decision. With TP, attention heads are split across GPUs. But MLA has effectively one shared latent per token, so TP attention replicates the KV cache on every GPU, which wastes most of the memory you just budgeted. The answer is data-parallel attention: each GPU runs attention for its own subset of requests with its own KV cache, then joins the expert-parallel MoE layers. For GQA models like Qwen3, TP attention is fine as long as the KV head count divides evenly, though DP attention can still help at high concurrency.

A rough decision table:

Launching with vLLM and SGLang

Flag names move between releases, so treat these as a starting point and check the docs for your version.

# vLLM: TP for attention, EP for the MoE layers
vllm serve Qwen/Qwen3-235B-A22B-FP8 --tensor-parallel-size 8 --enable-expert-parallel --max-model-len 32768 --gpu-memory-utilization 0.90

# vLLM: DP attention + EP, the usual shape for MLA models
vllm serve deepseek-ai/DeepSeek-V3 --data-parallel-size 8 --enable-expert-parallel --max-model-len 32768

# SGLang: TP 8 with DP attention and EP across all 8 GPUs
python -m sglang.launch_server --model-path deepseek-ai/DeepSeek-V3 --tp 8 --enable-dp-attention --dp 8 --ep-size 8

Confirm from startup logs that experts are actually distributed (the per-GPU weight footprint should be about one-eighth of the MoE weights) and that the KV cache allocation matches your sizing math.

Batching: MoE needs bigger batches than you think

With top-8 routing over 256 experts, a batch of B tokens gives each expert, on average, B × 8 / 256 = B / 32 tokens. At a decode batch of 64 concurrent sequences, the average expert sees two tokens per step. That expert GEMM is entirely memory-bound: you stream the expert's weights from HBM to multiply them by two vectors.

The consequences:

• Throughput rises steeply with concurrency. Going from 64 to 512 concurrent sequences raises per-expert work 8× while weight traffic stays nearly flat. Size your KV budget to reach the batch where expert GEMMs become efficient.
• Separate prefill from decode when you can. Prefill already provides thousands of tokens per step and saturates experts; decode does not. Chunked prefill or prefill/decode disaggregation keeps long prompts from stalling decode batches.
• Latency SLOs cap the batch. Larger batches raise time per output token. Find the knee by sweeping concurrency against your p95 inter-token latency target, as described in our vLLM production guide.

Routing imbalance

Average load is not real load. Routers learn preferences, and real traffic is skewed: a code-heavy workload lights up different experts than a chat workload. If one GPU holds several hot experts, it becomes the straggler, and every all-to-all waits for it.

What to do about it:

1. Measure. Log per-expert token counts per layer over a representative traffic sample. The ratio of the busiest GPU's token count to the mean is your imbalance factor; above roughly 1.3 is worth addressing.
2. Replicate hot experts. Expert-parallel load balancers (EPLB in vLLM and SGLang) place redundant copies of popular experts on less-loaded GPUs and rebalance periodically. This costs a little memory per redundant expert.
3. Re-shuffle placement. Even without replication, co-locating hot and cold experts evens out per-GPU load.
4. Re-measure after traffic shifts. A placement tuned for one product mix can degrade when a new customer workload arrives.

Quantization choices

Native FP8 checkpoints (DeepSeek-V3, Qwen3 FP8 variants) are the safe default on Hopper. Further quantizing expert weights to 4-bit with methods like AWQ or GPTQ halves the weight footprint again and frees memory for KV cache. Because a token touches only a few experts, quality loss in rarely used experts can hide in average benchmarks. Evaluate on your own tasks, not just leaderboard suites. FP8 KV cache is another cheap win, roughly halving per-token KV memory with usually small quality impact. Our inference cost guide covers the evaluation workflow.

Getting the hardware

Single H200 SXM instances are available on demand in our GPU cloud, which is enough for gpt-oss-120b replicas and for prototyping smaller MoE models. For an 8-GPU NVLink domain to serve DeepSeek-class models, contact our sales team about reserved HGX capacity. Current on-demand and reserved rates are on pricing.

FAQ

Can DeepSeek-V3 run on a single 8×H200 node?

Yes. Its native FP8 weights are about 671 GB against 1,128 GB of HBM, and MLA keeps the KV cache small, roughly 70 KB per token in BF16. That leaves room for substantial concurrency at 32k context.

Should I use tensor parallelism or expert parallelism for MoE?

For models with many small experts, expert parallelism for the MoE layers is usually faster because each GPU runs full-size expert GEMMs. Pair it with data-parallel attention for MLA models and tensor-parallel attention for GQA models.

Why is my MoE throughput low at small batch sizes?

Each expert sees only a small fraction of the batch, so expert GEMMs are memory-bound. Increase concurrency, separate prefill from decode, and free KV memory with FP8 KV cache to reach efficient batch sizes.

What is routing imbalance and how do I fix it?

It is uneven token load across experts, which turns the GPU holding hot experts into a straggler. Measure per-expert load, then use an expert-parallel load balancer to replicate or redistribute hot experts.