Expert Parallelism

Expert parallelism is a parallelism strategy specific to MoE models: each GPU holds a different subset of the experts, and tokens are routed to whichever GPU owns the expert they activate. Distinct from tensor parallelism (split each layer's weights) and pipeline parallelism (split layers across devices).

The advantage: at inference, only the active experts run, so expert-parallel MoE serves at lower per-token compute than a dense model of the same total parameters. The cost: routing tokens between GPUs adds an all-to-all communication step at every MoE layer.

Mixtral 8x7B, DeepSeek-V3, and Qwen3-MoE typically deploy with combined expert + tensor parallelism on multi-GPU servers. Single-GPU deployments use a degenerate form where the routing happens inside one device.

An operator standing up DeepSeek-V3 (671B total, ~37B active) on 8×H100 needs to decide how to split it. Loading the full dense weights across 8 GPUs with pure tensor parallelism wastes bandwidth shuttling activations for experts that never fire on a given token. Instead, they configure expert parallelism across the 8 GPUs — each GPU owns roughly 1/8th of the expert pool — combined with tensor parallelism within each GPU's local experts for the attention layers. At serving time, a batch of tokens routed to experts scattered across all 8 devices triggers an all-to-all exchange every MoE layer; on NVLink-connected H100s this stays cheap, but the same setup over PCIe-only interconnects (or across multiple nodes without InfiniBand) turns communication into the bottleneck. This is why MoE serving frameworks like SGLang expose separate --ep-size and --tp-size flags — getting the ratio wrong silently caps throughput well below what the active-parameter count would suggest.