MoE Routing

MoE routing is the gating mechanism that decides which experts a token activates in a Mixture-of-Experts layer. Top-k routing (each token picks its k highest-scoring experts) is dominant — Mixtral and DeepSeek use top-2 and top-8 respectively.

Routing quality depends on training: untrained or poorly-trained routers cluster tokens onto a few experts (load imbalance), wasting capacity. Auxiliary load-balancing losses during training fix this.

At inference, routing introduces an all-to-all communication step in distributed setups; on a single GPU, it's a sparse gather. Quantizing experts independently is harder than quantizing dense weights — different experts have different activation distributions.

An operator quantizing Mixtral 8x7B to run on a single 24 GB GPU discovers that naively applying a uniform Q4_K_M scheme across all experts leaves some experts noticeably more degraded than others, because MoE routing during training didn't distribute tokens evenly — a handful of experts saw far more training signal than others and have different activation ranges. Tools like llama.cpp's importance-matrix quantization help by calibrating per-tensor scales against a real text sample rather than assuming uniform distributions, meaningfully closing the gap versus naive quantization. At serving time, the operator also notices that batching requests with very different topics hurts throughput more than expected on a single GPU, since divergent routing decisions per token mean less benefit from batched matmuls than a dense model would give — the sparse gather pattern doesn't batch as cleanly as dense compute does.