RMSNorm

RMSNorm is a simpler variant of LayerNorm that normalizes activations by their root-mean-square instead of their variance, skipping the mean subtraction and bias term. Used in Llama, Mistral, Qwen, and most modern open-weight LLMs.

The benefit is small but real: ~7% faster than LayerNorm with no quality loss on language modeling benchmarks. The simplicity also makes RMSNorm easier to fuse into preceding/following kernels for further speedup.

Quantization-time gotcha: RMSNorm scales are FP16/FP32 even when the surrounding linear layers go to INT4. Some early GGUF converters lost the scales; check that your converter handles them.

Say you're converting a Qwen2.5 7B checkpoint to GGUF for llama.cpp and quantizing to Q4_K_M. If your conversion script mishandles the RMSNorm weight tensors — flattening them into the same INT4 quant group as the adjacent linear layer instead of keeping them at FP16 — you'll see the model degrade sharply on long-context generation even though short prompts look fine, because norm scale errors compound across layers. The fix is to check the GGUF metadata for attn_norm.weight and ffn_norm.weight tensor types before running inference; llama.cpp's official converter keeps these at F32 by default. This is also why RMSNorm-heavy architectures like Llama 3.1 and Mistral tolerate aggressive weight quantization better than older LayerNorm-based models: fewer parameters (no bias, no mean term) means fewer places for quantization error to hide.