Layer Normalization
Layer normalization is a technique that stabilizes training and inference by normalizing activations across the features of each token independently, rather than across the batch. In transformer models, it is applied after each sub-layer (attention or feed-forward) and before the residual connection. This prevents activations from growing too large, which is critical for deep networks. Operators encounter layer norm as a fixed computation in every forward pass—it adds a small, constant latency per token but is essential for model stability, especially at lower precision (FP16, INT8) where range control matters.
Deeper dive
Layer normalization computes the mean and variance of the activations for a single token across all hidden dimensions, then shifts and scales them using learned parameters (gamma and beta). Unlike batch normalization, it does not depend on batch statistics, making it suitable for autoregressive decoding where batch size is often 1. In transformers, the standard placement is "post-norm" (after each sub-layer, before residual), but some modern architectures like Llama 2 use "pre-norm" (before the sub-layer) for better training stability. The operation is element-wise and memory-bound—on GPU, it's typically fused into the preceding kernel (e.g., in flash attention implementations) to avoid extra memory round-trips. Quantization-aware training often adjusts layer norm parameters to preserve dynamic range.
Practical example
When running Llama 3.1 8B at Q4_K_M on an RTX 4090, each token passes through 32 transformer layers, each containing two layer norms. The runtime spends roughly 1-2% of total compute on these norms—negligible compared to matrix multiplies. However, if you quantize to Q2 or Q3, the reduced precision can cause layer norm outputs to saturate, degrading perplexity. Operators see this when comparing quantized model quality: a model that runs well at Q4 may hallucinate more at Q2 partly due to layer norm precision loss.
Workflow example
In llama.cpp, layer normalization is implemented as a fused operation in the ggml backend. When you run ./main -m model.gguf -p "Hello" -n 1, the inference loop calls ggml_norm for each layer. You can inspect the model's layer norm parameters in the GGUF file using python -c "import gguf; r = gguf.GGUFReader('model.gguf'); print(r.fields['blk.0.attn_norm.weight'].data)". In Hugging Face Transformers, the LlamaModel class applies nn.LayerNorm with eps=1e-5—changing epsilon can affect stability at very low precision.
Reviewed by Fredoline Eruo. See our editorial policy.