ALiBi (Attention with Linear Biases)

ALiBi is a positional encoding scheme that biases attention scores by a linear function of token distance, instead of injecting position into queries and keys. Used in MPT (early MosaicML models) and BLOOM.

Advantage: clean extrapolation to lengths beyond training. A model trained on 2K context with ALiBi often handles 8K+ at inference with no fine-tuning, where RoPE degrades.

ALiBi has fallen out of favor — most current open-weight LLMs use RoPE with YaRN or similar extensions instead. Worth knowing because BLOOM and several derivative models still ship with it, and the local inference stack handles ALiBi differently from RoPE.