ALiBi (Attention with Linear Biases)

ALiBi is a positional encoding scheme that biases attention scores by a linear function of token distance, instead of injecting position into queries and keys. Used in MPT (early MosaicML models) and BLOOM.

Advantage: clean extrapolation to lengths beyond training. A model trained on 2K context with ALiBi often handles 8K+ at inference with no fine-tuning, where RoPE degrades.

ALiBi has fallen out of favor — most current open-weight LLMs use RoPE with YaRN or similar extensions instead. Worth knowing because BLOOM and several derivative models still ship with it, and the local inference stack handles ALiBi differently from RoPE.

An operator resurrecting an old BLOOM-176B derivative for a legacy multilingual project notices their llama.cpp config script — copy-pasted from a Llama 3 setup — has RoPE scaling flags that do nothing, because BLOOM uses ALiBi instead of RoPE for positional encoding. The inference engine has to branch on architecture: ALiBi models bias attention scores directly with a fixed linear penalty per distance, so there's no rope_freq_base or rope_freq_scale to tune at all. On the plus side, when the operator needs to push context past BLOOM's training length for a batch job, it degrades far more gracefully than an untuned RoPE model would — no YaRN or NTK scaling required, since ALiBi's linear bias was designed to extrapolate. Worth knowing before you assume every GGUF conversion in your model zoo takes the same context-scaling flags.