YaRN (Yet another RoPE eNlargement)

YaRN is a context-extension method that modifies RoPE frequencies to let a model trained on, say, 8K context generalize to 32K or 128K with minimal fine-tuning. Used in Qwen 2.5, Mistral Nemo, and several Llama 3 long-context derivatives.

Compared to naive frequency scaling (linear or NTK-by-parts), YaRN preserves position discrimination at long range better, with measurable improvement on needle-in-haystack benchmarks past 32K.

Practical implication: when you see "extended to 128K with YaRN" on a model card, expect quality degradation past the original training context to be smaller than with vanilla RoPE scaling, but still real — long-context performance is rarely as good as short-context.

A developer building a document-summarization pipeline picks Qwen 2.5 14B specifically because its model card advertises YaRN-extended context to 128K, needing to feed in 80-page PDFs converted to text. Running it in llama.cpp requires explicitly setting the YaRN scaling parameters (--rope-scaling yarn, plus the original and target context lengths) — forgetting this and just cranking --ctx-size to 128K without enabling YaRN produces degraded, sometimes incoherent output because the model was never trained to extrapolate RoPE frequencies that far without the scaling correction. Once configured correctly, retrieval accuracy on content near the end of a 100K-token document is noticeably better than naive linear RoPE scaling would give, though the developer still tests with needle-in-haystack-style probes rather than trusting the 128K claim blindly, since quality past the original 8K-32K training window is never quite as sharp as native short-context performance.