Context Window

The context window is the maximum number of tokens a model can attend to at once — both prompt and previously generated tokens. Llama 3.1 8B has 131,072 (128K). Llama 4 Scout has 10 million. Older models like the original GPT-3 had 2,048.

Bigger context windows aren't free. Memory grows linearly with context (KV cache scales with length), and attention compute grows quadratically without optimizations like Flash Attention or sparse attention. A model that "supports 128K" may run out of VRAM well before reaching that ceiling on consumer hardware.

For local inference, the practical question is rarely "does this model support long context" but "does my hardware have enough VRAM to actually use it." Use /will-it-run to compute the max context that fits on your specific hardware.

Say you're building a codebase-QA tool with Llama 3.1 8B and want to feed it a 200K-token repo dump. The model's 128K context window hard-caps you well before that — you'd need to chunk the repo, summarize sections, or switch to Llama 4 Scout's 10M window instead. But before celebrating a huge context number, check the hardware side: a 128K KV cache on an 8B model at FP16 can eat several extra GB of VRAM on top of the weights themselves. An operator running an RTX 3060 (12GB) often finds the card OOMs at 40-60K tokens of actual context long before hitting the model's advertised ceiling, forcing a smaller quant or a shorter working context just to keep the session stable.