Flash Attention

Flash Attention is a memory-efficient implementation of the attention mechanism that reduces memory usage from O(n²) to O(n) for sequence length n, while also being faster on modern GPUs.

The key insight: standard attention materializes the full N×N attention matrix in HBM (the slow GPU memory). Flash Attention tiles the computation, keeps intermediate results in fast SRAM, and never writes the full matrix to HBM. Same math, much less memory bandwidth pressure.

For local inference this matters most at long context. Without Flash Attention, a 32K-context generation on a 7B model can OOM a 24GB card; with Flash Attention, it fits with room to spare. llama.cpp added support in late 2024; vLLM and ExLlamaV2 use it by default.

An operator running Qwen 3 8B on a 24GB card wants to push context to 32K for a document-summarization workload. Without Flash Attention, the attention matrix alone at that length can consume several GB of scratch memory on top of the KV cache and model weights, triggering an OOM crash mid-generation. Enabling it in llama.cpp (the default in recent builds) or confirming vLLM's kernel path is active drops that scratch memory to a fraction of the naive implementation's footprint, letting the same card handle the full 32K context with headroom left for a larger batch size. It's one of the first things to check when a context-length OOM happens on hardware that should, on paper, have enough VRAM.