Attention Mechanism
Definition pending
We've cataloged "Attention Mechanism" but haven't written a full definition yet. Definitions are hand-curated rather than auto-generated, so it takes time to cover every term.
Want this one prioritized? Email us and we'll bump it.
Practical example
Attention is the mechanism that lets each token "look at" every other token in the sequence. Without attention, a model processing "the cat sat on the mat" can't connect "cat" with "sat" if they're 50 tokens apart. Attention solves this but has O(n²) complexity — double the context length = quadruple the compute. This is why 128K context models are expensive to run.
Workflow example
Profile attention cost: run ollama run --verbose and note the "eval time" per token. As your conversation history grows, per-token time increases quadratically. Set a max_context limit in your application: 4096 tokens for latency-sensitive, 8192 for quality-sensitive, 32768 for document analysis (and accept 5–10s latency). Monitor context_length vs p99_latency in Grafana — they should correlate.