Decoder
Definition pending
We've cataloged "Decoder" but haven't written a full definition yet. Definitions are hand-curated rather than auto-generated, so it takes time to cover every term.
Want this one prioritized? Email us and we'll bump it.
Practical example
The decoder generates tokens one at a time, each new token attending to all previous tokens. This is auto-regressive generation: previous tokens can't be changed, only new ones added. The critical operational implication: generation is sequential (can't parallelize across time), so total generation time = number of output tokens × time_per_token. A 500-token response × 20ms/token = 10 seconds.
Workflow example
Decoder throughput optimization: (1) use KV-cache to avoid recomputing attention for all previous tokens on each step (llama.cpp and vLLM do this by default), (2) set max_tokens limit per endpoint — 256 for chat, 2048 for document generation, (3) if users are getting bored waiting, reduce max_tokens and encourage concise responses in your system prompt, (4) batch multiple requests in a single forward pass (vLLM continuous batching).