Decoder-Only Transformer

Decoder-only is the architecture of GPT, Llama, Qwen, Mistral, DeepSeek, and almost every modern open-weight LLM. The model is a stack of transformer decoder blocks (causal self-attention only, no cross-attention to a separate encoder), trained autoregressively on next-token prediction.

The "encoder" doesn't exist as a separate component — input prompt tokens and generated tokens go through the same blocks. The causal mask prevents tokens from attending to future positions.

Decoder-only won the architecture war for generative LLMs because it scales cleanly with parameters and data, and a single model can do everything (chat, code, reasoning, summarization) with prompting alone.

An engineer explaining why the same Llama 3.1 8B checkpoint can handle chat, code completion, and summarization without separate fine-tuned heads points to its decoder-only architecture: there's no dedicated encoder module to swap out per task, just one stack of causal-attention blocks that treats the system prompt, user message, and generated tokens as one continuous sequence. This is also why prompt engineering works at all — everything the model conditions on, including few-shot examples, has to be serialized into that same token stream rather than passed to a separate encoder. It's also why KV-cache reuse is straightforward in engines like llama.cpp and vLLM: since every token, prompt or generated, flows through identical blocks with a causal mask, the cache from a shared prefix (like a long system prompt) can be computed once and reused across requests.