Encoder-Decoder

An encoder-decoder is a neural network architecture that processes an input sequence through an encoder to produce a compressed representation, then a decoder generates an output sequence from that representation. In transformers, the encoder uses bidirectional self-attention to understand the full context of the input, while the decoder uses masked self-attention and cross-attention to predict tokens one by one. Operators encounter this in models like T5, BART, or Flan-T5, which are used for translation, summarization, or text-to-text tasks. The architecture requires more VRAM than decoder-only models because both encoder and decoder must be loaded, and the encoder processes the entire input before decoding begins, adding latency.

The encoder-decoder architecture was popularized by the original Transformer paper for machine translation. The encoder takes the entire input sequence and applies bidirectional self-attention, meaning each token can attend to all other tokens in the input. This produces a sequence of hidden states that represent the input context. The decoder then generates the output sequence autoregressively: at each step, it attends to previously generated tokens (via masked self-attention) and to the encoder's hidden states (via cross-attention). This cross-attention mechanism is what distinguishes encoder-decoder from decoder-only models. In practice, models like T5 are encoder-decoder and are often used for tasks where the output is a transformed version of the input (e.g., translation, summarization). However, for pure text generation, decoder-only models (like GPT) are more common because they are simpler and faster. Running an encoder-decoder model locally means loading both components into VRAM, which roughly doubles the memory footprint compared to a decoder-only model of similar parameter count. For example, T5-3B requires about 12 GB at FP16, while a decoder-only 3B model might fit in 6 GB.

Consider running T5-small (60M parameters) on an RTX 3060 (12 GB VRAM). At FP16, the model uses ~120 MB, so it fits easily. But T5-3B at FP16 uses ~12 GB, barely fitting on a 12 GB card with no room for context. Quantizing to Q4 reduces memory to ~3 GB, making it usable. In contrast, a decoder-only model like Llama 3.2 3B at Q4 uses ~1.8 GB, leaving more VRAM for context. Operators should expect encoder-decoder models to consume roughly double the memory of decoder-only models of the same parameter count.

In Hugging Face Transformers, loading an encoder-decoder model like T5 for summarization: from transformers import T5ForConditionalGeneration, T5Tokenizer; model = T5ForConditionalGeneration.from_pretrained('t5-small'). The tokenizer encodes the input, then model.generate() runs the encoder once and the decoder autoregressively. In llama.cpp, encoder-decoder support is limited; most local inference tools (Ollama, LM Studio) focus on decoder-only models. For T5, operators typically use Hugging Face or vLLM, which supports encoder-decoder models but requires more VRAM and yields lower tokens/sec than decoder-only models.