Speculative Decoding

Speculative decoding speeds up LLM inference by using a small fast "draft" model to propose the next several tokens, then verifying them all in parallel with the large "target" model. When the draft is right (which it often is for routine tokens), you get 2-4× speedup; when wrong, you fall back to the standard autoregressive flow.

The key insight: verifying N tokens with the target model takes only one forward pass, while generating them autoregressively takes N. The draft model burns extra compute but saves more in reduced target-model passes.

For local AI: pair a 1B draft model with a 7B-70B target model from the same family (same tokenizer, similar training). llama.cpp supports this via --draft-model, vLLM via --speculative-model. Real speedups vary 1.5-3× depending on workload — code completion benefits most; creative writing benefits least.

An operator running Llama 3.1 70B Q4 locally for code review wants to speed up generation without dropping to a smaller, less capable model. Pairing it with Llama 3.2 1B as a draft model via llama.cpp's --draft-model flag lets the 1B propose 4-8 tokens at a time, which the 70B then verifies in a single forward pass instead of running autoregressively token by token. On predictable code (boilerplate, repeated patterns) the draft model's guesses land correctly most of the time, pushing effective throughput noticeably higher; on a creative-writing task with the same setup the speedup shrinks because the draft model's predictions diverge from the target model's choices more often, forcing more frequent fallback to standard decoding.