Chain-of-Thought (CoT)

Chain-of-thought prompting is asking a model to show its reasoning step-by-step before giving the final answer. It dramatically improves accuracy on math, logic, and multi-step problems — frontier models gain 10-30 percentage points on hard benchmarks like GSM8K when CoT is enabled.

Two flavors: prompted CoT ("let's think step by step") works on any sufficiently large model, and trained CoT (also called "reasoning models") where the model is RL-trained to produce visible reasoning by default. DeepSeek R1, OpenAI's o1, QwQ, and Phi-4 Reasoning are reasoning models.

Tradeoffs: CoT outputs are 5-10× longer than direct answers, increasing latency and cost. For tasks that don't need reasoning (code completion, simple lookups), CoT just adds overhead. For tasks that do (math, planning, debugging), it's the difference between right and wrong.

Say you're building a local agent to solve inventory reconciliation math with Qwen 2.5 14B. Direct prompting ("what's the discrepancy?") gets the arithmetic wrong on multi-step cases fairly often. Switching to chain-of-thought — appending "think through each warehouse transfer step by step before totaling" — fixes most of those errors, at the cost of several times more output tokens and proportionally higher latency on your local GPU. If you're instead running QwQ-32B or a DeepSeek R1 distill, you get CoT by default without prompting for it, but you pay the same token tax on every response, even trivial ones. The practical tuning move: route simple lookups to a non-reasoning model like Llama 3.1 8B, and reserve the CoT-heavy model for multi-step planning calls, so you're not burning decode time on questions that don't need it.