Chain of Thought Prompting Explained

How Chain of Thought Works

The idea behind chain-of-thought is simple and powerful. Language models are next-token predictors. When you ask a complex question and expect a direct answer, the model must compress all of its reasoning into the probability distribution for the very next token. This works for simple factual lookups ("What is the capital of France?") where the answer is directly represented in the model's weights, but it fails on problems that require multiple reasoning steps because the model cannot do intermediate computation between the question and the answer token.

Chain-of-thought fixes this by giving the model space to compute. When you ask "think step by step" or "show your reasoning," the model generates intermediate reasoning tokens before the final answer. Each reasoning token is informed by the tokens before it, which means the model can perform sequential computation across many tokens rather than trying to compress everything into a single prediction. Each step in the chain gives the model a new "working memory" position to build on, allowing it to carry information through a complex problem the same way you would carry intermediate results through a math problem on paper.

The original chain-of-thought paper by Wei et al. (2022) demonstrated this on arithmetic, commonsense reasoning, and symbolic reasoning tasks. The results were striking: GSM8K math accuracy jumped from 17.9% to 58.1% with CoT on PaLM 540B. The improvement was consistent across model sizes, but notably larger models benefited more than smaller ones, which led to the observation that chain-of-thought is an "emergent ability" that becomes reliably effective above a certain model scale.

Two Ways to Trigger Chain of Thought

Zero-shot CoT is the simplest approach: append "Let's think step by step" or "Think through this carefully, showing your work" to your prompt. This single addition triggers the model to generate reasoning steps before its answer. Zero-shot CoT is remarkably effective given its simplicity. It works because the model's training data contains many examples of step-by-step reasoning (textbooks, tutorials, problem-solving forums), and the instruction activates those patterns. The specific phrasing matters less than communicating the intent: any instruction that tells the model to reason before answering will work.

Few-shot CoT includes examples of problems with step-by-step solutions before presenting the actual problem. You show the model two or three examples where each example contains the question, the reasoning steps, and the final answer. The model then follows the demonstrated reasoning pattern for the new question. Few-shot CoT is more reliable than zero-shot CoT because the examples constrain not just whether the model reasons, but how it reasons: the level of detail, the types of steps, the format of the reasoning, and the way the final answer is derived from the steps.

For production applications, few-shot CoT is generally better because it gives you more control over the reasoning format and quality. For quick prototyping and experimentation, zero-shot CoT is faster to implement and often good enough to determine whether CoT will help your specific task.

When Chain of Thought Helps

CoT produces the largest improvements on tasks that require multi-step reasoning where the final answer depends on correctly executing a sequence of intermediate steps. These include:

Mathematical reasoning is the classic CoT use case. Word problems that require setting up equations, performing calculations, and combining results benefit enormously. Without CoT, models frequently get the computation wrong or skip steps. With CoT, accuracy typically doubles or more on standard math benchmarks.

Logical deduction tasks where the model must combine multiple facts to reach a conclusion. "If all managers report to the VP, and Alice is a manager, and the VP reports to the CEO, who does Alice ultimately report to?" requires tracking relationships through a chain, which CoT handles well.

Multi-factor analysis where the model needs to weigh several considerations. "Which cloud provider should I choose for this workload?" requires analyzing cost, performance, geographic availability, compliance requirements, and existing infrastructure, then synthesizing a recommendation. Without CoT, the model often gives a superficial answer that misses important factors. With CoT, it works through each factor systematically.

Code debugging where the model needs to trace execution, identify the discrepancy between expected and actual behavior, and propose a fix. The step-by-step trace is literally the reasoning chain.

Complex classification where the correct category depends on evaluating multiple criteria. "Is this customer email a complaint, a feature request, or a support question?" is simple. "Classify this customer communication as one of 15 intent categories, considering tone, urgency, customer tier, and whether it references a previous interaction" is complex and benefits from CoT.

When Chain of Thought Does Not Help

CoT is not universally beneficial. It adds tokens (which cost money and add latency) and can actually hurt performance on certain task types.

Simple factual lookups do not benefit from CoT. "What year was Python created?" does not need intermediate reasoning. Adding CoT to this kind of task wastes tokens and can introduce errors if the model's "reasoning" introduces wrong intermediate steps.

Tasks where the model already performs near perfectly gain nothing from CoT. If your prompt achieves 99% accuracy on zero-shot, adding CoT adds cost without improving quality.

Creative generation tasks (writing stories, generating marketing copy, brainstorming ideas) are often hurt by CoT because the reasoning step can make the output feel mechanical and over-analyzed. Creative tasks benefit from the model generating fluidly rather than analytically.

Very small models do not benefit from CoT reliably. The original research showed that CoT is an emergent capability that appears above a certain scale threshold. Models below roughly 10 billion parameters tend to produce low-quality reasoning chains that do not improve (and sometimes degrade) the final answer. With frontier models in 2026, this is rarely an issue, but it matters if you are using smaller open-source models for cost reasons.

Extended Thinking and Built-In CoT

Modern frontier models have incorporated chain-of-thought directly into their architecture rather than requiring it as a prompting technique. Anthropic's Claude models offer an "extended thinking" mode where the model generates a long internal reasoning chain (visible to the developer but presented separately from the final response) before producing the answer. OpenAI's o-series models (o1, o3) are specifically trained to perform chain-of-thought reasoning internally, producing higher-quality reasoning than prompt-triggered CoT.

These built-in reasoning capabilities change the prompt engineering calculus. For models with native extended thinking, you may not need to prompt for CoT at all, the model does it automatically when the task warrants it. For the o-series models, adding "think step by step" to the prompt is actively discouraged by OpenAI because the model's internal reasoning process is already more sophisticated than what prompt-triggered CoT produces.

The practical guidance is: check whether your model supports a native reasoning mode. If it does, use that mode instead of prompt-based CoT, because native reasoning is typically higher quality and better calibrated for cost. If you are using a model without native reasoning (many open-source models, older API versions), prompt-based CoT remains the best technique for improving accuracy on complex tasks.

CoT and Cost

Chain-of-thought has a direct cost implication. The reasoning tokens count toward your output token usage, which means CoT responses are more expensive than direct answers. A task that produces a 50-token answer without CoT might produce a 500-token response with CoT (450 tokens of reasoning + 50 tokens of answer). At typical API pricing, this is a 10x increase in output cost for that call.

Whether this cost is justified depends entirely on whether the accuracy improvement matters for your use case. If CoT moves accuracy from 60% to 90%, the cost increase is almost certainly worth it because you avoid the downstream cost of wrong answers (human review, user dissatisfaction, incorrect actions). If CoT moves accuracy from 95% to 97%, the cost increase may not be justified unless the remaining errors are extremely expensive.

One cost-effective pattern is to use CoT selectively. Route simple queries to a standard prompt (no CoT) and complex queries to a CoT prompt. You can detect complexity with heuristics (question length, presence of numbers or comparisons, keywords that suggest multi-step reasoning) or with a cheap classifier model that decides whether CoT is needed. This gives you CoT accuracy where it matters without paying for reasoning tokens on simple queries. The cost optimization guide on routing by complexity covers this pattern in detail.

Implementing Chain of Thought

For zero-shot CoT, simply add an instruction to your prompt. Effective phrasings include "Think through this step by step before answering," "Show your reasoning, then give your final answer," and "Break this problem down into steps." All achieve the same effect. If you want the reasoning and answer separated, add "Put your final answer after 'ANSWER:'" so you can parse the answer programmatically.

For few-shot CoT, include two to four examples that demonstrate the reasoning pattern you expect. Each example should have the question, a clear reasoning chain that uses the level of detail you want, and the final answer clearly marked. The quality of your examples determines the quality of the model's reasoning on new inputs, so invest time in making your examples clear, correct, and representative of the reasoning difficulty level you expect.

For production systems, consider pairing CoT with structured output. Have the model produce its reasoning in a "thinking" field and its answer in a "result" field of a JSON schema. This gives you the accuracy benefit of CoT, a parseable output, and a debugging trace you can log for quality analysis.