Self-Consistency: Sampling Multiple Reasoning Paths
Why a Single Reasoning Chain Is Not Enough
Chain-of-thought prompting dramatically improves accuracy on complex reasoning tasks, but it has a fundamental vulnerability: a single reasoning chain can go wrong at any step, and once the model makes an error mid-chain, every subsequent step builds on that mistake. The final answer is wrong even though most of the reasoning was correct. This is the same problem humans face when solving a math problem, one arithmetic error early in the process cascades through the rest of the work.
The variability exists because language models are probabilistic. With any temperature above zero, the same prompt produces different outputs on different runs. Sometimes the model takes a reasoning shortcut that works. Sometimes it takes a shortcut that fails. Sometimes it uses a different solution strategy entirely. The quality of any single output is essentially a random draw from the model's distribution of possible reasoning chains for that problem.
Self-consistency exploits this variability rather than fighting it. By sampling multiple reasoning chains, you get a distribution of answers. If the model "knows" the right answer (meaning the correct reasoning appears in its distribution more often than any specific wrong answer), then majority voting across samples will surface the correct answer even when individual chains contain errors.
How Self-Consistency Works
The technique has three steps. First, prompt the model with a chain-of-thought instruction (either zero-shot "think step by step" or few-shot with examples). Second, generate multiple completions for the same prompt, typically between 5 and 20, using a temperature above zero (0.5 to 0.7 works well for most tasks). Third, extract the final answer from each completion and take the majority vote.
The key insight is that you only vote on the final answer, not on the reasoning steps. Different chains may arrive at the same answer through completely different reasoning paths, and that is fine. A math problem might be solved by algebra in one chain, by working backwards in another, and by estimation followed by refinement in a third. If all three reach the same numerical answer, that agreement is strong evidence of correctness regardless of method.
Wang et al. introduced self-consistency in their 2022 paper and demonstrated consistent improvements over standard chain-of-thought across multiple benchmarks. On GSM8K (grade school math), self-consistency with CoT improved accuracy from 58.1% to 74.4% on PaLM 540B, a 16-point gain from the sampling and voting alone. On ARC-Challenge (science reasoning), it improved from 85.2% to 88.7%. These gains came purely from generating more samples and voting, with no changes to the model or the prompt.
Implementing Self-Consistency
Implementation is straightforward with any model API that supports temperature control and multiple completions. Generate N completions of the same prompt, parse the final answer from each, and count votes.
from anthropic import Anthropic
from collections import Counter
client = Anthropic()
def self_consistent_answer(prompt, n_samples=10, temperature=0.7):
answers = []
for _ in range(n_samples):
response = client.messages.create(
model="claude-sonnet-4-20250514",
max_tokens=2000,
temperature=temperature,
messages=[{"role": "user", "content": prompt}]
)
# Extract the final answer after "ANSWER:" marker
text = response.content[0].text
if "ANSWER:" in text:
answer = text.split("ANSWER:")[-1].strip()
answers.append(answer)
if not answers:
return None, 0.0
vote_counts = Counter(answers)
best_answer, best_count = vote_counts.most_common(1)[0]
confidence = best_count / len(answers)
return best_answer, confidenceThe prompt should include an explicit answer marker so you can parse the final answer reliably. Something like "After your reasoning, write your final answer on a new line starting with ANSWER:" works well. Without a clear delimiter, parsing becomes fragile because the model may phrase its conclusion differently across samples.
For structured output, you can use JSON mode and extract a specific field from each response. This makes parsing deterministic and eliminates the need for text-based answer extraction. If your model supports tool use or function calling, define a tool schema for the answer and let the API enforce the format.
Choosing the Number of Samples
More samples improve accuracy but cost more. The relationship follows diminishing returns: the jump from 1 to 5 samples captures most of the improvement, 5 to 10 adds a meaningful but smaller gain, and 10 to 20 adds very little. The original paper found that 40 samples was sufficient to plateau on most benchmarks, but practical deployments rarely need more than 10.
The optimal number depends on how difficult the task is and how much you are willing to spend. For tasks where the model's base accuracy is high (above 85%), 3 to 5 samples are usually enough because correct answers already dominate the distribution. For tasks where base accuracy is lower (50-70%), 7 to 15 samples provide more reliable voting because wrong answers are more spread out.
You can also use adaptive sampling: start with 3 samples, check if they all agree, and only generate more samples if there is disagreement. This gives you the cost efficiency of few samples on easy questions and the accuracy of many samples on hard questions. In practice, most questions are easy and reach consensus quickly, so adaptive sampling reduces average cost significantly.
Temperature and Diversity
Temperature controls how diverse the sampled reasoning chains are. At temperature 0, every sample produces the same output, which makes self-consistency pointless because you are voting on N identical answers. At temperature 1.0, outputs are highly diverse but individual chains may be lower quality because the model makes more random token choices.
The sweet spot for self-consistency is typically 0.5 to 0.7. This provides enough diversity that different reasoning strategies emerge while keeping individual chains coherent and high-quality. If your votes are too uniform (every sample gives the same answer), increase temperature. If individual chains are too noisy (many clearly wrong reasoning steps), decrease temperature.
Top-p (nucleus sampling) provides an alternative way to control diversity. Setting top-p to 0.9 with temperature 0.7 gives slightly more controlled diversity than temperature alone. In practice, the specific sampling parameters matter less than ensuring you have enough diversity for voting to add value without so much diversity that individual chain quality degrades.
When Self-Consistency Helps Most
Mathematical reasoning is the strongest use case. Math problems have a single correct numerical answer, and the model can reach that answer through multiple valid solution paths. Voting across paths is highly effective because correct answers cluster while wrong answers scatter (there are many ways to get the wrong number, but only one way to get the right one).
Logical reasoning and constraint satisfaction problems benefit similarly. If a puzzle has one valid solution, multiple reasoning attempts will converge on it while incorrect attempts will diverge.
Multiple-choice and classification tasks are natural fits because the answer space is finite and small. Voting across 5 responses to a 4-choice question is straightforward and effective.
Code generation benefits from a variant of self-consistency where you generate multiple solutions and test each against provided test cases. The solution that passes the most tests wins. This is not pure majority voting on text, but it uses the same principle of sampling diverse attempts and selecting the best.
When Self-Consistency Does Not Help
Open-ended generation has no single correct answer, so majority voting is meaningless. You cannot vote on which creative story is "correct." For generation tasks, techniques like best-of-N sampling (generate N outputs and pick the highest quality one using a separate evaluation) are more appropriate than self-consistency.
Tasks where the model consistently gets the wrong answer will not be fixed by self-consistency. If the model's base accuracy is 10%, voting across samples just selects the most popular wrong answer. Self-consistency amplifies existing capability, it does not create capability that is not there. If the model cannot solve the problem at all, you need a better prompt, more context, or a more capable model.
Tasks where cost is the primary constraint may not justify the N-times cost increase. If you are making 100,000 API calls per day and each call already costs $0.01, adding 10x self-consistency samples turns a $1,000/day cost into $10,000/day. The accuracy improvement needs to justify that increase, and for many applications it does not.
Self-Consistency vs Other Techniques
Self-consistency is complementary to other prompting techniques, not a replacement. It works on top of chain-of-thought, you still need CoT as the base technique because self-consistency votes on the outputs of CoT chains. Without CoT, the model produces direct answers with less variability, and voting adds less value.
Compared to tree-of-thought, self-consistency is simpler to implement but less structured. Tree-of-thought explicitly explores and evaluates different reasoning branches within a single generation, while self-consistency generates independent chains and votes post-hoc. Tree-of-thought can be more sample-efficient (it discovers the correct path with fewer total tokens) but requires more complex orchestration.
Compared to simply increasing temperature, self-consistency adds the voting step that filters out outlier responses. Higher temperature alone gives more diversity but does not select the best answer from that diversity. Self-consistency adds the selection mechanism that makes diversity productive.
In practice, the prompting techniques you stack depend on how much accuracy matters relative to cost. For the highest accuracy, combine few-shot CoT as the base prompt, self-consistency with 10 samples for voting, and structured output for reliable parsing. For reasonable accuracy at lower cost, use zero-shot CoT with 3 samples. For minimum cost, use a single zero-shot prompt. The prompt optimization guide covers how to find the right balance for your specific application.
Production Considerations
Self-consistency in production requires parallel API calls. Generating 10 sequential completions adds 10x latency, which is unacceptable for real-time applications. Make the calls concurrently to reduce wall-clock time to approximately the duration of a single call plus overhead. Most API client libraries support async/concurrent requests.
Rate limits become a consideration. If your API plan allows 100 requests per minute and you use 10-sample self-consistency, your effective throughput drops to 10 questions per minute. Plan your rate limits around the multiplied request volume.
Caching helps significantly. If the same question appears multiple times (common in classification workloads), cache the voted answer so subsequent identical questions return instantly without generating new samples. The LLM caching guide covers exact-match and semantic caching strategies that work well with self-consistency.
Confidence scoring is a natural byproduct. The vote margin, how many samples agree on the winning answer, directly indicates confidence. If 9 of 10 samples agree, confidence is high. If the vote is 4-3-2-1, confidence is low and you might escalate to a human reviewer or use a more capable model for that specific question. This confidence signal is more reliable than asking the model to self-report its confidence, which models are notoriously poor at.
Self-consistency improves accuracy by generating multiple reasoning chains and voting on the most common answer. It is most effective on tasks with definitive correct answers (math, logic, classification) where the model already has moderate accuracy. The technique adds N-times cost, so use it selectively on questions where accuracy justifies the expense, and consider adaptive sampling to reduce cost on easy questions.