Self-Consistency
Definition
Self-consistency is a prompting technique that generates multiple reasoning paths for the same question and selects the most frequent answer, improving accuracy on complex reasoning tasks.
Why It Matters
Single LLM responses can be inconsistent or contain errors. Self-consistency leverages the fact that correct reasoning paths tend to converge on the same answer, while errors are more random. By sampling multiple responses and majority voting, you get more reliable results.
How It Works
- Generate N responses with chain-of-thought prompting (typically with higher temperature)
- Extract the final answer from each response
- Return the most frequently occurring answer
The reasoning paths may differ, but correct answers tend to appear more often than incorrect ones.
When to Use
Use self-consistency for: high-stakes decisions where accuracy matters, math and logical reasoning problems, tasks where single responses are unreliable, and situations where you can afford the extra API calls. The technique multiplies your token costs by N, so balance accuracy gains against cost.
Source
Self-consistency sampling from diverse reasoning paths and marginalizing over them significantly improves chain-of-thought prompting performance.
https://arxiv.org/abs/2203.11171