Chain-of-Thought (CoT)

Why It Matters

Chain-of-thought unlocked a step-change in LLM capability. Before CoT, models struggled with multi-step problems like math, logic puzzles, and complex analysis. They’d jump straight to answers and often get them wrong. With CoT, the same models perform dramatically better by reasoning through problems step by step.

This isn’t just a prompting trick. CoT reveals how LLMs “think” and provides debugging insight when things go wrong. When a model shows its reasoning, you can identify exactly where it went off track. This transparency is invaluable for iterating on complex applications.

For AI engineers, CoT is a core technique for any task requiring reasoning. Agentic systems use CoT to plan actions. Analysis tools use CoT to work through complex documents. Math and coding assistants depend on CoT for accuracy. It’s foundational.

Implementation Basics

CoT is simple to implement but has important variations:

1. Zero-Shot CoT Just add “Let’s think step by step” to your prompt. Surprisingly effective, since models trained on reasoning data often engage CoT automatically with this trigger. Good for quick prototyping.

What's 23 × 47?
Let's think step by step.

2. Few-Shot CoT Provide examples that demonstrate the reasoning format you want. The model learns to mirror your reasoning style. This gives you more control over the output structure.

Q: What's 15 × 12?
A: Let's break this down:
- 15 × 10 = 150
- 15 × 2 = 30
- 150 + 30 = 180
The answer is 180.

Q: What's 23 × 47?

3. Self-Consistency Generate multiple CoT paths and take the majority answer. Different reasoning paths may reach different conclusions, and consensus is more reliable. Adds cost but improves accuracy for critical tasks.

4. Tree of Thoughts Explore multiple reasoning branches, evaluate each, and pursue the most promising. More sophisticated than linear CoT. Good for problems with multiple valid approaches.

Practical Considerations

• CoT increases token usage, so you’re paying for reasoning tokens. Budget accordingly.
• Not all tasks benefit. Simple factual lookups don’t need reasoning steps.
• Extract the final answer programmatically. Don’t rely on users parsing reasoning chains.
• For reasoning-heavy models like o1, CoT is built in. You don’t need to prompt for it explicitly.

CoT is powerful but not magic. It helps models with problems that benefit from step-by-step reasoning. For tasks that require external information or real-time data, CoT alone won’t help. You need grounding and tools.