Chain-of-Thought and Step-by-Step Reasoning
Use chain-of-thought prompts to guide the model through complex problems step by step, dramatically improving accuracy on math, logic, and multi-hop reasoning tasks.
Chain-of-Thought and Step-by-Step Reasoning is a free AI Engineering Academy lesson on CoddyKit — lesson 2 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the AI Engineering Academy learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.
What Is Chain-of-Thought Prompting?
Chain-of-Thought (CoT) prompting is a technique where you ask the model to show its reasoning step by step before giving a final answer. Instead of jumping straight to the conclusion, the model produces intermediate reasoning steps that lead to the answer. This simple change dramatically improves accuracy on complex tasks involving math, logic, and multi-step reasoning.
The technique was introduced in a 2022 Google Brain paper that showed CoT prompting could enable large language models to solve math word problems that they completely failed at with direct prompting. The key insight is that generating intermediate steps forces the model to allocate more computation to hard problems.
The Magic of 'Let's Think Step by Step'
The simplest form of chain-of-thought prompting is adding the phrase 'Let's think step by step' to your prompt. This zero-shot CoT trigger was discovered by researchers who found that this single phrase caused models to spontaneously produce reasoning chains even without few-shot examples.
import openai
client = openai.OpenAI()
# Without CoT - the model often gets multi-step math wrong
direct_prompt = 'A store has 48 apples. They sell 3/4 of them, then receive a new shipment of 20. How many apples do they have?'
# With zero-shot CoT
cot_prompt = direct_prompt + "\n\nLet's think step by step."
response = client.chat.completions.create(
model='gpt-4o-mini',
messages=[{'role': 'user', 'content': cot_prompt}]
)
print(response.choices[0].message.content)Why CoT Works: Computation Allocation
LLMs generate text one token at a time, and each token is produced with a fixed amount of computation (one forward pass of the model). When you ask for a direct answer, the model must arrive at the correct answer in a single inference step, which is often too little computation for complex problems.
Chain-of-thought gives the model token budget to work with the problem. Each reasoning step is a token, and generating those tokens is equivalent to additional computation. This is why CoT works even on tasks where the model clearly knows all the individual facts — the bottleneck is not knowledge but the ability to combine facts over multiple reasoning steps in a single pass.
Few-Shot Chain-of-Thought
For maximum reliability, combine few-shot examples with chain-of-thought reasoning. Each example shows the full reasoning chain that leads to the correct answer. This teaches the model both the reasoning style you want and the correct format for the final answer.
import openai
client = openai.OpenAI()
prompt = '''Solve these math problems step by step.
Problem: A recipe needs 2.5 cups of flour for 12 cookies. How much flour for 30 cookies?
Reasoning: First, find flour per cookie: 2.5 / 12 = 0.208 cups each.
Then multiply by 30: 0.208 * 30 = 6.25 cups.
Answer: 6.25 cups
Problem: A train travels 180 km in 2.5 hours. At the same speed, how long to travel 270 km?
Reasoning: First, find speed: 180 / 2.5 = 72 km/h.
Time = distance / speed = 270 / 72 = 3.75 hours.
Answer: 3.75 hours
Problem: A shirt costs $45 after a 25% discount. What was the original price?
'''
response = client.chat.completions.create(
model='gpt-4o-mini',
messages=[{'role': 'user', 'content': prompt}]
)
print(response.choices[0].message.content)CoT for Multi-Hop Reasoning
Chain-of-thought is not limited to math. It dramatically improves performance on multi-hop reasoning tasks where you must connect several facts to answer a question. For example, answering 'Who was the president of the US when the iPhone was invented?' requires knowing when the iPhone was invented, then looking up who was president at that time — two separate knowledge lookups chained together.
Without CoT, the model must short-circuit this chain in a single step and often produces the wrong answer. With CoT, it explicitly retrieves each fact in a reasoning step, dramatically improving accuracy on these tasks.
Structuring the Output with CoT
In production applications, you often want the reasoning to be separate from the final answer so you can parse the answer programmatically. A common pattern is to instruct the model to wrap reasoning in tags and the final answer in its own section, making it easy to extract the answer without parsing the full reasoning text.
import openai
import re
client = openai.OpenAI()
prompt = '''Analyze the following customer complaint and determine the priority level.
First think through the issue, then state your final classification.
Format your response exactly as:
<reasoning>
Your step-by-step analysis here.
</reasoning>
<priority>HIGH | MEDIUM | LOW</priority>
Complaint: "The checkout page crashes every time I try to buy something. I have tried three times and lost my cart each time."
'''
response = client.chat.completions.create(
model='gpt-4o-mini',
messages=[{'role': 'user', 'content': prompt}]
)
content = response.choices[0].message.content
priority = re.search(r'<priority>(.*?)</priority>', content).group(1)
print('Priority:', priority)Self-Consistency: Vote Across Multiple Chains
Self-consistency is an extension of CoT that generates multiple independent reasoning chains for the same problem using a high temperature, then takes a majority vote over the final answers. The intuition is that correct reasoning paths tend to converge on the same answer, while incorrect paths tend to diverge.
This significantly improves accuracy on math and commonsense reasoning benchmarks at the cost of additional API calls. It is most useful when accuracy matters more than latency or cost, such as in scientific or financial applications where a wrong answer has serious consequences.
import openai
from collections import Counter
client = openai.OpenAI()
def self_consistent_answer(question, n=5):
prompt = question + '\n\nLet\'s think step by step. State your final answer as: Answer: <value>'
answers = []
for _ in range(n):
resp = client.chat.completions.create(
model='gpt-4o-mini',
messages=[{'role': 'user', 'content': prompt}],
temperature=0.7
)
text = resp.choices[0].message.content
# Extract final answer
if 'Answer:' in text:
ans = text.split('Answer:')[-1].strip().split('\n')[0]
answers.append(ans)
# Majority vote
return Counter(answers).most_common(1)[0]
result, count = self_consistent_answer('What is 15% of 240?')
print(f'Answer: {result} (agreed by {count}/5 chains)')Tree of Thought: Exploring Multiple Paths
Tree of Thought (ToT) extends chain-of-thought by exploring multiple reasoning branches simultaneously and evaluating which branch is most promising before continuing. Think of it like a search algorithm: rather than committing to one reasoning path, you generate several partial thoughts, score them, and expand the most promising ones.
This is particularly useful for planning and open-ended problems where the optimal reasoning path is not immediately clear. ToT requires significantly more API calls than basic CoT but can solve problems that linear CoT fails on, such as multi-step puzzles and creative problem-solving tasks.
When Not to Use Chain-of-Thought
CoT adds latency and token cost to every request. It is not always the right choice. Avoid CoT when:
- The task is simple and the model reliably gets it right without reasoning (e.g., single-step classification)
- You need very low latency and cannot afford extra tokens
- The reasoning itself would confuse downstream parsing (though structured output tags solve this)
- You are using a very small model that does not benefit from reasoning steps
CoT provides the most value on complex, multi-step tasks where the model demonstrably fails without it. Always measure before adding it to a production prompt.
CoT for Code and Debugging
Chain-of-thought is highly effective for code generation and debugging tasks. Asking the model to explain its plan before writing code — 'First explain your approach, then write the code' — produces more correct implementations because the planning step catches logical errors before they are encoded in the implementation.
For debugging, prompting the model to 'trace through the code step by step and identify where the output diverges from the expected value' is far more effective than just asking 'what is wrong with this code?' The step-by-step trace acts as a simulation of code execution within the model's reasoning.
Combining CoT with System Prompts
You can encode chain-of-thought instructions permanently in your system prompt so that every user message benefits from step-by-step reasoning without users needing to request it explicitly. This is the standard pattern for production applications where you want reliable CoT behavior across all queries.
import openai
client = openai.OpenAI()
response = client.chat.completions.create(
model='gpt-4o-mini',
messages=[
{
'role': 'system',
'content': ('You are a precise analytical assistant. '
'For any question involving calculation, logic, or multi-step reasoning, '
'always think through the problem step by step before giving your answer. '
'Show your reasoning clearly before stating the final answer.')
},
{
'role': 'user',
'content': 'If I invest $5000 at 7% annual interest compounded annually, how much will I have after 10 years?'
}
]
)
print(response.choices[0].message.content)Quick Check
Test your understanding of AI Engineering concepts from this lesson.
Lesson Recap
In this lesson you learned: chain-of-thought prompting improves accuracy by making the model generate explicit reasoning steps before answering, self-consistency runs multiple chains and takes a majority vote for more reliable results, and CoT is most valuable for math, multi-hop reasoning, and debugging tasks, but adds latency and cost. Next up we explore how to write effective system prompts and define AI personas.
Frequently asked questions
Is the “Chain-of-Thought and Step-by-Step Reasoning” lesson free?
Yes — the full text of “Chain-of-Thought and Step-by-Step Reasoning” is free to read here on the web, and the AI Engineering Academy course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the AI Engineering Academy course, upgrade to CoddyKit PRO.
What will I learn in “Chain-of-Thought and Step-by-Step Reasoning”?
Use chain-of-thought prompts to guide the model through complex problems step by step, dramatically improving accuracy on math, logic, and multi-hop reasoning tasks. You practise AI Engineering Academy with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.
Do I need any experience to start AI Engineering Academy?
No prior experience is required. AI Engineering Academy on CoddyKit is structured for beginners through advanced learners; this is — lesson 2 of 4, so you can start here or from the beginning and move at your own pace.
How long does the “Chain-of-Thought and Step-by-Step Reasoning” lesson take?
Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.
Can I write and run code in this AI Engineering Academy lesson?
Yes. Every AI Engineering Academy lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.
All lessons in this course
- Zero-Shot and Few-Shot Prompting
- Chain-of-Thought and Step-by-Step Reasoning
- System Prompts and Persona Definition
- Prompt Iteration and Debugging