Chain-of-Thought Prompting
Eliciting step-by-step reasoning.
Chain-of-Thought Prompting is a free AI Prompt Engineering lesson on CoddyKit — lesson 1 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the AI Prompt Engineering learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.
Reasoning as Token Budget
Chain-of-thought (CoT) prompting elicits intermediate reasoning steps before the final answer. The key insight: transformers do a fixed amount of computation per token, so allowing the model to emit reasoning tokens effectively grants it more serial compute to reach the answer.
Forcing an immediate answer caps the compute available for a hard problem; CoT removes that cap by spreading the computation across generated tokens.
# Direct: model must compute everything before the first token
DIRECT = 'Q: ' + q + '\nA:'
# CoT: model can use tokens as scratch space
COT = 'Q: ' + q + '\nA: Let us reason step by step.'Zero-Shot CoT
The famous trigger phrase Let us think step by step (Kojima et al., 2022) elicits reasoning with no examples. It works because instruction-tuned models have internalized the pattern that such a phrase precedes a worked solution.
Zero-shot CoT is cheap and surprisingly effective, but its reasoning quality is less controllable than few-shot CoT with curated exemplars.
def zero_shot_cot(question):
stage1 = llm('Q: ' + question + '\nA: Let us think step by step.')
# Extract the final answer in a second, constrained call
stage2 = llm(stage1 + '\nTherefore, the final answer is:')
return parse_answer(stage2)Few-Shot CoT
Few-shot CoT (Wei et al., 2022) provides exemplars that include the reasoning trace, not just the answer. This teaches both how to reason and the desired format, yielding more reliable and consistent chains than zero-shot.
Curate exemplars whose reasoning style, granularity, and length match what you want copied. Inconsistent exemplar reasoning produces inconsistent chains.
FS_COT = (
'Q: Roger has 5 balls, buys 2 cans of 3. How many balls?\n'
'A: 5 + 2*3 = 5 + 6 = 11. The answer is 11.\n\n'
'Q: ' + question + '\nA:'
)Separating Reasoning From the Answer
Always make the final answer machine-extractable: end the chain with a fixed marker (for example The answer is X or a JSON field). Parsing free-form chains is brittle.
For user-facing products, you may hide the chain and surface only the parsed answer, keeping the reasoning as an internal scratchpad.
import re
def extract(chain):
m = re.search(r'[Tt]he answer is\s*(.+?)[.\n]', chain)
if not m:
raise ValueError('no answer marker found')
return m.group(1).strip()Faithfulness Is Not Guaranteed
A critical caveat: the verbalized chain may not reflect the model's true computation. Studies show models can produce plausible-sounding reasoning that post-hoc rationalizes an answer driven by hidden biases (for example, the position of an option).
Do not treat CoT as a faithful explanation or an audit trail. It improves accuracy on many tasks but is not a window into the model's actual mechanism.
# Faithfulness probe: perturb an irrelevant cue (e.g., always make
# option A correct in the few-shot). If the chain still 'justifies'
# choosing A, the stated reasoning is unfaithful to the real cause.Decoding Settings for CoT
For a single deterministic chain, low temperature reduces variance. But CoT pairs powerfully with sampling: at moderate temperature you can draw multiple diverse chains and aggregate them (covered in self-consistency).
Avoid greedy decoding when you intend to sample multiple paths; greedy collapses diversity and defeats aggregation.
single = llm(COT, temperature=0.0) # one stable chain
paths = [llm(COT, temperature=0.7) for _ in range(10)] # diverseStructured and Tabular Reasoning
For complex problems, impose structure on the chain: numbered steps, a state table, or a plan-then-execute split. Structure reduces skipped steps and makes intermediate state verifiable.
Program-aided approaches go further, emitting executable code as the reasoning and offloading arithmetic to an interpreter, eliminating a whole class of computation errors.
PAL = (
'Solve by writing Python. Q: ' + q + '\n'
'A:\ndef solve():\n'
' # the model writes code here, then we exec() it\n'
)
# Offload arithmetic to a deterministic interpreterLength, Cost, and Latency
CoT multiplies output tokens, raising cost and latency. For easy items, the overhead buys nothing; for hard items it is essential. A pragmatic pattern is adaptive reasoning: trigger CoT only when a cheap difficulty estimate or a low-confidence direct answer warrants it.
On reasoning models with built-in thinking, you instead control a reasoning-effort budget rather than prompting the chain yourself.
def adaptive(question):
direct = llm('Q: ' + question + '\nA (answer only):', temperature=0)
if confidence(direct) > 0.85:
return direct
return zero_shot_cot(question) # escalate to reasoning only if neededCoT on Reasoning-Native Models
Modern reasoning models perform extended internal deliberation automatically. Stuffing them with verbose Let us think step by step instructions can be redundant or even counterproductive, interfering with their trained thinking process.
For these models, prefer concise task statements and explicit answer-format requirements, and tune the reasoning-effort parameter instead of hand-crafting chains.
# Reasoning-native model: let it think, just constrain the output
resp = reasoning_model(
prompt=question,
reasoning_effort='medium',
output_format='Final answer on the last line as: ANSWER=<x>'
)When CoT Backfires
CoT can hurt on tasks where deliberation overrides correct intuition, on simple pattern-matching, or when verbalization introduces errors the model would not make implicitly. It also widens the attack surface for prompt injection inside long chains.
Benchmark CoT against direct answering per task; never assume reasoning prompts are universally beneficial.
def should_use_cot(task_acc_cot, task_acc_direct, cot_cost_mult):
gain = task_acc_cot - task_acc_direct
# Only adopt CoT if accuracy gain justifies the token multiplier
return gain > MIN_GAIN and gain / cot_cost_mult > MIN_EFFICIENCYPutting CoT Into Production
Production CoT: curate consistent exemplars (or rely on the model's native reasoning), enforce a parseable answer marker, sample multiple chains for hard items, validate arithmetic via code execution where possible, and gate reasoning behind a difficulty estimate to control cost.
Log chains for offline analysis but never expose them as ground-truth explanations.
def production_solve(q):
if easy(q):
return extract(llm(DIRECT_PROMPT.format(q=q), temperature=0))
chains = [llm(FS_COT.format(q=q), temperature=0.7) for _ in range(8)]
return majority_vote([extract(c) for c in chains])Quick Check
Reason about why CoT helps and where it does not.
Recap
Key takeaways:
- CoT trades extra tokens for extra serial compute; it helps multi-step tasks, not trivial ones.
- Zero-shot CoT uses a trigger phrase; few-shot CoT supplies consistent reasoning exemplars.
- Always end with a machine-extractable answer marker; chains are not faithful explanations.
- Sample multiple chains for hard items, offload arithmetic to code, and gate CoT behind difficulty.
- On reasoning-native models, prefer concise prompts plus a reasoning-effort budget over verbose chain instructions.
Frequently asked questions
Is the “Chain-of-Thought Prompting” lesson free?
Yes — the full text of “Chain-of-Thought Prompting” is free to read here on the web, and the AI Prompt Engineering course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the AI Prompt Engineering course, upgrade to CoddyKit PRO.
What will I learn in “Chain-of-Thought Prompting”?
Eliciting step-by-step reasoning. You practise AI Prompt Engineering with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.
Do I need any experience to start AI Prompt Engineering?
No prior experience is required. AI Prompt Engineering on CoddyKit is structured for beginners through advanced learners; this is — lesson 1 of 4, so you can start here or from the beginning and move at your own pace.
How long does the “Chain-of-Thought Prompting” lesson take?
Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.
Can I write and run code in this AI Prompt Engineering lesson?
Yes. Every AI Prompt Engineering lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.
All lessons in this course
- Chain-of-Thought Prompting
- Self-Consistency Sampling
- Tree-of-Thought Exploration
- When Reasoning Prompts Help