When Reasoning Prompts Help
Tasks that benefit and costs involved.
Reasoning Is Not Free
Reasoning prompts (CoT, self-consistency, ToT) trade tokens, latency, and money for accuracy. The central engineering question is not whether reasoning can help, but whether it helps this task enough to justify its cost.
Defaulting every prompt to step-by-step reasoning is a common and expensive anti-pattern that can also reduce quality on simple tasks.
def value_of_reasoning(acc_reason, acc_direct, token_mult, dollar_per_acc):
gain = acc_reason - acc_direct # accuracy delta
cost = token_mult # token/latency multiplier
return gain, gain / cost, gain * dollar_per_accTasks That Benefit Most
Reasoning prompts deliver the largest gains on multi-step, compositional problems: arithmetic word problems, symbolic and logical reasoning, multi-hop QA, planning, and code with non-trivial control flow.
The common thread is a problem that decomposes into sub-steps where intermediate computation reduces the chance of a wrong leap to the answer.
BENEFIT_HIGH = [
'multi_step_arithmetic',
'logical_deduction',
'multi_hop_qa',
'planning_and_scheduling',
'algorithmic_code',
]All lessons in this course
- Chain-of-Thought Prompting
- Self-Consistency Sampling
- Tree-of-Thought Exploration
- When Reasoning Prompts Help