Token Budgets Per Step
Cap input and output tokens per node so a runaway loop can't bankrupt you.
Token Budgets Per Step is a free AI Agents lesson on CoddyKit — lesson 1 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the AI Agents learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.
Tokens Are Money
Every token you send or receive costs money and time. Production agents track token usage at every step and enforce caps.
Cap max_tokens Everywhere
Always set max_tokens on every call. Without it, a buggy prompt can produce 10,000 token responses:
response = client.chat.completions.create(
model='gpt-4o-mini',
messages=messages,
max_tokens=1024, # cap per response
)Per-Step Caps in an Agent Loop
Different steps need different budgets:
STEP_BUDGETS = {
'planner': 512,
'classifier': 64,
'final_synthesis': 2048,
'tool_call': 256
}
response = client.chat.completions.create(
model=model,
messages=messages,
max_tokens=STEP_BUDGETS[step_name]
)Trim Input Tokens
Input tokens cost too. Trim aggressively:
import tiktoken
enc = tiktoken.encoding_for_model('gpt-4o-mini')
def trim_to_budget(text, max_tokens=4000):
toks = enc.encode(text)
if len(toks) > max_tokens:
return enc.decode(toks[:max_tokens])
return textPer-Run Total Budget
Sum input + output across all steps; abort if exceeded:
MAX_TOKENS_PER_RUN = 50_000
total = 0
for step in agent_steps:
r = call_step(step)
total += r.usage.total_tokens
if total > MAX_TOKENS_PER_RUN:
return 'Budget exhausted; partial result'Cost Caps in Dollars
Sometimes budgeting in dollars is more meaningful:
PRICES = {'gpt-4o-mini': {'in': 0.15e-6, 'out': 0.60e-6}}
for step in agent_steps:
r = call_step(step)
cost = r.usage.prompt_tokens * PRICES[model]['in'] + r.usage.completion_tokens * PRICES[model]['out']
total_cost += cost
if total_cost > MAX_COST_PER_RUN:
breakAdaptive Budgets
Spend more tokens on the hardest steps:
if step_difficulty == 'easy':
max_tokens = 256
elif step_difficulty == 'hard':
max_tokens = 2048Output Length Estimation
Sometimes ask the model how long the answer should be:
preview = llm.invoke(f'How many tokens (approx) does {question} need to answer?').content
budget = parse_int(preview, default=512)
final = llm.invoke(question, max_tokens=budget)Truncate Tool Outputs
Big tool outputs (HTML, logs) bloat context. Truncate before sending to the model:
tool_result = run_tool(...)
text = json.dumps(tool_result)
if len(text) > 4000:
text = text[:4000] + '\n...[truncated]'Compress History
For long conversations, summarise older turns to save tokens:
if total_message_tokens(messages) > 8000:
messages = compact(messages)Output Format Affects Tokens
JSON is verbose. For short structured replies, consider:
- Single tool-call argument
- Comma-separated list
- Compact custom format
Track Token Distribution
Per-step histograms reveal which steps to optimise:
metrics.observe('step.planner.tokens', r.usage.total_tokens, tags={'agent': 'qa'})Why max_tokens?
Why set max_tokens on every LLM call?
Recap
Cap max_tokens on every call. Trim inputs. Per-step budgets. Per-run total caps. Track distributions. Compress history.
Frequently asked questions
Is the “Token Budgets Per Step” lesson free?
Yes — the full text of “Token Budgets Per Step” is free to read here on the web, and the AI Agents course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the AI Agents course, upgrade to CoddyKit PRO.
What will I learn in “Token Budgets Per Step”?
Cap input and output tokens per node so a runaway loop can't bankrupt you. You practise AI Agents with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.
Do I need any experience to start AI Agents?
No prior experience is required. AI Agents on CoddyKit is structured for beginners through advanced learners; this is — lesson 1 of 4, so you can start here or from the beginning and move at your own pace.
How long does the “Token Budgets Per Step” lesson take?
Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.
Can I write and run code in this AI Agents lesson?
Yes. Every AI Agents lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.