Latency and Cost per Step
Measure token usage and wall-clock time per node so you can find the slow and expensive steps.
Latency and Cost per Step is a free AI Agents lesson on CoddyKit — lesson 3 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the AI Agents learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.
Why Per-Step Metrics?
Total latency and cost tell you the system is slow or expensive — but not WHICH step is slow or expensive. Per-step measurement is required to fix it.
Recording Latency
Add it to every span:
start = time.time()
# ... do work ...
span.duration_ms = (time.time() - start) * 1000Latency Categories
Common slow steps in agents:
- LLM call — 500ms to 10s
- Web search — 200ms to 2s
- Embedding — 50ms to 200ms
- DB query — 5ms to 500ms
Recording Cost
For LLM steps, multiply tokens by price:
PRICES = {
'gpt-4o-mini': {'in': 0.150 / 1e6, 'out': 0.600 / 1e6},
'gpt-4o': {'in': 2.50 / 1e6, 'out': 10.00 / 1e6},
'claude-sonnet-4-5': {'in': 3.00 / 1e6, 'out': 15.00 / 1e6}
}
def compute_cost(model, tokens_in, tokens_out):
p = PRICES[model]
return tokens_in * p['in'] + tokens_out * p['out']
cost = compute_cost('gpt-4o-mini', 1000, 500)
print(f"Cost for 1000 in / 500 out tokens on gpt-4o-mini: ${cost:.6f}")
Rollup Per Trace
Sum spans up to the trace level:
def trace_metrics(spans):
return {
'duration_ms': sum(s.duration_ms for s in spans),
'cost_usd': sum(s.cost_usd or 0 for s in spans),
'tokens': sum((s.tokens_in or 0) + (s.tokens_out or 0) for s in spans),
'step_count': len(spans)
}
from types import SimpleNamespace
spans = [
SimpleNamespace(duration_ms=120, cost_usd=0.002, tokens_in=100, tokens_out=50),
SimpleNamespace(duration_ms=340, cost_usd=0.005, tokens_in=200, tokens_out=80),
]
print(trace_metrics(spans))
p50 / p95 / p99
Don't look at averages — agent latencies are heavy-tailed. Report percentiles:
import numpy as np
durations = [t.duration_ms for t in last_1000_traces]
print('p50:', np.percentile(durations, 50))
print('p95:', np.percentile(durations, 95))
print('p99:', np.percentile(durations, 99))Cost per User
Sum cost by user_id daily:
SELECT user_id, SUM(cost_usd) AS daily_cost
FROM traces
WHERE created_at::date = current_date
GROUP BY user_id
ORDER BY daily_cost DESC
LIMIT 100Cost per Step Name
Which steps dominate cost? Group by name:
SELECT name, SUM(cost_usd) AS total
FROM spans
WHERE trace_id IN (SELECT id FROM traces WHERE created_at::date = current_date)
GROUP BY name
ORDER BY total DESCLatency Waterfalls
For one trace, plot a Gantt-style waterfall of spans. The longest bar = your bottleneck.
Most tracing UIs render this automatically.
Alert on Outliers
- Alert if p95 latency doubles week-over-week
- Alert if daily cost spikes above 2x normal
- Alert if any single user costs > $X / day
Compare Two Models
Per-step metrics let you A/B compare models:
# Run 100 traces with gpt-4o-mini and 100 with haiku
# Compare p95 latency and average cost
# Pick the winner for the use case
print("Run 100 traces with gpt-4o-mini and 100 with haiku")
print("Compare p95 latency and average cost")
print("Pick the winner for the use case")
Track Token Distribution
Histogram of input vs output tokens reveals issues: e.g. consistently hitting max_tokens = output truncation = bug.
Why Percentiles?
Why report p95 latency instead of average?
Recap
Latency and cost per step, rolled up per trace and per user, percentiles not averages, alerts on outliers. This is your dashboard.
Frequently asked questions
Is the “Latency and Cost per Step” lesson free?
Yes — the full text of “Latency and Cost per Step” is free to read here on the web, and the AI Agents course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the AI Agents course, upgrade to CoddyKit PRO.
What will I learn in “Latency and Cost per Step”?
Measure token usage and wall-clock time per node so you can find the slow and expensive steps. You practise AI Agents with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.
Do I need any experience to start AI Agents?
No prior experience is required. AI Agents on CoddyKit is structured for beginners through advanced learners; this is — lesson 3 of 4, so you can start here or from the beginning and move at your own pace.
How long does the “Latency and Cost per Step” lesson take?
Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.
Can I write and run code in this AI Agents lesson?
Yes. Every AI Agents lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.