단계별 지연 시간 및 비용

단계별 지표가 필요한 이유

지연 시간 기록

start = time.time()
# ... do work ...
span.duration_ms = (time.time() - start) * 1000

지연 시간 범주

비용 기록

PRICES = {
    'gpt-4o-mini':       {'in': 0.150 / 1e6, 'out': 0.600 / 1e6},
    'gpt-4o':            {'in': 2.50  / 1e6, 'out': 10.00 / 1e6},
    'claude-sonnet-4-5': {'in': 3.00  / 1e6, 'out': 15.00 / 1e6}
}

def compute_cost(model, tokens_in, tokens_out):
    p = PRICES[model]
    return tokens_in * p['in'] + tokens_out * p['out']

cost = compute_cost('gpt-4o-mini', 1000, 500)
print(f"Cost for 1000 in / 500 out tokens on gpt-4o-mini: ${cost:.6f}")

트레이스별 집계

def trace_metrics(spans):
    return {
        'duration_ms': sum(s.duration_ms for s in spans),
        'cost_usd':    sum(s.cost_usd or 0 for s in spans),
        'tokens':      sum((s.tokens_in or 0) + (s.tokens_out or 0) for s in spans),
        'step_count':  len(spans)
    }

from types import SimpleNamespace
spans = [
    SimpleNamespace(duration_ms=120, cost_usd=0.002, tokens_in=100, tokens_out=50),
    SimpleNamespace(duration_ms=340, cost_usd=0.005, tokens_in=200, tokens_out=80),
]
print(trace_metrics(spans))

p50 / p95 / p99

import numpy as np
durations = [t.duration_ms for t in last_1000_traces]
print('p50:', np.percentile(durations, 50))
print('p95:', np.percentile(durations, 95))
print('p99:', np.percentile(durations, 99))

사용자별 비용

SELECT user_id, SUM(cost_usd) AS daily_cost
FROM traces
WHERE created_at::date = current_date
GROUP BY user_id
ORDER BY daily_cost DESC
LIMIT 100

단계 이름별 비용

SELECT name, SUM(cost_usd) AS total
FROM spans
WHERE trace_id IN (SELECT id FROM traces WHERE created_at::date = current_date)
GROUP BY name
ORDER BY total DESC

지연 시간 워터폴

이상값 알림

두 모델 비교

# Run 100 traces with gpt-4o-mini and 100 with haiku
# Compare p95 latency and average cost
# Pick the winner for the use case
print("Run 100 traces with gpt-4o-mini and 100 with haiku")
print("Compare p95 latency and average cost")
print("Pick the winner for the use case")

토큰 분포 추적

백분위수를 사용하는 이유

정리