0Pricing
AI Engineering Academy · Lesson

Measuring LLM Latency: TTFT and TPOT

Define time to first token and time per output token as the two key latency metrics, instrument your application to measure both, and establish per-endpoint SLA targets.

Measuring LLM Latency: TTFT and TPOT is a free AI Engineering Academy lesson on CoddyKit — lesson 1 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the AI Engineering Academy learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.

Why LLM Latency Has Two Components

Measuring LLM latency as a single number is misleading. There are actually two distinct phases: the time until the first token arrives (perceived responsiveness), and the time it takes to generate subsequent tokens (output speed). A model can have great TTFT but slow TPOT, making long responses feel sluggish even though the initial response felt instant.

TTFT: Time to First Token

Time to First Token (TTFT) is the duration from sending the API request to receiving the very first token of the response. It includes network latency, queuing time at the inference server, and prefill time (processing the input tokens). TTFT dominates perceived responsiveness — users notice when nothing appears for more than 1-2 seconds, regardless of how fast tokens stream afterward.

import time
from openai import OpenAI

client = OpenAI()

def measure_ttft(prompt: str) -> float:
    start = time.perf_counter()
    first_token_time = None
    stream = client.chat.completions.create(
        model='gpt-4o',
        messages=[{'role': 'user', 'content': prompt}],
        stream=True
    )
    for chunk in stream:
        if chunk.choices[0].delta.content:
            first_token_time = time.perf_counter()
            break  # stop after first token
    return first_token_time - start

TPOT: Time Per Output Token

Time Per Output Token (TPOT) is the average time between successive tokens once generation has started. It is calculated as total generation time divided by total output tokens. TPOT determines reading speed — humans read at roughly 250 words per minute, so a TPOT above 100ms per token (10 tokens/second) will feel noticeably slow for long responses.

import time
from openai import OpenAI

client = OpenAI()

def measure_tpot(prompt: str) -> dict:
    start = time.perf_counter()
    first_token_time = None
    token_count = 0
    stream = client.chat.completions.create(
        model='gpt-4o',
        messages=[{'role': 'user', 'content': prompt}],
        stream=True
    )
    for chunk in stream:
        delta = chunk.choices[0].delta.content or ''
        if delta:
            if first_token_time is None:
                first_token_time = time.perf_counter()
            token_count += 1
    end = time.perf_counter()
    ttft = first_token_time - start
    generation_time = end - first_token_time
    tpot = generation_time / max(token_count, 1)
    return {'ttft_ms': ttft * 1000, 'tpot_ms': tpot * 1000, 'tokens': token_count}

Total Latency vs TTFT vs TPOT

The relationship between these metrics is: total_latency = TTFT + (output_tokens × TPOT). For a response of 500 tokens at 50ms TPOT, generation takes 25 seconds. For streaming applications, optimize TTFT first — users tolerate slow streaming better than a blank screen. For batch processing without streaming, total latency is what matters, so optimize TPOT by choosing models with faster inference.

# Latency breakdown for a 200-token response
ttft_ms = 450       # half a second to first token
tpot_ms = 25        # 25ms per token = 40 tokens/sec
output_tokens = 200

total_latency = ttft_ms + (tpot_ms * output_tokens)
print(f'TTFT:  {ttft_ms}ms')
print(f'Generation: {tpot_ms * output_tokens}ms')
print(f'Total: {total_latency}ms ({total_latency/1000:.1f}s)')
# Output:
# TTFT:  450ms
# Generation: 5000ms
# Total: 5450ms (5.5s)

Factors That Affect TTFT

TTFT is dominated by prefill cost, which scales with the number of input tokens. A 10,000-token system prompt will have 10x higher TTFT than a 1,000-token prompt, all else being equal. Other factors include server load (queue time), network round-trip time to the API endpoint, and whether prompt caching reduces the effective prefill work. Minimize system prompt length to reduce TTFT.

# TTFT scales approximately linearly with input tokens
# Measured typical values for gpt-4o (2026):
# 500 input tokens:   ~400ms TTFT
# 2000 input tokens:  ~600ms TTFT
# 10000 input tokens: ~1500ms TTFT
# 32000 input tokens: ~4000ms TTFT

# Use prompt caching to avoid paying for repeated long prefixes:
# Cached tokens: ~200ms saved per 1000 cached tokens

Factors That Affect TPOT

TPOT is primarily determined by the model size and hardware. Smaller models (GPT-4o-mini) decode much faster than large models (GPT-4o). On self-hosted models, batch size and GPU memory bandwidth are the main factors. For OpenAI-hosted models, TPOT varies with server load but is typically 15-40ms per token. You cannot directly control TPOT on hosted APIs — model selection is your main lever.

# Typical TPOT benchmarks (approximate, 2026):
# gpt-4o-mini:   15-20ms per token (50-65 tokens/sec)
# gpt-4o:        25-40ms per token (25-40 tokens/sec)
# claude-3.5-haiku: 20-25ms per token
# claude-3.5-sonnet: 30-50ms per token
# local llama-3.1-8B on A100: 8-12ms per token
# local llama-3.1-70B on 4xA100: 25-35ms per token

Setting SLA Targets Per Endpoint

Not all endpoints need the same latency target. A chat endpoint has a tight TTFT SLA (users expect <500ms), while a batch summarization endpoint can tolerate multi-second latency. Define explicit SLA targets per endpoint and instrument each one separately. Common targets: interactive chat = p95 TTFT <600ms, document extraction = p95 total <10s, batch = no real-time SLA.

SLA_TARGETS = {
    'chat':       {'ttft_p95_ms': 600,  'total_p95_ms': 8000},
    'extraction': {'ttft_p95_ms': 1500, 'total_p95_ms': 10000},
    'summary':    {'ttft_p95_ms': 2000, 'total_p95_ms': 30000},
    'batch':      {'ttft_p95_ms': None, 'total_p95_ms': None},
}

Logging Latency Metrics

Log TTFT and TPOT for every production request with structured logging. Include the model name, endpoint, prompt token count, output token count, and whether a cache hit occurred. This gives you the data to compute percentile distributions (p50, p95, p99), spot regressions after model updates, and correlate latency spikes with queue depth or provider incidents.

import structlog

log = structlog.get_logger()

def log_latency(endpoint: str, model: str, metrics: dict):
    log.info(
        'llm_latency',
        endpoint=endpoint,
        model=model,
        ttft_ms=round(metrics['ttft_ms'], 1),
        tpot_ms=round(metrics['tpot_ms'], 1),
        output_tokens=metrics['tokens'],
        total_ms=round(metrics['ttft_ms'] + metrics['tpot_ms'] * metrics['tokens'], 1)
    )

Computing Percentiles from Samples

Raw averages are misleading for latency — a few slow outliers inflate the mean without affecting most users. Always report p50, p95, and p99 percentiles. P95 is the most common SLA metric: it means 95% of requests completed within that time. Use NumPy or the statistics module to compute percentiles from your logged latency samples.

import numpy as np

def compute_percentiles(samples: list, label: str = 'latency_ms'):
    arr = np.array(samples)
    stats = {
        'count': len(arr),
        'p50': np.percentile(arr, 50),
        'p95': np.percentile(arr, 95),
        'p99': np.percentile(arr, 99),
        'mean': np.mean(arr),
        'max': np.max(arr)
    }
    print(f'{label}:')
    for k, v in stats.items():
        print(f'  {k}: {v:.1f}')
    return stats

Reducing Latency with max_tokens

Setting an appropriate max_tokens limit reduces worst-case total latency by preventing runaway long responses. If your use case needs at most 200-token answers, set max_tokens=250. This also caps cost. Combine with streaming to ensure users see output immediately while the full response is still generating. Never leave max_tokens unlimited in production endpoints.

response = client.chat.completions.create(
    model='gpt-4o',
    messages=[{'role': 'user', 'content': question}],
    max_tokens=300,  # cap at 300 tokens
    stream=True
)

# With max_tokens=300 and TPOT=30ms:
# worst-case total generation = 9000ms
# Without limit: could run to 4096+ tokens = 123s+

Latency Optimization Priorities

When latency is too high, address bottlenecks in priority order. First, enable streaming so users see output immediately even if total latency is high. Second, shorten the system prompt to reduce TTFT. Third, add prompt prefix caching to amortize prefill cost across repeated requests. Fourth, switch to a smaller model if quality allows. Finally, consider self-hosted inference for maximum control over both TTFT and TPOT.

# Latency optimization checklist (in priority order):
# 1. Enable streaming (perceived latency: immediate)
# 2. Shorten system prompt by 50% (TTFT: -20%)
# 3. Enable prompt prefix caching (TTFT: -40% on cache hits)
# 4. Downgrade to gpt-4o-mini for simple queries (TPOT: -40%)
# 5. Self-host llama-3.1-8B for high-volume simple queries
#    (TPOT: 8ms vs 25ms; TTFT: 100ms vs 450ms)

Quick Check

Test your understanding of TTFT and TPOT as LLM latency metrics.

Lesson Recap

In this lesson you learned: TTFT (Time to First Token) measures perceived responsiveness and scales with input token count, TPOT (Time Per Output Token) determines generation speed and is mainly controlled by model size, and per-endpoint SLA targets with percentile metrics are the right way to monitor latency in production. Next up we implement load balancing across multiple API keys.

Frequently asked questions

Is the “Measuring LLM Latency: TTFT and TPOT” lesson free?

Yes — the full text of “Measuring LLM Latency: TTFT and TPOT” is free to read here on the web, and the AI Engineering Academy course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the AI Engineering Academy course, upgrade to CoddyKit PRO.

What will I learn in “Measuring LLM Latency: TTFT and TPOT”?

Define time to first token and time per output token as the two key latency metrics, instrument your application to measure both, and establish per-endpoint SLA targets. You practise AI Engineering Academy with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.

Do I need any experience to start AI Engineering Academy?

No prior experience is required. AI Engineering Academy on CoddyKit is structured for beginners through advanced learners; this is — lesson 1 of 4, so you can start here or from the beginning and move at your own pace.

How long does the “Measuring LLM Latency: TTFT and TPOT” lesson take?

Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.

Can I write and run code in this AI Engineering Academy lesson?

Yes. Every AI Engineering Academy lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.

All lessons in this course

  1. Measuring LLM Latency: TTFT and TPOT
  2. Load Balancing and Multi-Key Strategies
  3. Fallback Providers and Circuit Breakers
  4. Timeout Budgets and Graceful Degradation
← Back to AI Engineering Academy