Timeout Budgets and Graceful Degradation
Set aggressive timeout budgets at each layer of your pipeline, implement graceful degradation that serves cached or simplified responses when the LLM exceeds its budget.
Timeout Budgets and Graceful Degradation is a free AI Engineering Academy lesson on CoddyKit — lesson 4 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the AI Engineering Academy learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.
What Is a Timeout Budget?
A timeout budget is a maximum total time allocated for a request to complete across all stages of your pipeline. Instead of setting an arbitrary timeout on each individual API call, you define the end-to-end budget for the user-facing operation and distribute it across retrieval, LLM generation, and post-processing steps. This ensures you always respond within an acceptable time, even if some stages are slow.
Distributing Budget Across Pipeline Stages
A typical RAG chat pipeline has three stages: retrieval, LLM generation, and response formatting. Assign a time slice to each based on how long each normally takes and how much slack users tolerate. The remaining slack is your degradation buffer — if any stage uses its full allocation, you start cutting corners in subsequent stages to stay within the overall budget.
# Total user-facing SLA: 8000ms
BUDGET_TOTAL_MS = 8000
BUDGET_STAGES = {
'retrieval': 1500, # vector search + rerank
'llm_call': 5500, # token streaming
'formatting': 500, # post-processing
'slack': 500, # buffer for overhead
}
assert sum(BUDGET_STAGES.values()) == BUDGET_TOTAL_MSTracking Budget Consumption
Use a BudgetTracker that records the start time and checks remaining budget at each stage transition. Before starting a stage, verify enough budget remains. This allows downstream stages to adapt — a retrieval step that takes 1200ms of its 1500ms budget leaves only 300ms of slack, which should trigger a simpler LLM prompt or skip the re-ranking step.
import time
class BudgetTracker:
def __init__(self, total_ms: float):
self.start = time.perf_counter()
self.total_ms = total_ms
def elapsed_ms(self) -> float:
return (time.perf_counter() - self.start) * 1000
def remaining_ms(self) -> float:
return self.total_ms - self.elapsed_ms()
def check(self, stage: str, required_ms: float = 0) -> bool:
remaining = self.remaining_ms()
if remaining < required_ms:
print(f'Budget exhausted before {stage}: {remaining:.0f}ms left, need {required_ms}ms')
return False
return TrueGraceful Degradation Definition
Graceful degradation means serving a lower-quality but still useful response when the full pipeline cannot complete within the budget, rather than returning an error. Examples include: returning a cached response, skipping re-ranking, truncating the context window, using a faster but less accurate model, or returning a pre-written fallback message. The goal is always to give the user something rather than nothing.
# Degradation ladder for a RAG chat endpoint:
# Level 0 (normal): retrieve 10 chunks + rerank + GPT-4o -- 8000ms budget
# Level 1 (fast): retrieve 5 chunks + skip rerank + GPT-4o -- 5000ms budget
# Level 2 (minimal): retrieve 3 chunks + GPT-4o-mini -- 3000ms budget
# Level 3 (cached): return semantic cache hit -- 100ms
# Level 4 (sorry): return static 'Try again in a moment' -- 1msImplementing the Degradation Ladder
At each pipeline decision point, check the remaining budget and choose the appropriate quality level. The code below chooses retrieval depth and model based on remaining budget. This means under normal load users get the best quality, while during high-latency periods they still get a useful response rather than a timeout error.
async def smart_rag_query(question: str, budget_ms: float = 8000) -> str:
tracker = BudgetTracker(budget_ms)
# Retrieval stage
if tracker.remaining_ms() > 5000:
chunks = await retrieve_and_rerank(question, top_k=10)
elif tracker.remaining_ms() > 3000:
chunks = await retrieve(question, top_k=5) # skip rerank
elif tracker.remaining_ms() > 1500:
chunks = await retrieve(question, top_k=3) # minimal retrieval
else:
return await get_cached_or_static(question)
# LLM stage
if tracker.remaining_ms() > 4000:
model = 'gpt-4o'
else:
model = 'gpt-4o-mini' # faster fallback
timeout = tracker.remaining_ms() / 1000 - 0.5
return await generate_answer(question, chunks, model, timeout)Setting Timeouts at the API Call Level
Always set explicit timeouts on every external API call. The OpenAI Python SDK accepts a timeout parameter in seconds. Set it to slightly less than the remaining budget so you have time to handle the exception and potentially degrade gracefully before the overall response deadline. Never rely on the SDK's default timeout — it may be too long for user-facing requests.
async def generate_answer(question: str, chunks: list, model: str, timeout_sec: float) -> str:
context = '\n\n'.join(chunks)
prompt = f'Answer using this context:\n{context}\n\nQuestion: {question}'
try:
resp = await client.chat.completions.create(
model=model,
messages=[{'role': 'user', 'content': prompt}],
max_tokens=500,
timeout=max(timeout_sec, 1.0) # minimum 1 second
)
return resp.choices[0].message.content
except openai.APITimeoutError:
return 'I was unable to generate a response in time. Please try again.'Returning Partial Streaming Responses
With streaming, you can return partial responses that were generated before the budget expired. When a timeout occurs mid-stream, stop reading new tokens, append an ellipsis or a brief continuation prompt, and close the stream. The user sees a response that cuts off cleanly rather than a blank error. This is only possible with streaming — non-streaming calls are all-or-nothing.
async def stream_with_budget(question: str, budget_ms: float):
tracker = BudgetTracker(budget_ms)
collected = []
stream = await client.chat.completions.create(
model='gpt-4o-mini',
messages=[{'role': 'user', 'content': question}],
stream=True
)
async for chunk in stream:
if tracker.remaining_ms() < 200: # 200ms safety margin
collected.append(' [response truncated]')
break
delta = chunk.choices[0].delta.content or ''
collected.append(delta)
yield delta
# Ensure stream is closed even if budget exceeded
await stream.close()Semantic Cache as Degradation Layer
A semantic cache is an excellent degradation layer because it has near-zero latency. Before calling the LLM, query your semantic cache for similar previous questions. If a cache hit with high similarity (above 0.92 cosine similarity) is found, return the cached answer immediately. This both speeds up responses and provides an instant fallback when the LLM is slow or unavailable.
async def query_with_cache_fallback(question: str, budget_ms: float = 8000) -> str:
# Try semantic cache first (fast)
cached = await semantic_cache.lookup(question, threshold=0.92)
if cached:
return cached.response
tracker = BudgetTracker(budget_ms)
# Try full pipeline
if tracker.remaining_ms() > 3000:
try:
return await smart_rag_query(question, tracker.remaining_ms())
except Exception:
pass # fall through to static response
# Last resort
return 'I am experiencing high load right now. Please try again in a moment.'Logging Degradation Events
Every time your pipeline degrades to a lower quality level, log it as a structured event. Include the degradation level reached, the budget remaining at each stage, and the final latency. Analyzing these logs tells you how often each degradation level is triggered, which helps you tune budgets, identify which stages are consistently over-budget, and justify infrastructure investments.
import structlog
log = structlog.get_logger()
def log_degradation(level: int, stage: str, remaining_ms: float, total_ms: float):
log.warning(
'pipeline_degradation',
degradation_level=level,
triggered_at_stage=stage,
remaining_budget_ms=round(remaining_ms),
total_budget_ms=total_ms,
budget_consumed_pct=round((total_ms - remaining_ms) / total_ms * 100)
)Setting User Expectations with UI Signals
When serving a degraded response, signal to the user that the quality may be lower than usual. For a chat interface, display a subtle indicator like 'Fast response mode — some details may be limited.' For an extraction API, include a degraded: true field in the JSON response so downstream consumers can handle degraded results differently. Transparency preserves user trust even during outages.
from pydantic import BaseModel
from typing import Optional
class ChatResponse(BaseModel):
content: str
degraded: bool = False
degradation_level: Optional[int] = None # 0=full, 1=fast, 2=minimal, 3=cached
latency_ms: int
# API response when degraded:
# {
# 'content': 'Here is a brief answer...',
# 'degraded': true,
# 'degradation_level': 2,
# 'latency_ms': 2800
# }Tuning Budget Allocations Over Time
Initial budget allocations are estimates. After running in production for a week, analyze the distribution of time spent in each stage using your tracing data. If retrieval consistently takes 800ms instead of the budgeted 1500ms, reallocate that slack to the LLM stage, allowing more output tokens or a larger context window. Budget tuning is an ongoing operation activity, not a one-time configuration.
# Budget tuning based on production p95 data:
ACTUAL_P95 = {
'retrieval': 780, # vs budget 1500ms -> 720ms headroom
'llm_call': 4200, # vs budget 5500ms -> 1300ms headroom
'formatting': 120, # vs budget 500ms -> 380ms headroom
}
TOTAL_HEADROOM = sum(
BUDGET_STAGES[k] - ACTUAL_P95[k] for k in ACTUAL_P95
)
print(f'Total headroom: {TOTAL_HEADROOM}ms')
# Reallocate headroom to allow longer LLM responsesQuick Check
Test your understanding of timeout budgets and graceful degradation.
Lesson Recap
In this lesson you learned: timeout budgets allocate time across pipeline stages so you always respond within an acceptable deadline, degradation ladders serve progressively lower-quality responses rather than errors when time is exhausted, and logging degradation events helps you identify and fix chronic bottlenecks. Next up we explore the LLM-as-judge pattern for automated quality evaluation.
Frequently asked questions
Is the “Timeout Budgets and Graceful Degradation” lesson free?
Yes — the full text of “Timeout Budgets and Graceful Degradation” is free to read here on the web, and the AI Engineering Academy course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the AI Engineering Academy course, upgrade to CoddyKit PRO.
What will I learn in “Timeout Budgets and Graceful Degradation”?
Set aggressive timeout budgets at each layer of your pipeline, implement graceful degradation that serves cached or simplified responses when the LLM exceeds its budget. You practise AI Engineering Academy with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.
Do I need any experience to start AI Engineering Academy?
No prior experience is required. AI Engineering Academy on CoddyKit is structured for beginners through advanced learners; this is — lesson 4 of 4, so you can start here or from the beginning and move at your own pace.
How long does the “Timeout Budgets and Graceful Degradation” lesson take?
Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.
Can I write and run code in this AI Engineering Academy lesson?
Yes. Every AI Engineering Academy lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.
All lessons in this course
- Measuring LLM Latency: TTFT and TPOT
- Load Balancing and Multi-Key Strategies
- Fallback Providers and Circuit Breakers
- Timeout Budgets and Graceful Degradation