Rate Limiting und Verwaltung von API-Kontingenten

LLM providers cap usage in two ways:

• Requests per minute (RPM)
• Tokens per minute (TPM)

Exceed them and calls return 429 Too Many Requests, breaking your agents under load.

Waiting for 429s and retrying is wasteful. Proactive rate limiting spaces out requests so you stay under the cap, smoothing traffic and avoiding errors entirely.

LangChain can throttle model calls with a built-in rate limiter that releases a fixed number of requests per second.

Pass the limiter to the chat model. Every call now waits its turn automatically.

Some 429s and transient errors are unavoidable. Retry with growing delays so you do not hammer the provider.

Providers often return a Retry-After header telling you how long to wait. Honoring it is more polite and effective than a fixed delay.

Beyond provider limits, you set your own per-user quotas to control cost and fairness. Track usage in a store like Redis and reject or queue once a user exceeds their allowance.

The common pattern is a token bucket: tokens refill at a steady rate, each request consumes one, and an empty bucket means wait. It allows short bursts while enforcing an average rate.

When demand spikes past your limits, queue requests instead of dropping them. A background worker drains the queue at a safe rate, keeping the system stable.

For high throughput you can rotate across multiple API keys or providers, distributing load so no single key hits its cap. Track each key's usage independently.

Track 429 rates and how close you run to caps. Rising 429s signal you need a higher tier, better throttling, or more keys before users notice failures.

You learned to manage limits and quotas in production:

• Providers cap RPM and TPM; 429s break agents
• Use an InMemoryRateLimiter to throttle proactively
• Add retry with backoff and honor Retry-After
• Enforce per-user quotas with a token bucket
• Queue, rotate keys, and monitor 429 rates

Good limit management keeps scaled agents reliable and affordable.