0Pricing
AI Engineering Academy · Lesson

Load Balancing and Multi-Key Strategies

Implement round-robin and weighted load balancing across multiple API keys and accounts to multiply your rate limit headroom and reduce p99 latency spikes.

Load Balancing and Multi-Key Strategies is a free AI Engineering Academy lesson on CoddyKit — lesson 2 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the AI Engineering Academy learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.

Why One API Key Is Not Enough

A single OpenAI API key has a fixed rate limit measured in requests per minute (RPM) and tokens per minute (TPM). At Tier 1, GPT-4o allows 500 RPM and 30,000 TPM. For a production application with hundreds of concurrent users, a single key will hit these limits constantly. Multiple API keys multiply your available headroom proportionally.

Creating Multiple API Keys

You can create multiple API keys within a single OpenAI organization, or create multiple OpenAI accounts (each billed separately). Store each key in your environment configuration and treat them as a pool. Keep keys in a secrets manager like AWS Secrets Manager or HashiCorp Vault rather than in your source code or .env files committed to version control.

import os

API_KEYS = [
    os.environ['OPENAI_KEY_1'],
    os.environ['OPENAI_KEY_2'],
    os.environ['OPENAI_KEY_3'],
    os.environ['OPENAI_KEY_4'],
]

# Total effective RPM = 500 * 4 = 2000 RPM
# Total effective TPM = 30000 * 4 = 120000 TPM

Round-Robin Load Balancing

Round-robin distributes requests evenly across all keys by cycling through them in order. It is simple to implement and ensures each key handles roughly the same load over time. Use a thread-safe counter or an atomic integer to avoid two concurrent requests picking the same key simultaneously. Round-robin works well when all keys have identical rate limits.

import itertools
import threading
from openai import OpenAI

class RoundRobinPool:
    def __init__(self, keys: list):
        self._clients = [OpenAI(api_key=k) for k in keys]
        self._cycle = itertools.cycle(range(len(self._clients)))
        self._lock = threading.Lock()

    def get_client(self) -> OpenAI:
        with self._lock:
            idx = next(self._cycle)
        return self._clients[idx]

pool = RoundRobinPool(API_KEYS)
client = pool.get_client()

Weighted Load Balancing

Weighted load balancing assigns higher-tier keys (with higher rate limits) a larger share of traffic proportional to their capacity. If key A is Tier 3 (10,000 RPM) and key B is Tier 1 (500 RPM), key A should receive ~95% of requests. Weighted balancing prevents lower-tier keys from becoming bottlenecks when mixed with higher-tier ones.

import random

class WeightedPool:
    def __init__(self, key_configs: list):
        # key_configs = [{'key': '...', 'weight': 10}, ...]
        self._clients = [OpenAI(api_key=c['key']) for c in key_configs]
        self._weights = [c['weight'] for c in key_configs]

    def get_client(self) -> OpenAI:
        return random.choices(self._clients, weights=self._weights, k=1)[0]

pool = WeightedPool([
    {'key': os.environ['OPENAI_KEY_TIER3'], 'weight': 20},
    {'key': os.environ['OPENAI_KEY_TIER1'], 'weight': 1},
])

Tracking Per-Key Rate Limit State

The OpenAI API returns rate limit headers with every response: x-ratelimit-remaining-requests and x-ratelimit-remaining-tokens. Track these headers per key to know which keys are close to exhaustion. When a key reports fewer than 10 remaining requests in the current minute, temporarily route traffic away from it to prevent 429 errors before they occur.

class SmartPool:
    def __init__(self, keys: list):
        self._clients = [OpenAI(api_key=k) for k in keys]
        self._remaining = {i: 500 for i in range(len(keys))}  # initial RPM

    def get_best_client(self):
        # Pick key with most remaining capacity
        best_idx = max(self._remaining, key=lambda i: self._remaining[i])
        return self._clients[best_idx], best_idx

    def update_remaining(self, idx: int, response_headers: dict):
        remaining = int(response_headers.get('x-ratelimit-remaining-requests', 0))
        self._remaining[idx] = remaining

Handling 429 Rate Limit Errors

When a key returns a 429 error, immediately retire that key from the pool for the duration specified in the Retry-After header (typically 60 seconds). Mark it as cooling down and route all traffic to remaining keys. After the cool-down window expires, restore the key to the pool. This prevents cascading failures where retries on the same key make the situation worse.

import time
from openai import RateLimitError

class CooldownPool:
    def __init__(self, keys: list):
        self._clients = [(OpenAI(api_key=k), None) for k in keys]  # (client, cooldown_until)

    def get_available_clients(self):
        now = time.time()
        return [
            (i, c) for i, (c, until) in enumerate(self._clients)
            if until is None or until <= now
        ]

    def mark_cooling(self, idx: int, retry_after: int = 60):
        client, _ = self._clients[idx]
        self._clients[idx] = (client, time.time() + retry_after)
        print(f'Key {idx} cooling down for {retry_after}s')

Using OpenRouter as a Multiplexer

OpenRouter is a proxy service that exposes hundreds of models through a single OpenAI-compatible API endpoint. By routing through OpenRouter, you automatically get load balancing across multiple underlying provider accounts, fallback to alternative providers, and access to open-source models as backups. The cost markup is small for the operational simplicity it provides.

from openai import OpenAI

# OpenRouter uses the same OpenAI SDK interface
client = OpenAI(
    api_key=os.environ['OPENROUTER_API_KEY'],
    base_url='https://openrouter.ai/api/v1'
)

response = client.chat.completions.create(
    model='openai/gpt-4o',  # OpenRouter model name format
    messages=[{'role': 'user', 'content': prompt}]
)
# Automatic failover if OpenAI is down

Monitoring Key Health with Metrics

Track per-key metrics including requests sent, 429 errors received, and cool-down time in the last hour. A key with a high 429 rate needs either traffic reduction or a tier upgrade. Expose these metrics on a /metrics endpoint in Prometheus format so your monitoring system can alert when any key is consistently hitting limits.

from dataclasses import dataclass, field
from collections import defaultdict

@dataclass
class KeyMetrics:
    requests_sent: int = 0
    rate_limit_errors: int = 0
    total_tokens_used: int = 0
    cooldown_count: int = 0

class MetricPool:
    def __init__(self, keys: list):
        self._clients = [OpenAI(api_key=k) for k in keys]
        self._metrics = [KeyMetrics() for _ in keys]

    def report(self):
        for i, m in enumerate(self._metrics):
            error_rate = m.rate_limit_errors / max(m.requests_sent, 1)
            print(f'Key {i}: {m.requests_sent} req, {error_rate:.1%} 429 rate')

Geographic Key Distribution

If your users are spread globally, consider maintaining separate API keys per geographic region and routing requests to the key closest to the user. Reduced network round-trip time improves TTFT. Deploy a lightweight load balancer in each region (AWS Lambda@Edge or Cloudflare Worker) that selects the appropriate key and proxies the request, shielding your keys from client exposure.

REGIONAL_KEYS = {
    'us-east': os.environ['OPENAI_KEY_US_EAST'],
    'eu-west': os.environ['OPENAI_KEY_EU_WEST'],
    'ap-southeast': os.environ['OPENAI_KEY_AP'],
}

def get_key_for_region(user_region: str) -> str:
    # Default to us-east if region unknown
    return REGIONAL_KEYS.get(user_region, REGIONAL_KEYS['us-east'])

Testing Your Load Balancer

Write a load test that fires 100 concurrent requests through your balancing pool and measures distribution, error rates, and latency percentiles. Verify that no single key handles more than its proportional share and that 429 errors are below 0.1%. Use asyncio.gather or a tool like Locust to simulate the concurrent load your production system will actually experience.

import asyncio
import time

async def load_test(pool, concurrency=100, total=1000):
    sem = asyncio.Semaphore(concurrency)
    results = []

    async def one_request():
        async with sem:
            client = pool.get_client()
            start = time.perf_counter()
            try:
                await client.chat.completions.create(
                    model='gpt-4o-mini',
                    messages=[{'role': 'user', 'content': 'Ping'}],
                    max_tokens=5
                )
                results.append(('ok', time.perf_counter() - start))
            except Exception as e:
                results.append(('error', str(e)))

    await asyncio.gather(*[one_request() for _ in range(total)])
    ok = [r for r in results if r[0] == 'ok']
    print(f'Success rate: {len(ok)/total:.1%}')
    return results

Choosing the Right Balancing Strategy

Match your balancing strategy to your rate limit structure. Use round-robin when all keys have identical tier limits and traffic is evenly distributed. Use weighted balancing when keys have different tier limits. Use health-aware routing (skipping keys close to exhaustion) when you need to minimize 429 errors under burst traffic. For most production systems, health-aware routing with exponential backoff gives the best balance of simplicity and resilience.

# Strategy selection guide:
# Scenario A: 4 keys all Tier 2 (same limits)
#   -> Round-robin: simple, even distribution
#
# Scenario B: 1 Tier 3 key + 3 Tier 1 keys
#   -> Weighted: Tier 3 gets 10x weight
#
# Scenario C: Variable traffic with burst periods
#   -> Health-aware: track remaining headers, skip near-limit keys
#
# Scenario D: Multi-region, latency-sensitive
#   -> Geographic: regional keys, route by user location

Quick Check

Test your understanding of load balancing strategies for LLM APIs.

Lesson Recap

In this lesson you learned: round-robin and weighted balancing distribute traffic across multiple API keys to multiply rate limit headroom, cool-down tracking prevents cascading 429 errors by temporarily removing throttled keys, and OpenRouter provides a managed multiplexing option with automatic fallback. Next up we implement fallback providers and circuit breakers.

Frequently asked questions

Is the “Load Balancing and Multi-Key Strategies” lesson free?

Yes — the full text of “Load Balancing and Multi-Key Strategies” is free to read here on the web, and the AI Engineering Academy course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the AI Engineering Academy course, upgrade to CoddyKit PRO.

What will I learn in “Load Balancing and Multi-Key Strategies”?

Implement round-robin and weighted load balancing across multiple API keys and accounts to multiply your rate limit headroom and reduce p99 latency spikes. You practise AI Engineering Academy with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.

Do I need any experience to start AI Engineering Academy?

No prior experience is required. AI Engineering Academy on CoddyKit is structured for beginners through advanced learners; this is — lesson 2 of 4, so you can start here or from the beginning and move at your own pace.

How long does the “Load Balancing and Multi-Key Strategies” lesson take?

Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.

Can I write and run code in this AI Engineering Academy lesson?

Yes. Every AI Engineering Academy lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.

All lessons in this course

  1. Measuring LLM Latency: TTFT and TPOT
  2. Load Balancing and Multi-Key Strategies
  3. Fallback Providers and Circuit Breakers
  4. Timeout Budgets and Graceful Degradation
← Back to AI Engineering Academy