상류 서비스 캐싱 및 요청률 제한

An eager model may call a tool again and again. Without guards, it can overload the database or API behind it. ⚠️

A cache stores recent results so an identical request returns instantly instead of hitting the upstream a second time.

Build the cache key from the tool arguments, so the same city or id reuses its stored answer and anything new fetches fresh.

Set a time-to-live so cached data expires. Stale answers get refreshed, keeping results both fast and reasonably current.

Cache idempotent reads, never actions with side effects. You would not want a cached write to silently skip a real change.

A rate limit caps how often a tool may run, so a runaway model cannot fire thousands of requests in a burst.

A common scheme is the token bucket: each call spends a token, tokens refill over time, and an empty bucket means wait.

Match your limit to the API quota. Staying under its quota avoids 429 responses and keeps your access from being throttled.

If an upstream replies 429 Too Many Requests, pause and retry later with backoff instead of pounding it harder.

A semaphore limits how many calls hit the backend at once, smoothing spikes so the upstream is never swamped.

When you do throttle, tell the model plainly that it is rate-limited so it can wait or try a different approach.

What kind of result is safe to cache?

You shielded upstreams with a cache plus rate limits and backoff, keeping backends safe from runaway model calls. 🎯