Rate Limiting (AI Context)

Why It Matters

LLM inference is expensive, both computationally and financially. Without rate limiting, a single user or a bot could burn through your entire infrastructure budget in minutes. Or worse, overwhelm your GPU clusters and take down the service for everyone.

Rate limiting protects three things: your infrastructure (GPUs are expensive and finite), your costs (every request costs money), and your users (fair access for all). For AI engineers, understanding rate limits is essential because they shape application architecture.

This is especially critical when wrapping external LLM APIs. OpenAI, Anthropic, and other providers enforce their own rate limits. Your application needs to handle these gracefully. Queue requests, implement exponential backoff, and communicate wait times to users.

Implementation Basics

Effective AI rate limiting considers multiple dimensions:

1. Request-Based Limits Simple counts: N requests per minute/hour per user. Easy to implement but doesn’t account for request complexity. A 10-token prompt costs the same as a 4000-token prompt.

2. Token-Based Limits Track tokens per minute (TPM) for both input and output. This better reflects actual resource consumption. A user generating long documents consumes more than one asking short questions.

3. Tiered Access Different limits for different user tiers. Free users get 10 requests/minute, paid users get 100, enterprise gets 1000. Implement with API keys and a rate limiting service like Redis.

4. Graceful Handling Return 429 status codes with Retry-After headers. Implement client-side queuing for better UX. Consider token bucket algorithms that allow bursts while maintaining average limits.

Implementation: Redis + sliding window counters is the standard pattern. For Python, libraries like slowapi integrate with FastAPI. Always implement rate limiting at the gateway level, not just application level.

Monitor your limits carefully. Too restrictive kills user experience, too loose kills your budget.