Provider rate limits apply to your whole account, so one heavy user can burn the capacity every other customer needs. Add your own limits per user and per tenant, counted in requests, tokens and cost, and enforce them before the call to the provider.
Why the provider limit is not enough
Provider limits are set at the organization level. One provider's documentation describes limits in requests per minute, input tokens per minute and output tokens per minute, enforced with a token bucket that refills continuously. When your account hits any of them, the provider returns a 429 error to whoever asks next. That might be your best customer.
The same documentation notes that limits are maximums, not guarantees, and that a sharp rise in traffic can trigger separate acceleration limits. So you cannot treat the published numbers as capacity you own.
Three things to count
Requests. Easy to count, easy to bypass by sending fewer, larger requests.
Tokens. The real cost driver. Count input and output separately, because output is usually more expensive. A request count alone misses a user who pastes a book into every message. For background, see rate limits in RPM, ITPM and OTPM.
Cost. Convert tokens to money using the model's current price and keep a running daily total per user. This is the number you protect the business with.
Window choices
Use two windows together. A short window, such as a minute, stops bursts. A long window, such as a day or a month, stops slow drain. A burst limit without a daily cap lets a script run all day at just under the limit.
Token bucket limits suit bursty chat traffic. Fixed monthly quotas suit plans you sell by tier.
What to do when a user hits the limit
- Return a clear error that says which limit and when it resets.
- Do not retry on your side. A retry loop turns one blocked user into more load.
- Offer a cheaper path when it fits, such as a smaller model for the next request.
- Alert yourself when a user hits the cap repeatedly. That pattern often means abuse or a broken integration.
Keep this separate from provider errors. A 429 from your own limiter and a 429 from the provider need different handling, as explained in telling rate limits from spend caps.
Where to store the counters
A fast shared store works well, because limits must hold across all your servers. Count before the call using an estimate, then correct with actual usage from the response. An estimate-then-adjust approach is also how the provider describes its own input token accounting.
Common mistakes
- Limiting by IP address only. Users share IPs and attackers rotate them.
- Limiting by user ID with free signups. A new account gets a new quota, so see free trial abuse in AI apps.
- Forgetting agent loops. A single user request that triggers many model calls needs its own cap on steps.
- Not capping output length per call.
Quick check
Should free and paid users share one limiter?
Use one mechanism with different numbers per plan. Keep free usage in a separate budget so it cannot starve paid traffic.
Sources
Last reviewed: 19 October 2026.
