An AI API rate limit caps how much an organization can send in a window of time, usually measured in requests per minute and tokens per minute. Providers use limits to prevent abuse and share capacity fairly. Hitting one returns an HTTP 429 error, and your capacity refills continuously rather than resetting on the clock.
The three numbers you will see
Most providers publish limits as a small set of counters. One provider's documentation lists requests per minute (RPM), input tokens per minute (ITPM) and output tokens per minute (OTPM) for each model class. Another lists RPM and RPD for requests, TPM and TPD for tokens, and notes that whichever limit you hit first is the one that applies.
| Counter | What it measures | When it bites |
|---|---|---|
| RPM | Number of calls | Many small requests, such as chat widgets or per-file checks |
| Input tokens per minute | Size of prompts | Large documents, long histories, big tool lists |
| Output tokens per minute | Size of generated text | Long answers, reasoning-heavy work |
A workload can sit far below one counter and still be blocked by another. A batch of tiny prompts trips the request counter. A few huge prompts trip the input counter.
How a token bucket refills
The Anthropic documentation says its API uses the token bucket algorithm. Your capacity is continuously replenished up to a maximum, rather than reset at fixed intervals. In practice this means a limit of 60 requests per minute may be enforced as roughly one request per second, so a short burst can fail even when your per-minute average looks fine.
The same page states that limits are maximums, not guaranteed minimums. Plan for the limit to be lower under unusual conditions, and read the response headers, which report the limit, what remains, and when it resets.
Limits are set per organization and tier
Limits are tied to a usage tier. The documentation says organizations are placed on a tier based on usage history and account standing, and may start in an evaluation tier with lower limits while history builds. OpenAI's guide describes a similar ladder where organizations move up as their credit purchases grow. You can also set lower limits for individual workspaces or projects, so one noisy service cannot use up the whole organization's allowance.
What counts toward the limit
This is where providers differ, and where cost and throughput meet:
- Cached input can be treated differently. The Anthropic page says that for most of its models, only uncached input tokens count toward ITPM.
- The max output setting can count or not. OpenAI says usage counts against the larger of the max tokens value and an estimate from the request length, so set it close to the response size you expect. The Anthropic page says its max tokens setting does not factor into OTPM calculations.
- Reasoning tokens usually count as output. See metering thinking tokens.
Always check the current page for the model you use, because these rules change.
What to do when you hit one
Wait for the retry-after value when the response includes it. If it does not, use exponential backoff with random jitter and cap both attempts and total retry time. OpenAI notes failed requests still count toward limits, so instant resends make things worse. The follow-up post on retries and backoff covers this in detail.
Where you can, move non-urgent work to an asynchronous path, which draws from a separate pool. See how batch APIs work.
Why security teams care
A rate limit is also a blast radius control. If an API key leaks, the limit bounds how fast it can be abused. If an agent loops, the limit slows the loop. Treat limits as one layer, with spend caps and key scoping alongside, as covered in is an AI spend cap a security control.
Worked example: which counter trips first
The numbers below are illustrative, not any provider's real limits. Say your tier allows 1,000 requests per minute, 2,000,000 input tokens per minute and 400,000 output tokens per minute for one model.
| Workload | Requests per minute | Input tokens per minute | Output tokens per minute | First limit hit |
|---|---|---|---|---|
| Chat widget, short prompts | 1,200 | 600,000 | 120,000 | Requests |
| Document summarizer, long prompts | 60 | 2,400,000 | 90,000 | Input tokens |
| Report writer, long answers | 100 | 300,000 | 500,000 | Output tokens |
Each workload is fine on two counters and blocked by the third. The fix differs every time. The chat widget needs a queue or request batching. The summarizer needs caching of repeated document text or smaller prompts. The report writer needs shorter outputs or a higher tier.
Build a limit budget before launch
- Pull your current limits for each model from the provider console. Write them in a table with the date.
- Measure p50 and p95 prompt size and output size from a pilot. Do not use the average, because bursts are what trip limits.
- Compute peak requests per minute from your busiest hour, then multiply by p95 sizes to get token rates.
- Compare each of the three counters with the limit and keep at least some headroom on each.
- Decide the response for each shortfall: queue, cache, shrink, move to batch, or request more capacity.
Common mistakes
- Planning on the requests number alone and forgetting tokens.
- Sharing one pool between a user-facing feature and a background job, so the job starves the feature. Separate workspaces or projects fix this.
- Treating the published table as a promise. The Anthropic page says limits are maximums, not guaranteed minimums.
- Ignoring that several models can share one combined limit. The same page notes that some model families draw from a single bucket.
Reading the response headers in code
The headers report the limit, the remaining amount and the reset time for requests and tokens. Log them, and if remaining falls under a threshold, slow your queue before you get an error. This pacing is cheaper than recovering from errors, and it keeps latency steadier for users. Record which counter is the tightest per workspace so your dashboard shows the true bottleneck, not just the error count.
Glossary
A request is one call to the API. A token is a unit of text the model reads or writes. Input tokens are what you send, output tokens are what comes back. A token bucket is a limiter that refills continuously up to a maximum. A tier is a bundle of limits. A workspace or project is a boundary inside an organization with its own limits. An acceleration limit is a limit that can trigger on a sharp rise in traffic. Keep these definitions in your team wiki so conversations about limits use the same words.
Try it yourself
Click through it at your own pace, or open it full screen. The demo scans an API for security issues rather than measuring AI limits, but it shows the same habit this post recommends: look at an API as a set of endpoints with their own behavior, and check each one separately.
Quick check
Is a rate limit the same as a spend limit?
No. A rate limit caps volume over a short window and refills continuously. A spend limit caps cost over a billing period. They return similar errors but need different handling.
Why did I get a 429 when my average was under the limit?
Limits can be enforced over shorter intervals than a minute, so bursts can exceed them. Smooth your traffic with a queue.
Do cached tokens count toward my limit?
It depends on the provider and model. Check the current rate limit page for the model you use.
Sources
Last reviewed: 19 October 2026.
