How should you retry AI API 429 errors without making them worse?

Retry a rate limit error by waiting for the retry-after value if the response has one, otherwise use exponential backoff with random jitter. Cap both the number of attempts and the total retry time. Do not retry quota or billing errors. Failed requests still count toward limits, so instant retries only dig the hole deeper.

Why naive retries hurt

When a service returns 429 or 503, many clients retry immediately. If a hundred workers do this together, they hit the limit again at the same moment. OpenAI's rate limit guide notes that failed requests still count toward your limits, so resending right away does not help. Anthropic's documentation adds that the limit uses a token bucket that refills continuously, and that short bursts can exceed it even when the minute average is fine.

Two further traps appear in practice. SDKs often retry on their own, and wrapping them in your own retry loop multiplies attempts. OpenAI's guide says to account for both layers if you add your own. And a retry of a long request repeats the whole cost of the input, so each failed attempt on a large prompt is expensive in tokens as well as time.

A retry policy you can copy

1. Classify the error. Rate limit and overloaded errors are retryable. Quota, billing and spend cap errors are not. Authentication and invalid request errors are not.
2. If a retry-after header exists, wait at least that long.
3. Otherwise wait base delay times two to the power of the attempt, then add random jitter so workers spread out.
4. Set a maximum delay, a maximum attempt count and a maximum total time. Fail the job visibly when they run out.
5. Make the operation safe to repeat. If a request triggers a side effect such as writing a record, use an idempotency key or check before writing.
6. Emit a metric for every retry, labeled by error type, so you can see pressure building before users do.

Pseudocode

for attempt in 0 to max_attempts:
    response = send(request)
    if response.ok: return response
    if not retryable(response): raise
    wait = response.retry_after or min(cap, base * 2**attempt)
    sleep(wait + random(0, wait * 0.25))
raise GiveUp

Better than retrying: send less at once

Retries are a repair. Prevention is a queue in front of the API with a concurrency limit that matches your tier. Put a token estimate on each job so the queue can respect input token budgets as well as request counts. Counting tokens before you send is covered in count tokens before you send.

Other ways to cut pressure:

• Combine small prompts into one request when you hit the request counter but have token room, a tip from OpenAI's guide.
• Cache repeated prefixes. On some providers cached input does not count toward the input token limit.
• Send non-urgent work through an asynchronous path with its own limit pool.
• Set max output size close to what you expect, since one provider counts the larger of that value and an estimate.

Know when to stop

A retry policy needs an exit. When a spend cap is reached, retries cannot succeed. The documentation for one provider says its SDK's automatic retries fail until access resumes. Treat that error as a page to a human, not a delay. See 429 rate limit or spend cap for how to tell.

Worked example: the delay schedule

Suppose the base delay is one second, the cap is 30 seconds, and jitter adds up to 25 percent. The waits grow like this, before jitter.

After the sixth attempt, or after a total budget such as two minutes, give up and surface the failure. The numbers are examples; choose your own from your user-facing timeout.

Why jitter matters

Picture fifty workers that all fail at the same instant. Without jitter, all fifty retry at one second, fail together again, then retry together at two seconds. The synchronized waves look like a traffic spike each time. With jitter, retries spread over the window, and the limit's continuous refill can serve them in turn.

Testing your policy

1. Point a test client at a stub server that returns 429 for the first three calls, then succeeds. Confirm the waits grow and the call finally succeeds.
2. Make the stub return a retry-after value and confirm your client honors it over its own schedule.
3. Make the stub return a spend cap style error and confirm there is no retry at all.
4. Run fifty concurrent clients against a stub with a low limit and look at the arrival pattern. It should not be synchronized.
5. Confirm that a request with a side effect is not applied twice after a retry.

Retry budgets across a system

If service A calls service B and both retry three times, one failure can produce nine calls. Set a retry budget at the edge and pass a deadline downstream, so inner layers do not keep retrying after the outer caller has given up.

Idempotency in practice

Retries are safe only when repeating the call is safe. Reading a completion is safe to repeat apart from cost. A workflow that creates a ticket, sends an email or charges a card is not. Wrap side effects in a step that checks a stored key before acting, and generate the key once per logical job, not once per attempt. Then a retry after a timeout cannot create a duplicate, even if the first attempt did reach the server.

Also remember that a retried model call costs tokens again. A job with a large prompt and a flaky network can quietly double its bill, so count retries in your cost reporting.

Documenting the policy

Put the retry policy in your service template and in your runbook: the retryable error classes, the base delay and cap, the attempt and total time limits, and the rule that spend and billing errors are never retried. Add the metric names and the alert thresholds. When a new team builds an integration, they copy a tested policy rather than writing their own from memory.

What this means for you

Retries should help a system recover, not pile on. A good policy waits, spreads out its attempts and gives up in a sensible time. A bad one amplifies a small hiccup into an outage and a larger bill.

Common mistakes

Typical errors are retrying immediately, retrying without jitter so workers collide, retrying errors that cannot succeed, stacking retries across layers, retrying side effects without protection and never emitting a metric. Set a base, a cap and a budget, honor the retry-after value and make operations safe to repeat.

Try it yourself

Click through it at your own pace, or open it full screen. It is a security monitoring demo, not a retry tester. Use it as a prompt to ask what you would watch continuously in your own service: retry counts by error type are a good first signal.

Quick check

Sources

• Rate limits, OpenAI API documentation
• Rate limits, Claude API documentation

Last reviewed: 19 October 2026.