Retry only errors that are likely to pass, with exponential backoff and a hard limit. Set timeouts that match how long a model call really takes, use streaming for long replies, and add a circuit breaker so a provider problem does not tie up your servers or multiply your bill.
Which errors deserve a retry
One provider's documentation lists its error codes and says its official SDK retries transient failures, such as connection errors, rate limits and server errors, with exponential backoff, twice by default, and honors a retry-after header when present. That is a good model.
Retry:
- Timeouts and connection resets
- Server errors such as 500 and 502 to 504
- Overload responses
- Rate limit responses that carry a retry-after value, waiting at least that long
Do not retry:
- Bad request errors. The request is wrong, and sending it again will not help.
- Authentication and permission errors.
- Billing errors.
- A spend cap error. The same documentation states that a monthly cap response has no retry-after header and keeps failing until access resumes.
Backoff with jitter and a limit
Wait a little longer after each failure and add random jitter, so many clients do not retry at the same moment. Cap the number of attempts, and cap total time. A user waiting on a chat reply will not wait through five retries.
If your own code and an SDK both retry, the attempts multiply. Three retries in each layer is nine calls. Pick one layer to own retries.
Timeouts that match reality
Model calls take seconds, and long outputs take much longer. Set a connect timeout short and a read timeout generous. For long outputs, use streaming. The documentation notes that some networks drop idle connections, which can make a long non-streaming request fail without a response, and recommends streaming or a batch interface for long-running requests.
Watch your maximum output setting too. A very large allowance combined with a long timeout holds a worker for a long time.
Idempotency and double charges
A timed-out request may still have run and may still be billed. If the call triggers side effects, such as sending an email, make those steps idempotent with a request ID of your own. Keep the provider's request ID in your logs so you can match charges to calls.
A circuit breaker in plain words
A circuit breaker counts recent failures. When they pass a threshold, it opens and your code stops calling the provider for a short period, returning your fallback at once. After the period, it lets a few test calls through. If they succeed, it closes.
It protects you in two ways. Workers are not stuck waiting on a dead dependency, and you do not hammer a struggling provider. Pair it with the plan in degrade, queue or fail over.
Metrics worth watching
Error rate by code, retry count per request, time to first token, total latency, and tokens per request. A rising retry count is an early warning.
Quick check
Should the client or the server retry?
The server, which can see the error type and the provider headers. Let the client retry only your own endpoint, and with limits.
Sources
Last reviewed: 19 October 2026.
