Decide per feature, ahead of time, whether it will degrade, queue or fail over when your LLM provider has an outage. Features that can wait should queue. Features that can work in a reduced form should degrade. Only critical ones justify the cost and risk of a second provider.
Start with a feature list
List every place your product calls a model, then answer three questions for each: what does the user lose if it is down for an hour, can the work wait, and is there a useful reduced version? Most startups find that two or three features matter and the rest can wait.
Option 1: degrade
Offer a reduced version without the model. Examples: show search results without a generated summary, use a rule-based template instead of a drafted reply, or return the last cached answer with a label.
Good for: features where the model adds polish to something that works without it.
Watch for: stale answers shown as fresh. Label them.
Option 2: queue
Accept the request, tell the user it will finish later, and process it when the provider recovers. Batch-style work such as document processing, nightly reports and email drafting fits well.
Good for: asynchronous work that users do not watch live.
Watch for: a queue that grows without a bound. Set a maximum age and a maximum size, and expire old jobs. When service resumes, release the backlog gradually. One provider's documentation warns that a sharp rise in traffic can trigger separate acceleration limits, and advises ramping up gradually.
Option 3: fail over to another provider or model
Route to a second provider or a different model from the same provider. This is the most work and the one most often done badly.
Hidden costs:
- Prompts tuned for one model may behave differently on another, so test the backup, do not assume.
- A second provider means a second contract, data processing terms and sub-processor entry. See the DPA checklist for LLM API vendors.
- Tool calling formats and structured output behave differently across providers.
- Security review must cover both.
Good for: the few features where an hour of downtime costs real revenue.
What an outage looks like in practice
Error codes help you react. One provider documents a 529 error for temporary overload and a 500 for internal errors, and recommends retrying 500s with exponential backoff. A billing or spend cap error looks different and will not fix itself by waiting. Your code should tell these apart. See telling rate limits from spend caps.
Also check whether the problem is the provider or you. Outage or attack: how to tell in the first hour is a useful companion.
Practice it
Run a game day: block the provider's hostname in a test environment and watch what each feature does. Fix what surprises you. Write the three-option decision for each feature in one table that on-call staff can read at night.
Quick check
Is a second provider always worth it?
No. Many teams are better served by queueing and degrading, which cost little and need no second set of contracts.
How long should retries run?
Short and bounded. After a few attempts, move to your fallback instead of piling up requests.
Sources
Last reviewed: 19 October 2026.
