An upgrade passes in staging and breaks production because staging differs from production in data, traffic, configuration, scale or dependencies. Close the cheapest gaps, such as matching configuration and using realistic fake data, and accept that some differences remain, which is why a staged production rollout is still needed.
The usual gaps
| Gap | How it hides an upgrade bug |
|---|---|
| Data shape | Staging has small, clean rows. Production has odd encodings, nulls and huge records the new parser chokes on |
| Traffic mix | Staging sees a script. Production sees every client version and unusual request ever sent |
| Scale | Connection pools, timeouts and memory behave differently under real load |
| Configuration | Feature switches, limits and environment variables differ |
| Dependencies | Staging uses a stub or a different version of a database, proxy or queue |
| Time | Certificates, tokens and scheduled jobs fire on real calendars |
| Network | No real load balancer, TLS termination or proxies in the path |
Google's SRE material makes a related point about canarying: a representative evaluation needs enough size and duration, because stopping after a handful of requests gives little signal for systems with varied traffic. A staging run is a small sample by nature.
Cheap ways to close the gaps
- Generate configuration from one source. Diff staging and production configuration in CI. Differences should be intentional and listed.
- Match versions of infrastructure. Run the same database major version, proxy and queue in staging. Mismatches here cause upgrade failures that no unit test sees.
- Use realistic fake data. Build a generator from production schemas and known edge cases, with no real personal data. See database snapshots as hidden production copies for why copying production into staging creates its own risk.
- Replay sanitized request shapes. Capture the structure of real requests, strip values, and replay them against the upgraded build.
- Add soak time. Leave the upgraded build running through a scheduled job and a token or certificate renewal if the change touches those areas.
Accept the residual and plan for it
Even a good staging environment will not match production. Treat staging as a filter for obvious failures, and production as the final test, done carefully.
- Roll out in stages and compare the new version with a control.
- Keep a rollback ready and rehearsed. See how to roll back a bad dependency upgrade.
- Make the first production slice small and observable.
Martin Fowler's description of a canary release notes that the rollback is simply to route users back to the old version, and that a canary also gives capacity testing in production with a safe way back.
What to log after a surprise
When production breaks something staging passed, add a test or a staging change that would have caught it. Over a year this is how the gaps close. Keep a short list in the repository: date, what broke, what staging was missing, what changed.
Common objections
"Our staging is too expensive to match." You do not need identical scale. You need identical versions and configuration, and realistic data shape.
"We cannot use production data." Correct, and you should not. Build representative fake data instead.
Safeguard cannot make staging match production, but SBOMs generated for each environment's build let you compare dependency versions, which is a common hidden gap.
Quick check
Should I test upgrades directly in production?
Not blind. Test in staging first, then roll out to a small production slice with monitoring and a fast rollback.
What gap causes the most upgrade surprises?
Often it is the version or configuration of a surrounding component, such as a database, proxy or queue. It is also the cheapest gap to close.
Sources
Last reviewed: 19 October 2026.
