A canary should run until it has seen enough representative traffic to compare against the control, and long enough to cover slow failure modes such as memory growth, scheduled jobs and cache expiry. For many services that means at least one full traffic cycle, but the right length depends on your traffic and your risks.
What the guidance actually says
Google SRE Workbook describes canarying as deploying to a much smaller subset of production and comparing it with a control portion, effectively an A/B process. It says a canary should be sizeable and last long enough to be representative, and that stopping after a handful of queries gives little signal for systems with diverse requests. It also notes that the higher the processing rate, the less time is required.
That gives you two knobs, size and duration, and a rule: they trade off. A larger canary reaches a given sample sooner. A smaller canary needs more time.
Martin Fowler describes the same release pattern as rolling a change to a small subset of users first and, if problems appear, routing users back to the old version.
Set the duration from three things
- Sample size. How many requests, jobs or sessions must the new version handle before a difference in error rate would be visible? Rare endpoints need more time than hot ones.
- Traffic shape. If your load has a daily peak, a canary that never sees the peak has not been tested at it. If a weekly batch job uses the patched library, the canary must include a run of that job.
- Failure speed. Leaks, connection pool exhaustion and certificate or token expiry show up late. If the patched component holds state or connections, extend the window.
| Patched component | Likely slow failure | Extend the bake to cover |
|---|---|---|
| HTTP or database client | Connection leaks, pool exhaustion | Several pool lifetimes |
| Cache or session library | Eviction and expiry behavior | One full expiry cycle |
| Scheduler or queue library | Failures only when a job fires | One run of each job |
| Parser or serializer | Rare malformed input | Enough volume to see rare inputs |
A staged plan you can adapt
- Deploy to a small slice and confirm health checks and a smoke test.
- Hold, and compare the new version's error rate, latency and resource use with the control, per version and not as a global average.
- Step up the share in stages, holding at each stage for the agreed window.
- Promote to everyone only after the window that covers your peak and your scheduled jobs.
- Keep the old version deployable for a defined period after promotion.
Compare per version. The SRE material points out that a small canary population is diluted in overall traffic, so a global error rate can hide a real problem, while a per-version breakdown makes it plain.
Kubernetes Deployments support rolling updates and a minimum ready time per pod, which delays marking a new pod available. That is useful for catching early crashes, but it is not a canary by itself because it does not compare against a control by default. See also canary release metrics for security regressions.
When to shorten it
An actively exploited vulnerability changes the trade. Shrink the window, keep the per-version comparison, and keep a fast rollback. Write down that you shortened it and why.
Quick check
Is one hour enough for a canary?
Sometimes, for a high traffic stateless service and a small change. It is not enough if the change touches a component with slow failures or a job that runs daily or weekly.
Should the canary use real user traffic?
Yes, in a staged way. Real traffic exposes inputs that tests do not, which is the point of a canary.
Sources
- Canarying Releases, Google SRE Workbook
- Canary Release, Martin Fowler
- Deployments, Kubernetes documentation
Last reviewed: 19 October 2026.
