Yes, a wall clock can step backwards, for example at a leap second or after a sync correction. Code that assumes the difference between two timestamps is never negative can fail. Use a monotonic clock for durations, and check for negative results where you cannot.
What happened in 2017
Cloudflare published a post describing a bug at midnight UTC on January 1, 2017. In its DNS software, a number went negative when it should have been at worst zero. That value caused a panic in the code that selects an upstream resolver. The panic was recovered, but some DNS resolutions for some customers failed.
The post says the problem affected customers using CNAME records, and a small number of machines. At peak, about 0.2 percent of DNS queries were affected, and fewer than 1 percent of HTTP requests encountered an error. The most affected machines were patched within 90 minutes and the fix was rolled out worldwide by 06:45 UTC.
The cause, in the company's words, was the belief that time cannot go backwards. The code measured how long a resolver took to answer by subtracting a start time from the current time. When the clock stepped back a second, the measured duration was negative. A smoothing step averaged readings over time, so after a few measurements the smoothed value went negative too, and a random selection function that rejects negative input then panicked.
Why this is a general lesson
Leap seconds are rare, but a clock can also move backwards through time synchronization corrections, virtual machine pauses or manual changes. The bug pattern is common: a measured duration feeds into code that was written assuming it is positive.
| Pattern | Risk | Safer approach |
|---|---|---|
| Subtract two wall clock readings to get a duration | Result can be negative or jump | Use a monotonic clock where your language offers one |
| Feed a duration into a function with a domain limit | Panic, exception or division by zero | Check the range before the call |
| Smooth or average durations | One bad value can poison later values | Discard readings that are out of range |
| Assume the clock always moves forward in tests | Bug never appears | Add tests that step the clock back |
A short audit you can run
- Search your code for places that compute elapsed time by subtracting timestamps.
- For each, ask whether a monotonic source is available. If it is, use it for durations and keep the wall clock for display and logs.
- Where a duration feeds a selection, weighting or sizing function, add a guard against negative and zero values.
- Write one test that moves the clock backwards by a second and confirm nothing panics.
- Check how your services behave when clocks differ between servers. Signed requests, tokens and certificates often have time windows, so skew shows up as authentication failures.
- Make sure your monitoring notices clock drift on each host.
The Cloudflare post itself noted that one precaution is to check whether the difference between two timestamps is negative when using a non-monotonic source, and that its fix let the software discard current performance data and rebuild it after time skipped backward.
What this has to do with continuity
Small, rare conditions can take down a part of a system that otherwise looks healthy. Canary deployment of the fix, in the timeline of that post, let the team confirm a mitigation on one node before widening it. That habit applies to any urgent fix.
A worked example: guarding a latency average
This example is hypothetical, and written in plain steps so it applies to any language. A service measures how long each call to an upstream takes, keeps a smoothed average, and uses the averages to choose which upstream to call, weighting faster ones more heavily.
The risky version looks like this in words: record the wall clock before the call, record it after, subtract, feed the result into the average. If the clock steps back one second during a call that took five milliseconds, the difference is about negative 995 milliseconds. Averaged with a few normal readings, the result can go negative, and a weighting function that expects positive numbers misbehaves.
The safer version has three changes:
- Read a monotonic clock for both readings. Many languages and operating systems offer one that never steps back.
- If the platform offers just a wall clock, check whether the difference is negative. If it is, discard the sample rather than recording it.
- Clamp or validate inputs to the selection function, so a bad value cannot reach code that panics.
| Layer | Safeguard | What it prevents |
|---|---|---|
| Measurement | Monotonic clock | Negative durations from clock steps |
| Recording | Discard samples that are negative or absurdly large | Poisoning the average |
| Use | Validate before calling functions with a narrow domain | A panic or exception |
| Operations | Alert on clock offset between hosts | Silent skew |
Step by step: a clock audit for your codebase
- Search for every place the code subtracts one timestamp from another. Note the language call used.
- Classify each: duration measurement, expiry check, ordering of events, or display.
- For duration measurements, switch to the monotonic source. For expiry checks and display, the wall clock is correct, but think about what a backward step does.
- For ordering of events across servers, do not trust wall clocks to be perfectly aligned. Use sequence numbers or logical ordering where correctness depends on it.
- Add a unit test that injects a clock which goes backward and a clock which jumps forward. Check that nothing crashes and that the result is sensible.
- Add a monitor for clock offset on every host and alert when it exceeds a limit that you choose.
Related time problems worth a look
Leap seconds are one source of trouble. Others include daylight saving transitions, time zone data that is out of date, certificates and tokens whose validity windows depend on a correct clock, and dates stored in fixed width formats. Our posts on the year 2038 problem and clock sync for cloud teams cover two of these.
How the incident response looked
The report's timeline is a good model of a calm response to a surprise. Impact started at midnight UTC, the issue was escalated within ten minutes, confirmed within about half an hour, and a mitigation was deployed to one canary node, then a canary data center, then widened in stages until impact ended a little under seven hours after it began. Staged rollout of an urgent fix is a habit worth keeping even when pressure is high, because a fix can contain its own mistakes.
Try it yourself
Open the fix available now demo and look at how a fix is presented with its context before it is applied. Imagine the fix for a timing bug: which checks and which small first step would you want before widening it?
Click through it at your own pace, or open it full screen.
Quick check
Will a leap second happen again?
Check current announcements from the standards bodies that decide on leap seconds. This post does not make a prediction. Time can step back for other reasons too, so guard against it regardless.
Is the wall clock or the monotonic clock right for logs?
Use the wall clock for timestamps people read. Use the monotonic clock to measure how long something took.
Do I need to worry about this if I use a managed platform?
Your own code still computes durations and uses clocks. Platform providers manage their systems, but a bug in your application can occur wherever it runs, so the audit is still useful.
Sources
Last reviewed: 19 October 2026.
