After a bad security quarter, will things improve on their own?

Yes, partly. When a measure bounces around an average, an unusually bad period is usually followed by a more normal one, whatever you do. That is regression to the mean. It can make a new program look like a success when the numbers would have improved anyway.

What it is

Any metric has a stable part and a random part. A quarter with many incidents is likely to have both a high stable part and bad luck. The next quarter keeps the stable part, but the luck resets. So the next number tends to be closer to the long run average.

Illustrative case. A team averages 7 incidents per quarter. One quarter has 12, the worst in two years. Leadership starts a training campaign. Next quarter has 8. Training gets the credit for a drop of 4, but the typical result for a quarter after a 12 is something near 7 with no intervention at all.

Why security is especially prone

• Programs are launched after bad events, which is exactly when the metric is at its extreme.
• Counts are small, so noise is large. See small numbers and noise.
• Rankings of teams, vendors or sites by worst performers, followed by interventions on those, pick the unlucky ones.

The same applies in the other direction. The team with a spotless year is likely to look worse next year, even if it works the same way.

How to avoid fooling yourself

1. Use a long baseline. Compare against the average of eight or twelve quarters, not the single worst one.
2. Use a comparison group. Apply the program to some teams and not others, or stagger the rollout, and compare changes in both. If both improve, regression is a likely explanation.
3. Pick the metric before the intervention. Choosing after you see the data invites cherry picking.
4. Prefer process measures that the program should move directly, such as the share of staff who completed training or the share of repositories with branch protection, in addition to outcome counts.
5. Wait. Judge after several periods, not one.

What it does not mean

It does not mean programs never work, or that a bad quarter should be ignored. Investigate each incident and fix its causes. The caution is about the claim that follows: "the drop proves the program worked". That claim needs better evidence than one quarter before and one after.

Writing it up fairly

A fair statement: "Incidents were 12 in the quarter before the program and 8 in the quarter after. The eight quarter average is 7, so part or all of the change may be normal variation. We will assess again after three more quarters, and we are comparing with teams that have not yet started."

That sentence is less exciting and far more likely to survive a sceptical reader, an auditor, or your own review a year later.

Quick check

Last reviewed: 19 October 2026.