A compliance test can fail even when you changed nothing, because the test depends on things outside your control: a connection that expired, a date that passed, a new account or repository that appeared, a changed scope, or an updated check. Find the cause first, then fix the system or the evidence, never the dashboard color.
TL;DR: Most surprise failures come from seven causes. Work through them in order, from cheapest to check to most involved. Record what you found, because the note itself is useful evidence later.
Definition box. A compliance test is an automatic check that reads a setting or record in one of your systems and reports pass or fail. It is a snapshot of one thing at one moment. If the system, the connection, or the clock moves, the result can move too.
Why would a test fail if I did not touch anything?
It helps to picture the test as a camera pointed at one part of your setup. The camera takes a picture on a schedule. If the scene is the same and the camera works, the picture stays the same. But three things can change without you lifting a finger.
The scene can change. Someone else on your team made a change, a new person joined, or a new repository was created with default settings.
The camera can break. The tool lost its permission to look, so it can no longer see the thing it checks.
The rule can change. The test has a time limit built in, such as "reviewed in the last 90 days," and the clock ran out. Or the vendor updated what the test looks for.
Sorting a failure into one of those three buckets, scene, camera, or rule, is the fastest way to decide what to do next.
What are the seven most common causes?
| # | Cause | Bucket | How to spot it |
|---|---|---|---|
| 1 | The connection expired or lost permission | Camera | Many unrelated tests fail at once for one system |
| 2 | A time based test ran out of time | Rule | Failure says "not completed in N days" |
| 3 | A new account, device, or repository appeared | Scene | Failure lists one or two items you do not recognize |
| 4 | Someone changed a setting | Scene | The audit log of that system shows a recent change |
| 5 | The scope changed | Rule | Test now covers more systems than before |
| 6 | The test definition was updated | Rule | Release notes or a changed description |
| 7 | The source system had an outage or changed its interface | Camera | Test shows "error" or "unknown," not "fail" |
Let us take them one at a time.
1. The connection expired. Most tools reach your systems through a key, called a token, that lets them read data. Tokens can have an end date. GitHub's documentation, for example, recommends giving personal access tokens an expiration, and says tokens unused for a year are automatically removed. If the person who created the connection used a personal token, or left the company, the connection can stop working. When one connection dies, every test that depends on it fails together. That pattern is your clue.
2. A time based test ran out. Some controls have a rhythm. "Access reviewed every quarter" passes for 90 days after a review and then flips to failing. Nothing broke. The calendar moved.
3. A new item appeared. A teammate created a repository, hired a contractor, or enrolled a laptop. The new item has not met the rule yet. The test now fails on one item out of many.
4. Someone changed a setting. A teammate turned off a protection to unblock a release and forgot to turn it back on. The audit log in the source system will show it. GitHub's organization audit log, for instance, lists events from the last 180 days, is limited to organization owners, and can be exported as JSON or CSV.
5. The scope changed. You or a teammate added another system, cloud account, or team to the tool. The test is the same, but it now checks more things.
6. The test was updated. Tool vendors improve their checks. A stricter version of a test can fail where the old one passed.
7. The source was unavailable. If the system the tool reads was down, or changed how it answers requests, the tool may report an error. Treat "error" and "unknown" differently from "fail." Nothing was proven either way.
How do I find out which cause it is?
Use this ten minute routine. Write the answers in a ticket or note as you go.
- Open the failing test and read its detail. Note the exact words, the time of the last pass, and the time of the first fail.
- Look for a pattern. Are many tests failing for one system? That points to cause 1 or 7. Is it one test with one or two items listed? That points to cause 3 or 4.
- Check the connection. Open the tool's integration page and look for a warning, an expired date, or a "reconnect" button.
- Check the clock. Does the test have a time window? Compare the date of the last evidence to the window.
- Check the source system's own log. Look for changes near the time of the first fail.
- Check the tool's change notes. If the test description changed, note the new wording.
- Decide the bucket. Scene, camera, or rule.
- Fix the real thing. Turn the setting back on, renew the connection, complete the review, or add the new item to the rule.
- Rerun the test and keep the result. A fix with no rerun is a claim, not proof.
- Write one line. "Failed on date X because Y. Fixed on date Z by person P." This note is valuable. It shows an auditor that you notice problems and respond.
What should I not do?
Do not edit the test so that it passes. Do not mark the item as an exception just to make the dashboard green. Do not backdate evidence. Auditors look for exactly these patterns, and a clean story about a real failure and a real fix is far better than a spotless dashboard that nobody believes.
If something truly should not be covered by the rule, such as a test repository with no customer data, write down why, who approved it, and when it will be reviewed. Our guide on organizing exception evidence for an audit period shows how.
If the failure is really a broken connection, read automated evidence versus a connection record and what to do when an evidence connection breaks.
A worked example with invented numbers
The numbers below are invented for illustration.
A ten person company has 30 compliance tests. On Tuesday morning, 9 of them fail at the same moment. Every one of the 9 reads from the company's code host.
The owner follows the routine. Pattern check: nine tests, one system. That points to the camera. Connection check: the integration page shows the access token expired on Monday night. The engineer who created it set a 90 day expiration, and 90 days had passed.
The fix takes five minutes: create a new token with read-only access, set a new expiration, and put a reminder on the calendar for 75 days from now. The owner reruns the tests. Eight pass. One still fails.
The last one fails because a new repository created on Sunday does not require review before merging. That is the scene bucket. The owner turns on the protection, reruns the test, and it passes.
The note reads: "Tuesday: 9 failures. 8 caused by expired connection token (renewed, 90 day expiry, reminder set). 1 caused by new repository without review rule (fixed same day)." Total time: about 20 minutes. The gap in automated checks was roughly one day, and the note explains it.
How do I make surprise failures rarer?
Three habits cut most of them.
- Use a shared account for connections, not a personal one, and give it the smallest permission that works. Our read-only access checklist covers this.
- Put expiration dates in a calendar with a reminder two weeks ahead.
- Add a "new repository" and "new hire" step to your onboarding routine, so new items meet the rules from day one. See joiner, mover, leaver evidence.
Where does Safeguard fit?
Safeguard's compliance and evidence automation keeps checks running and keeps the dated results, so you can see when something changed and when it was fixed. On the software side, continuous dependency scanning and SBOM (software bill of materials, a list of the parts inside your software) records show how your code looked on each day, not only on the day of the last audit. You can try a free scan at gold.safeguard.sh/scan.
Try it yourself
Click through it at your own pace, or open it full screen.
Click through the demo and watch how a result changes over time. Notice that something can turn from passing to failing between two checks, with no action from you.
Quick check
Is a failing test always a real problem?
No. It can be a real gap, an expired connection, a time window that ran out, or a tool error. Find the cause before you decide how serious it is.
Should I mark it as an exception so it turns green?
Only if the item truly should not be covered, and only with a written reason, an approver, and a review date. Never use an exception to hide a real gap.
How long should I keep the note about the failure?
Keep it for the whole audit period and any retention time your auditor or contracts ask for. It shows that you detect and fix problems.
Sources
- Managing your personal access tokens, GitHub Docs
- Reviewing the audit log for your organization, GitHub Docs
Last reviewed: 19 October 2026.
