A passing compliance test proves that one setting or record matched one check at the moment the test ran. It does not prove that people followed the rule every day, that every system was covered, or that the rule itself is good enough. Treat a pass as a useful clue, not a verdict.
TL;DR: Green is good news, but it is narrow news. Before you trust a pass, ask five questions: what exactly was checked, what was left out, how often does it run, what does the rule miss, and who decides if it is enough. The answers decide how much weight an auditor or a customer will give it.
Definition box. A passing test means the automatic check found what it was told to look for. It says nothing about what the check was not told to look for.
Why does a green check feel more certain than it is?
Colors carry feeling. A screen full of green looks like safety. But a test is a small, specific question, and the answer is only as wide as the question.
Imagine a smoke alarm test. You press the button and it beeps. That proves the battery and the horn work. It does not prove there is an alarm in every room, that the alarm is near the kitchen, or that it would catch every kind of fire. A compliance test is the same. It proves the thing it pressed.
The Journal of Accountancy, writing about SOC reports, notes that tools can connect to a company's systems and pull data and evidence into a central, real-time dashboard, which saves time compared with collecting screenshots and spreadsheets. It also notes that most tool providers are not CPA firms and so cannot attest that controls are effective. That is the right way to hold it in your head. The tool collects. An independent professional judges.
What does a pass prove, and what does it not?
| A pass proves | A pass does not prove |
|---|---|
| The setting was on when the test ran | The setting was on yesterday or will be tomorrow |
| The items the tool could see met the rule | Items the tool could not see met the rule |
| The rule, as written in the test, was met | The rule is the right rule for your risks |
| The connection to the system was working | People followed the process in daily work |
| Evidence exists with a timestamp | The evidence tells the whole story |
Read the table row by row. Each pair is a place where a green check can be mistaken for something bigger than it is.
What are the five questions to ask of every pass?
1. What exactly was checked? Open the test and read the description in plain words. "Multi-factor sign-in is enabled for the organization" is different from "every person actually uses multi-factor sign-in." The first is a setting. The second is behavior.
2. What was left out? Look at the scope. Which systems, accounts, and repositories were included? Compare that list with what you really have. A test that passes for 22 of 25 accounts and ignores 3 is not the same as one that passes for all 25.
3. How often does it run? A test that runs once a day tells you more than one that ran at the start of the audit period. Look at the history, not just today's color. If you want to understand why history matters, see what is continuous compliance monitoring.
4. What does the rule miss? Every test has blind spots. A rule that says "pull requests need one approval" does not check whether the approver actually read the change. A rule that says "backups are enabled" does not check that a restore works.
5. Who decides it is enough? You, the tool vendor, and your auditor may all disagree. The tool vendor wrote a general test. You know your business. The auditor decides what they will accept for your report.
How do auditors treat automated evidence?
Auditors usually welcome automated evidence because it is dated and hard to edit. They then add their own work. They may choose a sample of items, such as a few dozen merged code changes out of hundreds, and trace each one back to its approvals. They may ask a person to explain how the process works day to day. They may test whether a control was operating across the full period, not only on one date.
Our guide on how auditors sample evidence explains the idea of a population and a sample, and why a full, accurate list matters.
This means a pass is the start of the conversation. The test result tells the auditor where to look. The sample tells them whether the control works in practice.
A worked example with invented numbers
All numbers below are invented to illustrate the idea.
A company runs this test: "Multi-factor sign-in is required for all accounts in the identity system." It passes every day for a quarter.
An engineer reading the detail notices the scope. The identity system has 25 accounts. The test covers 22 of them. The other 3 are shared service accounts used by automated jobs, which cannot use a phone prompt. The test was set up to ignore those accounts.
Is the pass wrong? No. It is accurate for what it checked. But if the company tells a customer "MFA is enforced for every account," that statement would be too wide. The honest sentence is: "MFA is required for all 22 human accounts. Three service accounts use a different control, which is a restricted key stored in a secrets vault and rotated every 90 days."
Now the company has a better answer. It names the gap, explains the substitute control, and can attach evidence for both. A buyer reading that answer trusts it more than a flat claim of full coverage.
The three service accounts also go into the evidence register with an owner and a review date. See what columns an evidence register should have.
How do I turn this into a habit?
Use these steps once a month, for your most important tests.
- Pick five green tests. Choose the ones a buyer is most likely to ask about, such as code review, access control, and vulnerability handling.
- Read each one in plain words. Write what it checks in one sentence.
- List what is out of scope. If you cannot find the scope, ask your tool vendor or look at the test configuration.
- Write the honest claim. Say what you can truthfully tell a customer, including limits.
- Store the note next to the evidence, with a date.
This takes under an hour. It also prepares you for security questionnaires, where precise wording matters. If you are building answers, see what to say when a customer asks for SOC 2 before you are ready.
Where does Safeguard fit?
Safeguard's compliance and evidence automation keeps dated test results so you can see history, not only today's color. For code, its policies and gates can check pull requests against rules you set, and its dependency scanning and reachability analysis (which looks at whether your code actually uses the vulnerable part of a library) add detail about what a scan result means. Safeguard does not give audit opinions. Your auditor does. You can try the free scan at gold.safeguard.sh/scan.
Try it yourself
Click through it at your own pace, or open it full screen.
Click through the demo and look at how a pass or block decision is made on a pull request. Ask yourself what the check looked at, and what it did not.
Quick check
If every test is green, are we ready for an audit?
Not necessarily. Green tests show your checks passed. Auditors also look at the full period, the scope, written policies, and human decisions. Use the five questions above to find gaps before they do.
Is a passing test useless then?
No. It is useful, dated, hard to edit evidence. It just answers a narrow question. Combine it with scope notes and explanations for a full picture.
Who is allowed to say a control is effective?
An independent auditor gives an opinion in an audit report. Your tool can show the checks passed, but it cannot give that opinion for you.
Sources
- Promises of fast and easy threaten SOC credibility, Journal of Accountancy
- System and Organization Controls: SOC Suite of Services, AICPA and CIMA
Last reviewed: 19 October 2026.
