"Automated evidence collection" is the core promise of every compliance automation platform, and it is described almost everywhere as though it were a single feature. It is not. It is one small program per provider, and the differences between a good one and a bad one are entirely in how they handle things going wrong.
Here is the anatomy.
The shape
A collector is a function with a narrow contract:
TYPES = ["github", "github-actions", "github-enterprise"]
async def collect(token, creds, conn, client) -> list[EvidenceItem]:
...
It receives a decrypted credential and an HTTP client, and returns a list of evidence items. Each item is a control test with a verdict:
{
"slug": "github-2fa-enforced",
"title": "Two-factor authentication required for all organization members",
"status": "valid",
"detail": "2FA enforcement is enabled; 47 of 47 members enrolled"
}
Three statuses, and the third is the one that matters: valid, failing, and pending. Pending means the collector could not determine the answer — the token lacked a scope, the endpoint returned 403, the org is on a plan without the feature. Collapsing pending into failing produces false alarms; collapsing it into valid produces a passed audit based on a question nobody answered.
Where it has to run
Collectors need decrypted credentials, which constrains the architecture more than it first appears.
The service that lists connections for the UI must redact credentials. The service that runs collectors must decrypt them. If those are the same service, one missing redaction in one endpoint exposes every customer's integration tokens.
So collectors live wherever the encryption key lives, they read connection records directly rather than through the redacting API, and that service exposes no route that returns a credential. Boring, and it is the difference between a token leak being possible and impossible.
The stable-slug rule
Each evidence item carries a slug that must be identical across every run and every code path that produces that check.
Get it wrong and a state change writes a new document instead of updating the old one. Now the control shows both "2FA enforced: valid" from last month and "2FA enforced: failing" from today, and which one an auditor sees depends on sort order.
Derive the slug from the check's identity, never from anything variable — not the org name, not the timestamp, not a display string that a product rename will change.
Probe and collect are different operations
A probe answers one question cheaply: does this credential authenticate? A collect does the full evidence run.
Keeping them separate matters because of what a UI shows after you save a connection. If saving marks the connection active without probing, then a typo, an expired token and a working credential are visually identical — and stay identical until the first scheduled collection fails, quietly, somewhere nobody is looking.
The honest states are verified (a probe succeeded), failed (a probe returned an explicit rejection), and unverified (no probe ran). Rendering unverified as verified is what lets a dummy token look connected for a quarter.
The failure modes
Silent staleness. A token expires. Collection fails. The last successful evidence stays in the record, still attached to the control, still looking current. Every evidence item needs a collection timestamp, and the UI needs to surface age, because evidence that stopped refreshing is more dangerous than evidence that was never collected — it looks like coverage.
Paging. An organisation with 3,000 users returns the first 100. A collector that does not page reports "all 100 users have MFA" and is wrong in the direction that passes.
Rate limits. Collectors run on a schedule across many tenants. Without backoff and jitter they synchronise, hit the provider's limit together, and the failure looks like a broken integration rather than a scheduling bug.
Scope creep in reverse. The credential your customer issued has read-only scopes. A new check needs an additional scope. Every existing connection now returns pending for it. That is correct behaviour and it needs a re-consent path, or the check simply never populates and nobody notices.
Prefix matching. The subtle one. A dispatcher that routes a connection type to a collector by string prefix will send google-meet to the Google Workspace collector, which reads the Admin SDK Directory API and knows nothing about video calls. Route by an explicit alias table, and keep a separate list of ids that look like a covered vendor but are a different product with a different API — otherwise someone restores the prefix rule as a "coverage fix" and the wrong collector runs again.
What good looks like
- every check maps to named controls in a documented framework, not to a vague category
- pending is a first-class status and appears in the UI as its own state
- evidence carries a collection timestamp, and stale evidence is visibly stale
- a probe runs on save, and unverified connections say so
- routing is explicit, and a provider without a collector is honestly labelled as connect-only
- adding a provider means adding a module, not editing a dispatcher — because the moment it requires touching shared routing logic, the long tail stops getting built
None of that is glamorous. All of it is the difference between evidence an auditor accepts and a dashboard that looks green.
