Compliance platforms compete on integration counts. Six hundred providers, eight hundred, a thousand. The number is on the pricing page and in the sales deck, and it is doing a job the underlying reality does not support.
Because "integration" covers at least three different things, and only one of them does the work you are buying.
The three tiers
Tier one: a catalogue entry. The provider has a card in the UI with a logo and a category. Clicking it may open a form. Nothing on the backend knows anything about that provider specifically.
Tier two: a stored connection. You supply an API token or complete an OAuth flow. The credential is encrypted and stored, the integration is recorded, and the platform can assert that a connection exists. That assertion is itself a small piece of evidence — this organisation has an identity provider — but nothing reads the provider's API.
Tier three: automated evidence collection. A provider-specific collector authenticates, calls that provider's API, and turns the responses into per-control evidence: MFA is enforced for 47 of 47 users, branch protection requires review on the default branch of 12 repositories, this bucket is not public. Refreshed on a schedule, attached to controls, timestamped.
Tier three is the product. Tiers one and two are a directory and a password manager.
When we counted honestly across our own catalogue, the split was 51 out of 646. Roughly 8%. The other 92% were real entries you could genuinely connect — and connecting them would not produce a single automated control test.
The sentence that gave it away
The tell in our case was the card description. Every one of the 646 read:
Connect Provider to collect category evidence automatically.
Generated from a template, applied uniformly, and true of 51 of them. Nobody wrote that line intending to mislead — it was a string interpolation in a catalogue builder, written once when the catalogue was small and correct.
That is how these claims usually happen. Not a decision, a default.
What it costs the buyer
The gap surfaces at the worst possible moment.
You scoped your audit preparation assuming the platform collects evidence for your stack. Two weeks out, you discover that your endpoint management, your ticketing system and your cloud storage are tier-two connections. The evidence for those controls is screenshots, taken by a person, dated, and re-taken next quarter.
That is not a small variance. For a first SOC 2 it can be the difference between a fortnight of work and two months, and the plan was built on a number that did not distinguish.
The questions that separate the tiers
If you are evaluating a platform, these get you a straight answer quickly:
- Which specific providers have a dedicated evidence collector? Ask for the list, not the count. A vendor who can produce it immediately has an internal registry, which is a good sign in itself.
- What evidence does it produce for provider X? Named control tests, not "compliance evidence". If the answer is a category rather than a list of checks, it is tier two.
- Does connecting verify the credential works? Storing a token and confirming a token authenticates are different operations. Many platforms show a connection as active the moment it is saved, which means a typo looks exactly like a working integration until someone checks.
- How often does collection run, and what happens when it fails? Evidence that silently stopped refreshing three months ago is worse than no evidence, because it looks current.
- Which of my providers are tier two? The useful answer names them. Then you can plan the manual work instead of discovering it.
What we changed
Three things, and the ordering matters.
We fixed the description, so a card that cannot collect evidence says so. The honest sentence is longer and less appealing, which is the correct trade.
We generated the list from the backend rather than maintaining a copy by hand. The hand-maintained copy had drifted by fifteen providers — four vendors with working collectors were listed as connection-only, so the product was understating its coverage. Drift runs in both directions and only one of them is embarrassing.
We added a check that fails the build when the UI's list and the backend's collector registry disagree. The old file carried a comment saying it must stay in sync. Nothing enforced it, which is precisely why it did not.
That last one is the general lesson. A capability claim rendered from a hand-maintained list will eventually be wrong, and the direction it goes wrong in is not up to you. If the claim matters, derive it from the thing that makes it true, and fail loudly when the two diverge.
The number worth publishing
An integration catalogue is genuinely useful. Being able to record that you use a provider, store the credential, and show an auditor a connection is worth something.
It is just not worth what "646 integrations" implies. The number a buyer can act on is the tier-three count, stated as itself: 51 providers with automated evidence collection; 646 in the catalogue. Less impressive, entirely defensible, and it means the person planning an audit can plan the right one.
