SAN FRANCISCO — Every January and every summer, a predictable wave hits security inboxes: the "State of Software Supply Chain" report, the "Global Threat Report," the "Cost of a Data Breach" study. They arrive as glossy PDFs, get syndicated into hundreds of trade-press headlines, and end up cited in board decks, RFPs, and occasionally congressional testimony. Few of the people forwarding them around Slack ever open the methodology appendix — if one exists.
That's the gap this piece is about. Not whether vendor-sponsored research is dishonest — the overwhelming majority isn't — but whether it is representative, and whether readers are equipped to tell the difference between an industry-wide finding and a description of one company's customer base.
The genre, and how it's built
Vendor-funded security research has become its own publishing category, sitting somewhere between marketing collateral and applied research. Three structures recur across the genre, and each has a different, well-documented relationship to the truth it claims to represent.
The first is telemetry-as-survey: a security vendor mines data generated by its own product and presents the result as a statement about the industry. CrowdStrike's annual Global Threat Report, for example, is explicitly built from data observed across the Falcon platform's install base — a fact the company states in its own methodology notes. That's a legitimate and often valuable dataset — Falcon sees a lot of real attacker activity — but it is, by construction, a sample of CrowdStrike's customers and the threats that touch their environments, not a random sample of the internet. The same structural pattern applies to any EDR, WAF, or scanning vendor that publishes a "threat landscape" report: the lens is the install base, and the install base is not neutral with respect to company size, industry vertical, or geography.
The second structure is audit-engagement-as-population: Synopsys's long-running Open Source Security and Risk Analysis (OSSRA) report, one of the most widely cited sources on open-source risk, draws its dataset from codebases that came through Synopsys's Black Duck audit services — typically audits commissioned in the context of M&A due diligence. That's disclosed in the report itself, but it's a meaningful selection effect: companies undergoing M&A audits skew toward certain deal sizes, certain industries, and certain motivations for wanting a clean bill of health. A finding like "X% of codebases contain outdated open-source components" describes that audited population, and generalizing it to "the industry" requires an inferential leap the report doesn't fully justify.
The third is component-registry-as-proxy: Sonatype's State of the Software Supply Chain report leans heavily on download and metadata analysis from Maven Central and other public package registries that Sonatype operates or has deep visibility into. Package download counts are a real and useful signal for tracking things like malicious package uploads or dependency freshness, but they are a proxy for developer behavior, not a direct measurement of it — a single CI pipeline can generate thousands of downloads with no human developer decision behind most of them.
None of this means these reports are fabricated or useless. Sonatype has done genuinely useful public-interest work tracking malicious package counts in open-source ecosystems, and OSSRA has usefully pushed the industry to take dependency hygiene seriously. The point is narrower: each report's methodology quietly encodes the sponsor's product category, and the write-up rarely spells out the resulting selection bias in the executive summary — the part everyone actually reads.
A useful contrast: the DBIR
Verizon's annual Data Breach Investigations Report is instructive precisely because it handles the same problem differently. The DBIR is compiled from incident data contributed by dozens of named partner organizations — law enforcement agencies, ISACs, CERTs, and other vendors — mapped to a public, versioned taxonomy (VERIS) that Verizon has published openly for years. It still has real limitations the report itself acknowledges: it can only describe breaches that were detected, investigated, and reported to a contributing organization, which skews toward larger, more mature organizations and jurisdictions with breach-notification laws. But because the contributor list, the taxonomy, and the caveats are public and stable year over year, outside researchers can actually interrogate the methodology rather than take the topline numbers on faith. That transparency is the exception in the category, not the rule — and it's worth noticing that Verizon's own commercial interest in the findings is far more diffuse than a single-product vendor's.
IBM's Cost of a Data Breach Report sits in an interesting middle position: the fieldwork is conducted by the Ponemon Institute, a genuinely independent research firm, under IBM sponsorship. That arrangement buys some methodological distance — Ponemon publishes its own interview-based methodology — but the study is still commissioned and funded by a company that sells breach-response and security services, and the sample is built from organizations willing to disclose detailed cost data to interviewers, which is itself a self-selecting population of the more security-mature.
Why this matters beyond academic nitpicking
The stakes are practical, not just epistemological. Vendor research increasingly feeds directly into procurement justifications, board risk appetite discussions, and public policy. Statistics from these reports get quoted in analyst notes, cited in RFP boilerplate, and repeated in policy discussions about software liability and secure-by-design mandates from bodies like CISA — often stripped of the caveats that would have appeared in the original report's fine print, if it had any. A number computed from one vendor's audit-engagement pipeline can end up, three citations later, presented as a settled fact about "the industry."
There's also a structural incentive worth naming plainly: a security vendor's research arm has a commercial interest in findings that make the problem their product solves look large, urgent, and growing. That doesn't require anyone to lie. It's enough that ambiguous data get interpreted in the direction that supports the narrative, that the scariest year-over-year comparison gets the headline, and that methodology sections stay dense and unread while pull quotes get all the amplification.
A short checklist for reading these reports
Security teams evaluating any vendor-sponsored report — including reports from companies like Safeguard — can apply a few durable questions: What is the underlying population (customers, audit clients, registry downloads, survey respondents), and how does that differ from "the industry"? Is the methodology section detailed enough to reproduce independently, and is raw or aggregated data available for external review? Does the sponsoring company sell a product whose value proposition depends on the finding being large or growing? Is the sample size and response rate disclosed for survey-based work, and is there a control or baseline for comparison? And finally: does the report distinguish between correlation in its own dataset and causal claims made in the summary?
None of these questions require statistical training to ask. They require only the habit of turning to the methodology appendix before the executive summary — and treating its absence as itself informative.
How Safeguard Helps
Safeguard's position on this is straightforward: software supply chain security decisions should rest on evidence your organization can verify directly, not on aggregate claims from a third party's install base. That's the design philosophy behind how Safeguard approaches SBOM generation, dependency risk scoring, and build provenance — every risk signal Safeguard surfaces is traceable back to your own artifacts, your own dependency graph, and your own CI/CD pipeline, not to a vendor's undisclosed sample.
Concretely, Safeguard aligns its provenance and attestation model with open, auditable frameworks — SLSA for build integrity, in-toto for attestations, and standard SBOM formats (SPDX and CycloneDX) — rather than proprietary scoring methodologies that can't be independently reproduced. When Safeguard reports a risk finding about a dependency or a build, the underlying data (the SBOM, the vulnerability match, the provenance chain) is inspectable by your own security team, not obscured behind a vendor's black-box telemetry.
That same standard applies to how we think teams should evaluate any third-party security claim, including our own: ask what population the finding describes, ask whether the methodology is public, and prefer tools that let you verify a claim against your own environment rather than take a report's word for the industry at large. In a field where the research and the sales pitch are so often published by the same hand, the most durable defense is data you can check yourself.