Building a Software Vendor Security Scorecard

Every organization depends on software vendors. SaaS platforms, commercial libraries, infrastructure tools, and managed services are all third-party software with their own supply chains, security practices, and vulnerability histories. When one of these vendors has a security incident, your organization inherits the impact.

Vendor security assessments are supposed to manage this risk. In practice, most vendor assessments are security theater: a questionnaire sent once during procurement, answered by the vendor's sales team, filed away, and never revisited. The questionnaire asks "Do you have a vulnerability management program?" and the vendor checks "Yes." Nothing useful is communicated.

A vendor security scorecard replaces this theater with objective, measurable criteria that track vendor security over time.

Scorecard Dimensions

A useful scorecard evaluates vendors across six dimensions, each weighted according to your organization's risk priorities.

1. SBOM Transparency (Weight: 20%)

Can the vendor tell you what software components are in their product?

Scoring criteria:

• 5: Vendor provides machine-readable SBOMs (CycloneDX or SPDX) for each release, updated with every version
• 4: Vendor provides SBOMs on request within 48 hours
• 3: Vendor provides a component list but not in standard SBOM format
• 2: Vendor can identify major components but not transitive dependencies
• 1: Vendor cannot or will not provide component information

SBOM transparency is the single best indicator of vendor supply chain maturity. A vendor that can produce an accurate SBOM has the dependency tracking infrastructure to respond effectively to supply chain incidents. A vendor that cannot is operating blind.

2. Vulnerability Response (Weight: 25%)

How quickly and effectively does the vendor respond to vulnerabilities?

Scoring criteria:

• 5: Vendor has a published SLA for vulnerability remediation (critical in under 72 hours), consistently meets it, and proactively notifies customers
• 4: Vendor responds to vulnerability reports within 48 hours and releases patches within published timeframes
• 3: Vendor responds to vulnerability reports within one week and releases patches on a regular schedule
• 2: Vendor responds inconsistently, patches are delayed, and communication is reactive
• 1: Vendor has no visible vulnerability response process, patches are infrequent, and communication is minimal

How to measure:

• Review the vendor's security advisory history. How often do they publish advisories? How quickly after CVE disclosure?
• Submit a test vulnerability report (if you have a legitimate finding) and measure response time
• Ask the vendor for their patch release cadence and SLA metrics
• Check CVE databases for the vendor's products and assess remediation timelines

3. Security Certifications (Weight: 15%)

What third-party validation does the vendor have?

Scoring criteria:

• 5: SOC 2 Type II plus industry-specific certifications (PCI DSS, HIPAA, FedRAMP) relevant to your use case
• 4: SOC 2 Type II with clean report
• 3: SOC 2 Type I or ISO 27001 certification
• 2: Vendor has completed a third-party penetration test and shares the summary
• 1: No third-party security validation

Certifications are not proof of security, but they indicate that the vendor has invested in security processes and submitted to external audit. The absence of any certification in a mature vendor is a red flag.

4. Incident History (Weight: 15%)

What is the vendor's track record?

Scoring criteria:

• 5: No known security incidents. Vendor has a public incident response page and transparent disclosure history
• 4: Minor incidents with prompt disclosure and effective response
• 3: Moderate incidents with reasonable response but delayed disclosure
• 2: Significant incidents with poor response or lack of transparency
• 1: Major incidents with evidence of negligence, cover-up, or repeated failures

How to measure:

• Search for the vendor in breach databases and security news
• Review the vendor's own disclosure history
• Check for regulatory actions or lawsuits related to security incidents
• Ask the vendor directly about past incidents and lessons learned

5. Development Security Practices (Weight: 15%)

How does the vendor build their software?

Scoring criteria:

• 5: Vendor follows a documented SDLC with security integrated at every phase (design review, SAST, DAST, SCA, penetration testing) and provides evidence
• 4: Vendor has automated security testing in CI/CD and conducts regular penetration testing
• 3: Vendor conducts periodic security testing and code review
• 2: Vendor relies primarily on reactive security (respond to reports) rather than proactive testing
• 1: Vendor has no visible security development practices

How to measure:

• Ask for the vendor's SDLC documentation
• Request evidence of security testing (sanitized scan results, penetration test summary)
• Check if the vendor participates in bug bounty programs
• Review the vendor's public repositories (if any) for security tooling integration

6. Contractual Commitments (Weight: 10%)

What does the vendor commit to contractually?

Scoring criteria:

• 5: Vendor agrees to notification SLAs for security incidents, provides SBOM access, allows security audits, and includes security-specific SLAs in the contract
• 4: Vendor agrees to incident notification SLAs and basic security commitments
• 3: Vendor includes standard security terms but resists customization
• 2: Vendor's contract has minimal security provisions
• 1: Vendor refuses to include security commitments in the contract

Computing the Score

The overall vendor score is the weighted sum of dimension scores:

Overall = (SBOM x 0.20) + (VulnResponse x 0.25) + (Certs x 0.15) + (Incidents x 0.15) + (DevSec x 0.15) + (Contracts x 0.10)

The result is a score from 1.0 to 5.0. Interpret as:

• 4.0 - 5.0: Strong. Vendor demonstrates mature security practices. Standard monitoring.
• 3.0 - 3.9: Adequate. Vendor meets minimum requirements. Enhanced monitoring recommended.
• 2.0 - 2.9: Concerning. Significant gaps exist. Require improvement plan or consider alternatives.
• 1.0 - 1.9: Unacceptable. Vendor does not meet minimum security standards. Do not proceed without executive risk acceptance.

Ongoing Assessment

Initial assessment is step one. Ongoing monitoring prevents vendor security from degrading unnoticed:

Quarterly: Re-score the vulnerability response dimension based on recent advisory history. Check for new incidents.

Annually: Full re-assessment across all dimensions. Request updated certifications, SBOM samples, and development practice documentation.

Event-triggered: Re-assess immediately when the vendor has a security incident, is acquired, undergoes a major leadership change, or when a new critical vulnerability is disclosed in their product.

Integration with Procurement

The scorecard should be a required input to procurement decisions:

• Vendors scoring below 3.0 require CISO approval for procurement
• Vendors scoring below 2.0 require executive-level risk acceptance
• Vendor scores are included in contract renewal evaluations
• Score trends (improving or declining) inform renewal decisions

How Safeguard.sh Helps

Safeguard.sh supports vendor security assessment by analyzing vendor software for dependency vulnerabilities, SBOM quality, and patch cadence. For vendors whose software is deployed in your environment, Safeguard generates SBOMs and tracks vulnerabilities regardless of whether the vendor provides this data themselves. The platform's vendor scorecard feature aggregates this operational data alongside manual assessment inputs, providing a continuously updated vendor risk profile that informs both procurement and ongoing vendor management decisions.