benchmarks
Safeguard articles tagged "benchmarks" — guides, analysis, and best practices for software supply chain and application security.
22 articles
Citation Accuracy: Griffin AI vs Mythos
An AI security tool that cites the wrong advisory is worse than one that says nothing. Griffin AI benchmarks citation accuracy at 0.89 similarity; Mythos does not.
SecBench Methodology Reviewed
SecBench positioned itself as a comprehensive cybersecurity knowledge and reasoning benchmark for LLMs. A methodology review of its construction, scoring, and the gaps that separate the advertised coverage from what the benchmark actually exercises.
Adversarial Resistance: Griffin AI vs Mythos
Griffin AI reports 98-100% hold rate against adversarial probes. Most Mythos-class tools have never published an adversarial number at all.
AI Safety Eval Datasets as Supply Chain
The datasets you use to evaluate model safety are themselves a supply chain, and almost nobody is treating them that way. A senior engineer's audit of how eval corpora get poisoned, contaminated, and silently drifted.
SWE-Bench With Security Extensions: Field Review
SWE-bench became the default benchmark for measuring AI coding agents, but the security extensions that were bolted on afterwards deserve their own scrutiny. A field review of what they measure, where they break, and whether you should trust the numbers.
Eval Methodology: Griffin AI vs Mythos
A benchmark number is only as good as the methodology that produced it. Here is how Griffin AI builds its harness and why most Mythos-class tools cannot be audited.
CyberSecEval Reviewed: What It Measures
A working engineer's review of CyberSecEval, the Meta-originated benchmark that has quietly become the default sniff test for AI-for-security claims. What it actually measures, what it misses, and how to read its scores without fooling yourself.
Published Benchmarks: Griffin AI vs Mythos
Griffin AI publishes a five-family eval harness with concrete numbers. Most Mythos-class competitors ask buyers to trust marketing claims instead of data.
Mean Time to Remediation Benchmarks: How Fast Should You Be Patching?
MTTR is the most important vulnerability management metric. But what is a good MTTR? Industry benchmarks, realistic targets, and strategies for improvement.
SAST Tool Accuracy Benchmarks 2024: What the Data Actually Shows
Static Application Security Testing tools vary dramatically in accuracy. We analyze detection rates, false positive rates, and language coverage across leading SAST tools using standardized benchmarks.