evals
Safeguard articles tagged "evals" — guides, analysis, and best practices for software supply chain and application security.
24 articles
Leakage Testing Methods For Security Benchmarks
A benchmark that the model has seen in training is a benchmark of memorisation. Specific leakage-testing methods separate generalisation from recall.
Regression Gates: Griffin AI vs Mythos
Every release risks making the model worse. Griffin AI's regression gates block bad builds before they ship. Mythos-class tools rarely describe a gate process at all.
Benchmark Contamination Concerns In Security Evals
When the test set is in the training set, the benchmark is broken. Security eval contamination is widespread and the mitigations are specific.
Refusal Rate Analysis: Griffin AI vs Mythos
A security AI that refuses too often is useless. One that refuses too rarely is dangerous. Griffin AI publishes calibrated refusal benchmarks; Mythos does not.
SEvenLLM Design And Coverage
SEvenLLM set out to measure how well LLMs handle Security Event analysis, the unglamorous day-to-day work of SOCs and IR teams. A design review of what the benchmark covers, how it was built, and where the coverage maps or does not map to real operations.
Citation Accuracy: Griffin AI vs Mythos
An AI security tool that cites the wrong advisory is worse than one that says nothing. Griffin AI benchmarks citation accuracy at 0.89 similarity; Mythos does not.
SecBench Methodology Reviewed
SecBench positioned itself as a comprehensive cybersecurity knowledge and reasoning benchmark for LLMs. A methodology review of its construction, scoring, and the gaps that separate the advertised coverage from what the benchmark actually exercises.
Adversarial Resistance: Griffin AI vs Mythos
Griffin AI reports 98-100% hold rate against adversarial probes. Most Mythos-class tools have never published an adversarial number at all.
SWE-Bench With Security Extensions: Field Review
SWE-bench became the default benchmark for measuring AI coding agents, but the security extensions that were bolted on afterwards deserve their own scrutiny. A field review of what they measure, where they break, and whether you should trust the numbers.
Eval Methodology: Griffin AI vs Mythos
A benchmark number is only as good as the methodology that produced it. Here is how Griffin AI builds its harness and why most Mythos-class tools cannot be audited.
CyberSecEval Reviewed: What It Measures
A working engineer's review of CyberSecEval, the Meta-originated benchmark that has quietly become the default sniff test for AI-for-security claims. What it actually measures, what it misses, and how to read its scores without fooling yourself.
Published Benchmarks: Griffin AI vs Mythos
Griffin AI publishes a five-family eval harness with concrete numbers. Most Mythos-class competitors ask buyers to trust marketing claims instead of data.