Deep dives, practical guides, and incident analyses from engineers who build Safeguard. No fluff, no vendor FUD — just what you need to ship secure software.
Griffin AI's evaluation harness results published for the first time. Benchmark methodology, comparison against baselines, and what the numbers mean for production use.
A benchmark you can't reproduce is marketing. A benchmark you can rerun on your own infrastructure is evidence. The reproducibility gap is wide.
Frontier models pass eval benchmarks that open-weight models miss by specific measurable margins. For security workflows, the gap matters.
Evals that run once are marketing. Evals that run on every build are infrastructure. Griffin AI runs the harness on every change; Mythos does not describe one.
Two years ago, AI vendors shipped without evals. In 2026, the posture has shifted. Customers expect benchmarks. Vendors without them lose deals.
Benchmark scores are only as honest as the dataset behind them. Griffin AI publishes golden-dataset design notes; Mythos-class tools rarely explain theirs.
A benchmark that the model has seen in training is a benchmark of memorisation. Specific leakage-testing methods separate generalisation from recall.
Every release risks making the model worse. Griffin AI's regression gates block bad builds before they ship. Mythos-class tools rarely describe a gate process at all.
When the test set is in the training set, the benchmark is broken. Security eval contamination is widespread and the mitigations are specific.
Weekly insights on software supply chain security, delivered to your inbox.