Every model update can make things better or worse. In traditional software, "worse" looks like a crash log. In an AI security product, "worse" looks like a quiet 4-point drop in exploit-hypothesis accuracy that nobody notices until a customer complains two months later.
Griffin AI runs regression gates on every release. A build that regresses past a threshold does not ship. This post explains how those gates work, why a gated release process is a structural trust signal, and why Mythos-class competitors almost never describe one.
What a regression gate is
A regression gate is a CI check that runs the benchmark harness on a candidate model or prompt change, compares the scores against a rolling baseline, and blocks the release if any score regresses past a pre-declared threshold.
For Griffin AI, the gate configuration is:
- Exploit hypothesis: block if accuracy drops more than 2 percentage points from the rolling 30-day median.
- Remediation PR compile rate: block if compile rate drops more than 3 points.
- Advisory summarization similarity: block if similarity drops more than 0.02.
- Cross-finding correlation precision/recall: block if precision drops more than 2 points or recall drops more than 3 points.
- Adversarial resistance: block if hold rate drops below 97% on any family.
The thresholds are not arbitrary; they are set at roughly 2x the week-over-week noise floor, so a regression that trips a gate is almost certainly real rather than noise.
Why gates are hard to fake
Any vendor can claim to have regression tests. A gate is different because a gate changes behavior: it blocks a release. If you see a vendor ship a release on a predictable cadence for two years without a single skipped or delayed build, either their model never regresses (implausible) or their gates do not gate (likely).
Griffin AI has skipped releases. The public changelog shows dates where a planned release was delayed by anywhere from 6 hours to 4 days because a regression gate tripped and needed investigation. Those delays are the signal. A process that never fails is a process that is not enforcing anything.
What Mythos-class vendors describe
In the public documentation of competitors we have reviewed, we have not seen:
- A declared regression threshold for any eval metric.
- A public changelog entry acknowledging a delayed release due to a regression.
- A description of what happens when a benchmark score drops between releases.
- Any mention of a rollback mechanism tied to an eval signal.
A few vendors describe A/B testing or "gradual rollout." Gradual rollout is not a regression gate; it is a blast-radius control. It tells you that a bad release will affect fewer customers for a shorter time. It does not tell you that the release would have been blocked if it was bad.
The rollback story
Gates are half the story. The other half is rollback. When a regression is detected post-release (because benchmarks are imperfect and some regressions only show up in production signal), the question is how fast the bad build can be pulled.
Griffin AI's rollback target is 15 minutes from detection to baseline restoration. The mechanism is a model-version pointer that can be flipped per tenant or globally, backed by a cache of the previous N release artifacts. The 15-minute target is what we measure against; our actual median in the last year is 11 minutes.
Why 15? Because that is roughly the window between a regression becoming user-visible and it becoming an escalation. Longer windows turn into support tickets. A support ticket about a silent model regression is a hundred times more expensive to resolve than a rollback.
Most Mythos-class competitors do not publish a rollback target. Some of them, we suspect, do not have a per-tenant rollback mechanism at all; their "model update" is a ship-forward operation, and the only way to recover from a bad release is another release.
Shadow evaluation
Gates and rollback are the two primary mechanisms, but they are complemented by a shadow-evaluation process.
Before a candidate build enters the gate, it runs in shadow mode against live traffic for 24-72 hours. Shadow mode means the candidate model processes real requests in parallel with the production model, and the outputs are compared. Differences are logged but not served to users.
Shadow traffic catches two kinds of regressions the offline harness misses:
- Distribution shift: the candidate model does fine on the held-out eval set but worse on the current production distribution, because the production distribution has drifted from what the eval set captures.
- Tool-use regressions: the candidate model reasons well in isolation but uses the available tools (retrieval, code search, SBOM lookup) worse than the production model. Tool-use is hard to capture in a pure text eval.
Shadow traffic, combined with the offline gates, catches about 95% of the regressions we would otherwise ship. The remaining 5% is where the rollback mechanism earns its keep.
The cultural layer
A regression-gate process is not purely technical. It is also a cultural commitment: engineering has to be willing to delay a release to fix a regression rather than ship the release and patch later.
That commitment is easy to claim and hard to demonstrate without pain. Every delayed release disappoints someone internal; a PM with a feature waiting, a sales engineer with a customer promise, an exec with a board slide. The gate is only real if it holds under that pressure.
Griffin AI's changelog has 14 release delays longer than 4 hours in the last 12 months. Each one is documented with the benchmark family that tripped, the root cause, and the fix. Those 14 entries are, in some sense, the most trustworthy thing on the changelog, because they are the 14 times we publicly admitted that a build was not good enough.
What buyers should ask
Three questions:
- "What is your declared regression threshold for each published benchmark?"
- "Can you show me a release in the last year that was delayed because a gate tripped?"
- "What is your rollback target from regression detection to production baseline?"
Griffin AI can answer all three with specific numbers. A Mythos-class competitor who cannot answer any of them is shipping to you without a safety net.
Why this is more important than it sounds
The dynamic that makes regression gates so load-bearing is subtle. AI models in production receive continuous implicit updates; retrieval corpora change, upstream advisories update, underlying foundation models version, prompt libraries evolve. A tool that does not gate these changes is a tool that is drifting without measurement.
Drift that is caught by a gate is a blip. Drift that is not caught by a gate becomes a new baseline. Over 12 months, ungated drift can cost a vendor 5-10 points on a benchmark they used to win on. The customer, meanwhile, has no idea. Their experience of the tool has slowly gotten worse.
That is the real cost of not publishing a regression-gate process. It is not that any single release will be bad. It is that the product itself will be quietly worse every quarter, and there will be no mechanism to reverse the trend because there was never a mechanism to detect it.
The bottom line
Gates are where published benchmarks become operational instead of decorative. A vendor that publishes benchmark numbers but does not gate releases on them is decorating a press release. Griffin AI gates releases on published thresholds, documents the delays, and holds a 15-minute rollback target. If your AppSec AI vendor cannot describe the equivalent, you are trusting their model to get better on its own. It will not.