Why AI features need a gate
Most engineering orgs ship AI features the same way they shipped non-AI features ten years ago. There is some manual testing, a code review, maybe a feature flag, and then the change goes out. The problem is that AI features fail in ways traditional testing does not catch. A feature can pass all of its unit tests, all of its integration tests, and all of its manual smoke tests, and then behave badly in production because the model started producing different outputs after a system prompt was tweaked.
The gap is closed by an eval harness. An eval harness is a set of test cases, each with an input and an expected behavior, run automatically against the system before every release. The expected behavior is not a single string match. It is a set of rules that the output has to satisfy, expressed in terms the team cares about. The harness produces a pass-fail signal that gates the release, the same way a unit test suite gates a backend release.
This sounds obvious. It is, in fact, what teams that ship AI reliably do. It is also what most teams do not yet have.
What goes into the test set
The test set is the most important part of the harness, and the place where most teams underinvest. A test set that consists of fifty hand-picked examples is better than nothing but is too small to catch the long tail of failure modes. A test set that consists of every interaction the agent has ever had is too large to run on every release.
The pattern that works is a layered test set. The inner layer is a small set of canonical examples, maybe a hundred, that cover the most common interactions and the highest-stakes failure modes. This layer runs on every commit and produces fast feedback. The middle layer is a larger set, maybe a few thousand examples, that covers a wider range of inputs including known edge cases and historical bugs. This layer runs on every release candidate. The outer layer is a sample of recent production traffic, run periodically against new model or prompt versions to catch regressions on real-world inputs.
Each layer is built by a different process. The inner layer is curated by the team. The middle layer is generated by a combination of curation and synthetic expansion. The outer layer is sampled from production with appropriate redaction. The cost of building all three is real, but each layer pays off in different scenarios, and skipping any of them creates a blind spot.
Defining the rubric
A test case needs an expected behavior, but the expected behavior of an AI feature is rarely a single string. It is a rubric. The rubric expresses what a good output looks like in terms the team can agree on. It might require that the output answers the user's question, that it does not contain certain content categories, that it calls a specific tool with specific arguments, or that it stays within a length budget.
The rubric is evaluated by a combination of structural checks, regex checks, and judgment by another model. Structural checks verify that the output has the required shape. Regex checks verify the presence or absence of specific patterns. Model-judged checks verify the harder qualities that resist mechanical evaluation, like whether the output answered the question. Each check produces a pass-fail signal, and the test case as a whole passes only if every check passes.
The model-judged checks have their own failure modes, which is why the harness has to validate the judge model's reliability on a held-out set. A judge that produces inconsistent verdicts is worse than no judge, because it gives the team false confidence in the gate. The validation set for the judge needs to be small enough to maintain by hand and stable enough to give a clear answer about whether the judge is calibrated.
Safety evals are different
Quality evals tell you whether the feature works. Safety evals tell you whether the feature can be misused. They have to be designed differently. A safety eval is not pass-fail in the same sense. It is pass with a known false-negative rate. The job of the safety eval is to surface the cases where the feature behaves badly, with the understanding that the feature will sometimes pass cases it should have failed and the test set has to be expanded over time.
The safety eval should include known prompt injection patterns, known data exfiltration attempts, known jailbreak techniques, and known confused-deputy patterns. The list grows over time as the field discovers new attack patterns and as the team's incident history reveals patterns specific to its workload. The maintenance of the safety eval is an ongoing investment, not a one-time setup.
Running the harness on every release
The harness is only useful if it actually gates releases. That means the CI pipeline runs the inner layer on every commit, the middle layer on every release candidate, and the outer layer on a schedule. A commit that fails the inner layer cannot merge. A release candidate that fails the middle layer cannot deploy. The thresholds are clear, the failures are actionable, and the team has confidence that the gate is doing real work.
The cost of running the harness has to be manageable. The inner layer should run in a few minutes. The middle layer might take longer but should fit within the normal release window. The outer layer can be slower because it runs on a schedule rather than blocking. If any layer takes long enough that engineers route around it, the gate is broken.
Pinning models and prompts
A subtle but important property of the harness is that it pins the model and the prompt versions. When a test case passes, what passed is a specific combination of model, prompt, and code. When any of those changes, the test case has to be re-run. Otherwise, the harness is testing yesterday's configuration and giving today's configuration a free pass.
Pinning is what makes the eval harness compatible with model upgrades. When a new model version arrives, the harness runs against it and produces a clear delta. The team can see which test cases pass on both versions, which pass only on the new version, and which fail on the new version. The decision to upgrade becomes a data-driven one rather than a leap of faith.
Reporting on the gate
A gate that produces no visibility is a gate that cannot be tuned. The harness should produce reports that the team can review. The reports should show pass rates over time, regressions on specific test cases, and the cost of the harness itself. When pass rates drop, the team should be able to see what changed. When test cases regress repeatedly, the team should be able to see whether the test or the system is at fault.
The reporting also feeds into compliance. For regulated workloads, an eval harness with clear reports is part of the evidence package. Auditors want to see that releases are gated, that the gate is real, and that failures are tracked and resolved. A harness that produces reports satisfies that demand naturally. A harness that does not produce reports leaves the team scrambling at audit time.
Common failure modes
The most common failure mode for an eval harness is that it stops being maintained. The test set ages, the rubrics drift, and within six months the harness is testing things that no longer matter and missing things that do. The fix is to treat the harness as a first-class part of the AI feature, with the same maintenance budget as the feature itself. A harness without an owner becomes a harness without value.
The second most common failure mode is that the harness only tests the happy path. Real failures happen on inputs the team did not anticipate. The harness has to include adversarial inputs, edge cases, and known incidents. Otherwise, it tests the system the team thinks it has rather than the system that exists.
How Safeguard Helps
Safeguard ships an eval harness pattern as part of the agent platform, with built-in support for layered test sets, rubric-based grading, and release gating. Pinned model and prompt versions are tracked automatically, judge calibration is monitored, and reports flow into the same audit and compliance views as the rest of the platform. The eval harness stops being a project the team has to build from scratch and starts being a configuration that the platform runs reliably.