AI Evals in the CI/CD Pipeline: A Governance Control, Not Just a Quality Metric

Most eval tooling on the market is sold and adopted as a quality dashboard — a graph someone glances at during a weekly review. Treated instead as a governance control, wired directly into the pipeline as a merge-blocking gate, it does something a unit-test suite structurally cannot: it catches a model, prompt, or tool-surface change that quietly degrades behavior nobody wrote a deterministic test for, because the failure isn't a broken function — it's a subtler drop in judgment quality that only shows up across a representative set of cases.

This distinction matters because the two framings produce very different organizational behavior. A dashboard gets checked when someone remembers to. A merge-blocking gate gets checked every time, which is the only way an eval suite actually earns the word “governance.”

The authoring session should prove and fix its own work first — targeted tests, a build, a real-browser check, a data read-back — before anyone else looks at it. That loop is necessary but not sufficient: it tells you the change works for the case the session happened to check. A separate, fresh-context verification and a continuous eval suite are what tell you it still works for the cases nobody thought to check today, and will still work after the next model or prompt update ships — which, for a fast-moving provider relationship like an OpenAI partnership, happens on a schedule you don't fully control.

An eval is not a vibe check. It needs a stable fixture, an allowed tool surface, a required outcome, an explicitly forbidden outcome, an objective checker or rubric, and a threshold that isn't quietly lowered when a run fails. Negative cases — the thing the agent must refuse or safely decline to do — deserve the same rigor as the happy path; a policy-safety eval that only tests success cases isn't testing the policy, it's testing that the model can follow instructions when nothing is trying to trip it up.

It's worth being precise about what an eval suite is for, because teams new to this discipline often try to make it do a deterministic check's job, or vice versa. A linter, a type checker, and a unit test are deterministic: the same input always produces the same pass/fail result, and they belong in the fastest, cheapest part of the pipeline. An eval exists specifically because agent behavior is probabilistic — the same prompt against the same model can vary run to run, especially near the edges of its instructions — so an eval's job is to sample that variability against a threshold, not to pretend the behavior is deterministic when it isn't.

Conflating the two produces two failure modes: teams that write brittle deterministic tests against probabilistic output (and then get frustrated when a legitimately-fine response fails a test expecting exact string matches), or teams that treat a flaky eval threshold as a bug to silence rather than a real signal about model variability that a single run can't capture.

The same principle scales down to a much smaller decision, too: whether a rubric-scored eval is trustworthy enough to gate a merge on its own, or whether it needs a sampled human review layered on top. A deterministic checker (does the JSON parse, is the required field present, does a regex match a forbidden pattern) can gate a merge unattended with real confidence. A rubric scored by another model call is weaker evidence — useful for trend-tracking and catching gross regressions, but worth pairing with periodic human spot-checks on a sample of runs before treating its score as merge-blocking on its own, especially early in a suite's life before its calibration against real human judgment has been established.

An internal knowledge assistant is scoped to answer only from approved help articles. A model upgrade subtly changes how it handles an ambiguous question, and it starts occasionally drawing on general world knowledge instead of refusing when the approved articles don't cover the question — a believable, unhelpful-but-not-obviously-wrong failure that a human reviewer skimming transcripts might not catch for weeks. A standing eval with a fixed set of out-of-scope questions and a forbidden-outcome checker (“must not answer from outside the approved corpus”) catches this on the same day the model changes, because the suite runs on a schedule independent of whether anyone remembered to check.

Testing only the happy path. An eval suite that only confirms the agent behaves well when asked nicely tells you nothing about what happens when a user phrases a request ambiguously, adversarially, or just unusually. The forbidden-outcome cases are where most of the real value sits, and they're the ones most often skipped because they're less satisfying to write.

Running the suite manually, on nobody's schedule. An eval suite that only runs when someone remembers to run it isn't a governance control, it's a manual QA step with extra steps. The value comes specifically from it running automatically on every relevant change and on a fixed schedule, so a regression is caught the same day it's introduced rather than whenever someone next thinks to check.

Building fixtures from real customer data. It's tempting to grab an actual production conversation as a realistic test case, but that risks leaking real customer information into a fixture that gets read by engineers, checked into version control, and possibly shared across environments. Synthetic fixtures that capture the same shape of edge case are worth the extra effort to construct.