A release gate for prompt, model and tool changes
The trap: you fix one failure, the failure rate drops, and something else breaks quietly. A banking agent kept escalating straightforward cases; the prompt was tuned, handoffs fell, and it looked like a win — until cases with missing records started passing automatically.
The measure that catches this is not how many failures were fixed. It is whether the right cases still reach the human. This gate tracks both numbers and refuses the release when routing recall falls.
What is inside
| Section | What it covers |
|---|---|
| Fix rate | Of the cases that failed before the change, the share that pass now. |
| Routing recall | Of the cases that should reach a human, the share that did. This is the number the release gates on. |
| Routing precision | Of the escalations, how many were needed — the check that you have not simply escalated everything. |
| Release checklist | Version pinning, the eval set, the rollback version, and who signs the promotion. |
| Working script | A comparison script you can run over two evaluation runs before promoting. |
How to use it
- Build a labelled set of cases that must escalate. Twenty is enough to start; it grows as misses are found.
- Record both numbers on every change, including changes that are supposed to be cosmetic.
- Gate on recall. Treat fix rate as a dashboard number, not a pass condition.
- Keep the previous version promotable so a rollback is one action, not one release.
- Add every production miss to the set, so the gate gets harder over time rather than easier.
How do you release a change to an AI agent safely?
Why gate on routing recall instead of accuracy?
Because accuracy hides the expensive failure. An agent that auto-resolves a case it should have escalated looks accurate on aggregate and produces the loss individually. Recall on the must-escalate set is the only number that moves the wrong way when you over-tune.
How large does the evaluation set need to be?
Large enough to contain the rare case you care about. Twenty hand-labelled must-escalate cases catch most regressions in a single workflow; a few hundred give you per-category recall. Start at twenty and grow it from production misses.
Does this apply to model upgrades too?
Yes — a model change is a release. Point releases and provider-side updates both move behaviour, which is why the version has to be pinned and the gate re-run before promotion.