Failure Drill
A failure drill is a rehearsed, timed exercise that tests whether agent controls actually work: stop the agent, verify it stopped, list everything it did, confirm every action has a rollback path, and prove restart requires recorded approval.
What a failure drill means
Agent governance claims are cheap; drills are how they get tested. The failure drill takes one agent, one environment, and about 45 minutes: write down the agent, who may pull the stop, who may restart it (a different person), the target time from stop-decision to last action, and where the record lives. Then run five steps — stop new actions (revoke credentials or block at the gateway, never trusting the agent's own code), stop in-flight work (drain queues and retries; zero actions after the stop), list everything the agent wrote in the window (complete, with before/after values, in minutes), decide what to undo (each write marked keep, auto-rollback, or hand to a named person — and actually reverse two of them), and restart under approval (blocked until the named approver signs off). Score four numbers: time to last action, actions after stop, minutes to the full write list, and whether unauthorized restart was blocked. The drill converts governance from a diagram into a rehearsed capability — the same transformation fire drills give building safety. Run it in staging first, then in production during a quiet window with the owner present.
Frequently asked questions
What is a failure drill for AI agents?
A timed rehearsal of the worst day: stop the agent, prove it stopped, list what it did, confirm rollback paths, and test approval-gated restart. It produces four measurable scores and usually finds the gaps — unowned writes, in-flight work nobody drains, restart without approval.
How often should teams run failure drills?
At minimum before each agent's production promotion, after significant changes to its tools or permissions, and periodically (quarterly is a common cadence) for agents in regulated workflows. The drill is cheap — 45 minutes — compared to the unplanned version of the same event.