What is AI agent governance?
How you decide what an AI agent may do, on whose authority, and how you prove afterwards what it did. With a worked example of authority, approvals and records.
On this page
AI agent governance is the set of decisions and controls that determine what an AI agent is allowed to do, on whose authority, which actions need a person's approval, and what record each action leaves. It covers the agent's whole life: before it runs, while it acts, when it has to stop, and after something goes wrong.
Key takeaways
- Governance attaches to actions. Reading an invoice and changing where a supplier gets paid need different rules, even inside the same agent.
- Governance decides what an agent may do and who answers for it. Security controls enforce those decisions, and observability shows what happened.
- A consequential action should carry five things: authority, identity, approval where required, controlled execution and a record.
- Routine actions can run under a bounded grant set in advance. A one-operation approval covers a specific change and expires.
- An approval request nobody answers should hold or escalate. Silence never counts as a yes.
- An agent owner needs the power to change its access and to stop it. A name in a register is not enough.
What AI agent governance covers
An AI policy document tells people what the organization intends. An agent needs something more operational, because it does more than produce text. It calls tools, reads systems and changes records.
So the useful questions are concrete. Which actions can this agent take? Who granted that? When does a person have to sign off? What happens when nobody does? Can you show, months later, exactly what happened on one run?
Governance overlaps with security and observability. Security includes protecting the agent and its systems, with controls such as identity checks, network rules and target-system permissions helping enforce access decisions. Observability is how you see what the agent actually did, through metrics, traces and guardrail events. Governance is the lifecycle around both: who decided what the agent may do, on what basis, when that decision gets reviewed, and who answers for the result. A well-secured, well-monitored agent can still have no owner and no agreed limits, and then nobody can say whether what it did was allowed.
Agent governance is also different from assessing a model. A model review asks whether a model's outputs are fit for their use. Agent governance asks what the agent is allowed to do with those outputs. In regulated settings you often need both, and the SR 26-2 scope explainer shows how that split looks in banking.
How NIST AI RMF and ISO/IEC 42001 apply to AI agents
Two published frameworks come up as soon as teams set up this lifecycle. The NIST AI Risk Management Framework is a voluntary framework for managing risks from AI. ISO/IEC 42001 sets requirements for an AI management system, meaning the organization-level policies, roles and processes through which a company manages its use of AI.
Neither is written specifically for agents, and following either one does not guarantee that an agent deployment complies with any law or regulation. Both work at the level of the organization's program. The controls on this page sit underneath that program. An owner for each agent, written limits, approvals tied to an operation and records that can rebuild a run are one practical way to show that the program's policies actually reach each agent.
Six control areas to check
These are design goals to verify in your implementation. A registry entry or policy document alone does not establish that a control is enforced.
Identity
Give each agent a distinct authenticated identity and a named owner. Decide how credentials expire and how one agent’s access can be revoked.
Permission scope
Separate read and write access. Check permissions against the workflow’s data and actions, and review them when ownership or purpose changes.
Runtime policy
Check proposed actions outside the model’s reasoning: permitted operations, parameter limits, spend and step ceilings. Test the paths that might bypass the check.
Human approval gates
Choose which actions need a named approver and which can run under a bounded grant. Bind each approval to the target and values it authorizes.
Evidence
Retain the identities, tool arguments, policy decisions, approvals and outcomes needed to reconstruct a run. Verify record integrity and retention in the system you use.
Inventory and cost
Keep a register of agents, workflows, owners and authority. Attribute spend to the work that incurred it. Finding unregistered agents is a separate discovery task.
The five parts of a governed action
Here is the running example for this page. It is illustrative and does not describe a specific customer or product.
An accounts payable agent checks incoming invoices against purchase orders. One supplier's email asks you to send future payments to a new bank account. Checking the invoice is lower risk in this example, though invoices still hold confidential data the agent should read only within its permissions. Changing the bank details changes where money goes, and a wrong change is hard to undo.
For the bank-detail change to be governed, five things need to be true. Each answers a question someone will eventually ask.
- AuthorityWhat may it do?The agent may prepare a bank-detail change. It may execute it only after the required approval.
- IdentityWho is acting?The agent's own identity, plus the person whose request started the run.
- ApprovalWho signs off?A named approver sees who asked, the evidence and the exact change.
- ExecutionIs it checked where it runs?The change runs only with the approved values, checked at the point of execution.
- RecordCan you prove it?The request, decision, approval and result are kept together.
Figure 1 as text
- Authority: what the agent may do. In the example it may prepare a bank-detail change, and may execute it only after the required approval.
- Identity: which agent acted, under its own identity, and which person started the run.
- Approval: a named person sees who asked, the supporting evidence and the exact change, then approves or rejects.
- Execution: the change runs only with the approved values, checked at the point of execution.
- Record: the request, the decision, the approval and the result are kept together for later reconstruction.
Authority is the grant itself: which actions, on which records, up to which limit. A useful way to set it is graduated authority, assigned per action: recommend only, act with approval, or act within a bounded grant.
Identity means the agent acts under its own identity, separate from the person who triggered it and from other agents. With a shared login you cannot tell which agent changed a record, and you cannot revoke one agent without stopping the rest. Record four roles on every run: the initiator, the agent, its owner and the approver.
Approval only works if the approver can see what they are approving. A bare "approve" button gives you a click, but the approver has nothing to judge.
Execution is where the rule is enforced. The check belongs in the path the action takes, outside the agent's own reasoning. An agent that decides for itself whether to respect a limit is a control in name only.
Record is what lets you reconstruct the run. Attempted, blocked and completed actions should each leave a distinct entry, along with the evidence that supported the decision.
Who is responsible, and when
Governance is easiest to run when each stage of the agent's life has a named owner. The stages below are a common pattern. Your organization will have its own names for the forums and roles.
- RegisterPurpose and ownerWhat the agent is for, which workflow it serves, who answers for it.Business owner
- GrantAuthority and limitsActions, data, amounts, approval rules, timeout and fallback.Owner with platform and risk teams
- EvaluateTest before releaseExpected results set in advance, including cases that must be refused.Engineering, reviewed by the owner
- OperateRun with approvalsActions that need approval wait for a named approver. Others run within their grant. Every action is recorded.Approvers in the workflow
- ReviewMonitor and respondError patterns, blocked actions, incidents and access reviews.Owner and platform team
- Change or retireRe-review or removeNew tools, data, model or prompt go back to stage 2. Retiring revokes its access in each system that granted it.Owner
Any change to purpose, tools, data, model or permitted actions sends the agent back to stage 2.
Figure 2 as text
- Register: record the agent's purpose, its workflow and its accountable business owner.
- Grant: the owner, with platform and risk teams, sets permitted actions, data access, limits, approval rules, the approval timeout and its fallback.
- Evaluate: engineering tests the agent against expected results set in advance, including requests it must refuse. The owner reviews the results.
- Operate: actions that need approval wait for a named approver in the workflow, other actions run within their grant, and every action is recorded.
- Review: the owner and platform team watch error patterns, blocked actions, incidents and access reviews.
- Change or retire: any change to purpose, tools, data, model or permitted actions returns the agent to stage 2. Retiring the agent revokes its access in each system that granted it.
The owner is the role most often left vague. In practice that person approves what the agent can reach, answers for its results in the forum where they already answer for the workflow, and can switch it off without filing a ticket.
Limits, evaluation and incident response
Limits
Limits make authority specific. Typical ones are a value threshold, a set of records the agent may touch, a rate or budget, and the data it may send outside. In the example, the agent may read the invoices it is assigned, and may change payment details only after the required approval. Write each limit as a check that can be run. "Use judgment" cannot be enforced.
Evaluation
A convincing answer can still be wrong. Evaluation is how you find out before your customers do. Decide what a correct result looks like before you run the test, keep some test cases apart from the ones used to tune the agent, and re-run the set whenever the prompt, model, tools or code change.
Look at errors by type of case, not only the overall score. A good average can hide a repeated failure on one document layout.
How do you know your AI still works?
Ravi Rajappa, AI Engineer, Moring
Ravi uses an invoice extractor to show how to define a correct result, build a test set with expected answers recorded first, and re-check after every change.
↗Incident response
When something goes wrong, the first questions are the same every time. Which agent acted? On whose request? What else did it touch? Is it still running? The five parts above are what make those questions quick to answer. Without separate identities and records, an investigation starts from a service account in a log and gets rebuilt by hand from tickets and chat history.
Stopping an agent
"Stop the agent" can mean several different actions, and each has a different effect. Decide in advance which one you will use for which situation, and who can trigger it.
Choose the effect you need
Each stop acts on a different part of the agent's access or work.
- 1Block one action typeDoes not undo past actions
Remove one permission. Other permitted work can continue.
- 2Pause new workIn-flight work needs a separate decision
Prevent the agent from picking up new requests.
- 3Revoke credentialsIssued tokens or sessions may survive until rejected or expired
Stop access wherever revocation is enforced.
- 4Retire the agentRetain records under your retention rules
Remove the agent and revoke its access in each granting system.
Detailed stop-level reference
Ways to stop an agent, from narrowest to widest
Swipe or scroll horizontally to see all columns.
| Stop level | What it does | What it does not do |
|---|---|---|
| Block one action type | Removes a single permission, such as changing payment details, while the agent keeps working on everything else. | Undo actions already taken. |
| Pause new work | Stops the agent from picking up new requests. | Cancel an operation already in progress. That needs its own decision. |
| Revoke credentials | Stops new access wherever the revocation is enforced, such as the identity provider, gateways and target systems that check it. | End access everywhere at once. Tokens, sessions or cached credentials already issued may stay valid until a system rejects them or they expire. It also does not stop actions already running or roll anything back. |
| Retire | Removes the agent and revokes its access in each system that granted it. | Delete its records. Keep them for as long as your retention rules require. |
When an agent is doing damage, contain it first. Keep logs and state for the investigation where you can, but evidence collection should not delay stopping a harmful action.
Two details catch teams out. First, stopping new work does not cancel work already underway, and rollback is a separate operation with its own prerequisites. Second, a stop you have never tried is a guess. Rehearse stopping one agent without stopping the others, and keep a dated record that you did.
A safe approval timeout
Every approval step needs an answer to one question: what happens if nobody responds? The safe default is that the operation holds or escalates to a named backup approver. The request never gets approved by default. The AWS Well-Architected Agentic AI Lens describes the same pattern, with timeout policies that hold an operation or route it to an escalation path.
- Approved in timeRun with the approved values onlyApproval and result go in the record.
- RejectedDo not runRecord the rejection and the reason. Notify the requester.
- Details changeAsk againA new target, amount or account needs a new approval.
- No answer by the deadlineHold or escalateRoute to a named backup. Never treat silence as approval.
Figure 4 as text
An approval is requested for an exact change, with who asked, the evidence and a response deadline. There are four outcomes:
- Approved in time: the action runs with the approved values only, and the approval and result are recorded.
- Rejected: the action does not run. The rejection and reason are recorded and the requester is notified.
- Details change: a new target, amount or account needs a new approval.
- No answer by the deadline: the operation holds or is escalated to a named backup approver. Silence is never treated as approval.
The bank-detail request reaches the accounts payable lead at 4 p.m. They are out of office. After the deadline set by the workflow owner, the request goes to the finance controller, who is the named backup. Meanwhile the weekly payment run goes ahead using the supplier's existing, verified details. The deadline length is the owner's call. It depends on what a delay costs compared with what a wrong change costs.
Bounded grants and one-off approvals
Not every action needs a person to approve it. Authority and approval are different instruments, and mixing them up causes problems in both directions.
A grant can repeat. One approval cannot.
A bounded grant authorizes a defined class of actions in advance, under stated conditions. In an illustrative version of the running example, the agent might add an internal triage label to invoices assigned to its queue, using only permitted labels, with a daily limit and each change recorded. Inside those conditions the agent acts without waiting. Outside them, the action is blocked or sent for approval.
A one-operation approval is narrower. It covers one target and one set of values, such as this supplier and this new account number, and it expires. If the target or the values change, it no longer applies, which is why the "details change" branch in Figure 4 asks again.
The two failure modes mirror each other. Sending every routine action for approval trains approvers to click through without looking. Treating one approval as permission for all future actions of that type removes the person from the process without anyone deciding to. Write a grant down as a grant, with its conditions and an owner who reviews it, and keep each approval tied to the operation it covers.
Where to start
Pick one workflow with one consequential action, and do these in order:
- List every action the agent can take, and mark which ones are hard to undo.
- Decide which actions run under a bounded grant and which need an approval each time.
- Name the owner, the approver and a backup approver.
- Write the limits and the approval timeout as rules that can be checked.
- Give the agent its own identity, and remove any shared logins it uses.
- Rehearse stopping it, and write down the date. The AI agent kill-switch explainer goes further into containment.
- Rebuild one past run from the records alone. Whatever you cannot rebuild tells you what to fix next.
For systems where agents delegate work, read agentic AI governance. The AI control plane explainer covers where these rules are decided and where they are enforced in your systems.
The reconstruction test
Can a reviewer rebuild one specific run from the records alone? The record should identify the agent, its authority at the time, the inputs and tool calls, the policy decisions, any human approval and the outcome. The agent’s own summary can help explain the run, but it is not a substitute for those records.
Frequently Asked Questions
What is AI agent governance in one sentence?
AI agent governance is the enforced boundary an autonomous agent runs inside : its identity, permissions, runtime policy, approval gates and evidence trail : so that what it did can be explained and who allowed it can be named.
How is AI agent governance different from model governance?
Model governance evaluates the model: training data, validation, bias, drift and documentation. AI agent governance addresses execution: which tools the agent invoked, what data it retrieved, what it tried to change, whether policy allowed it and which human approved it. A model can pass every model-governance test and still take an action nobody authorised.
Who governs AI agents in an enterprise?
Two roles, and separating them is the point. The business workflow owner is accountable for the outcome the agent supports. Risk, compliance or model risk sets the boundary the agent may not cross and reviews the evidence. A named owner makes accountability explicit; the owner also needs authority to change access and stop the agent.
What happens when an AI agent makes a bad decision?
The record is what makes it answerable: which agent acted, under whose authority, on what inputs, against which policy and with whose approval. Where the agent stayed inside its approved boundary the decision belongs to the approved process and its owner. Where it did not, the record shows exactly where the boundary failed, which is what a remediation and an examiner both need.
Is agent governance the same as guardrails?
No. The term guardrails can include output filters, prompt instructions and execution controls. Check what a specific guardrail enforces. Governance also assigns ownership, sets authority, manages change and records decisions across the agent’s lifecycle.
How do you start governing agents without slowing delivery?
Start with one workflow that already has an accountable owner and a real cost of error, register that single agent properly, and put identity, policy, approval and evidence around it. A governed workflow gives the next team a concrete pattern to adapt and evidence to inspect.
Related reading
Sources
- AWS Well-Architected Framework, Agentic AI Lens, AGENTSEC04-BP02: risk-tiered human approval with context for the approver, and timeout policies that default to a safe fallback by blocking or escalating.
- AWS Well-Architected Framework, Agentic AI Lens, AGENTSEC07-BP04: containment, capturing state, credential revocation and circuit breakers. Background for the stop-level table.
- NIST: AI Risk Management Framework: a voluntary framework for managing AI risks.
- ISO/IEC 42001: requirements for an AI management system.
- Moring: Who is accountable when an AI agent makes a bad decision?: the initiator, agent, owner and approver roles and the owner's obligations.
- Moring: When should a remediation agent stop?: in-flight work and rollback, and explicit fallback when no reviewer responds.
- Moring: Observability for AI agents: linked as further reading on observability.