Key takeaways
- Governing one agent and governing a system of agents are different problems. The controls that make a single agent defensible — identity, policy gates, records — must survive delegation, or they do not hold at all.
- Three things break first in multi-agent systems: attribution (who did this?), policy consistency (which rules applied?), and blast radius (how far can one bad decision travel?).
- The fix is structural, not procedural: enforcement sits at the gateway outside the agents, identity is issued per agent, and every action across the chain lands in one record.
- Regulated industries already know this pattern — it is the same accountability chain banking applied to automated trading and payments. Agents get the same treatment: attributable, approved, stoppable, recorded.
The one-sentence summary
Most teams govern agents one at a time and discover at the first incident or audit that a system of agents is a different problem: the controls have to survive delegation, and almost none do by default.
Why single-agent governance breaks at fleet scale
Start with the controls that work for one agent. The agent has an identity. A policy decides what it may do. A human approves the irreversible actions. A record is kept. That is a defensible single-agent stack, and it is roughly what governance reviews ask for.
Now connect three agents. Agent A triages a request, agent B drafts the payment instruction, agent C executes it against the treasury system. Ask the questions a governance review or an examiner will ask:
- Who initiated the payment? Agent C executed it, agent B drafted it, agent A decided to act — and behind all three is a user request that may or may not have authorised this. If the record only shows C’s tool call, attribution is gone.
- Which policy applied? If each agent carries its own instructions about what is allowed, the same payment can be judged by three different policy interpretations depending on which agent holds the decision. There is no version of “the policy” to point to.
- Who approved it? If the approval gate lives inside agent B’s prompt, agent C can act without it whenever the chain is arranged differently — a fourth agent, a retry, a re-routing.
- What is the record? Three agents usually mean three log streams, three schemas, and no query that returns the whole chain.
None of these are model problems. They are architecture problems, and they compound: every agent added to the fleet multiplies the number of delegation paths, and every path that is not attributed and recorded is a gap an auditor or an incident will find.
The four controls that survive delegation
1. Identity per agent — and on behalf of whom
Every agent in the fleet gets its own workload identity: its own credential, its own scopes, short-lived, and recorded on every call. Two attributes matter more than the rest:
- The acting agent is always named. The record shows which agent performed the action, not which service account happened to hold the key.
- The original user intent is carried through. When agent A acts on a user’s behalf and delegates to agent B, B’s actions record the on-behalf-of chain. This is the same principle banks applied to payment initiation: the record ends at a human who wanted something, not at the last machine that touched it.
Shared service accounts — the “one key for the whole agent stack” shortcut — destroy attribution. They are the first thing to remove.
2. Policy enforced outside the agents
The same action must be judged by the same policy no matter which agent requests it. That only holds if enforcement sits at a layer that sees every request: the gateway or control plane through which agent actions pass. Concretely:
- Policy decisions are made on the action and its arguments — the payment amount, the counterparty, the data involved — not on which agent asked.
- Policies are versioned. When a policy changes, the record shows which version applied to each historical action.
- Delegated calls are treated the same as direct ones. Agent C’s tool call is evaluated by the same engine that would have evaluated agent B doing it directly.
What this removes is the quiet failure mode of prompt-based governance: agents that carry their rules inside their instructions and drift from them under context pressure. Rules that live in prompts are suggestions. Rules enforced at the gateway are controls.
3. Approval gates that hold across delegation
Human approval on irreversible actions — payments, deletions, external commitments, personal-data processing — has to be requested before execution and recorded with what the approver saw. In a multi-agent system, the gate must be enforced where the action is about to happen, not inside whichever agent happened to draft it. Three properties make the gate real:
- It triggers on the action class, so a delegated request hits it exactly like a direct one.
- It cannot be satisfied by another agent. The approver is a named human with a role, and the record names them.
- It blocks, not warns. An unapproved irreversible action fails closed. A warning that the action can proceed anyway is not a gate.
4. One record per action, across the whole chain
The evidence trail is one correlated record per action: the original request, each delegation, each policy decision with its version, each approval, each tool call, each outcome — signed and tamper-evident. The test that matters: one query returns the whole chain. If reconstructing one workflow requires joining three log stores by hand, the audit fails at the join, no matter how complete each store is internally.
What changes operationally when you adopt this
Fleet-level governance changes four day-to-day things, and each one is a change teams notice within the first month:
- The stop control works at two levels. You can stop one agent, and you can stop a workflow — every agent in the chain — with one action. A bad agent that is instructing others is stopped everywhere at once.
- Onboarding an agent is a registration, not a deployment. Identity, scopes, and policy bindings are declared before the first call. Agents that are not registered do not reach tools.
- Reviews get faster, not slower. Because the record is per action and correlated, answering “what did this agent do last week” is a query, not an investigation. The teams that fear governance overhead are usually running the attribution-by-spreadsheet version of it.
- Regulator conversations change shape. The question “who approved this action” gets answered from the record, in the session. That difference — answered now versus reconstructed later — is most of what examiners are actually testing.
The pattern regulated industries already know
None of this is novel control theory. Automated trading, payment initiation, and straight-through processing all went through the same arc: distributed actors, delegation, and a regulator asking who did what. The industry’s answer was the same four controls — attributable actors, central rules, approvals on the consequential actions, one record. AI agents are a new class of actor, not a new class of accountability. The industries that treat them that way pass their first examination; the ones that treat agents as a software deployment keep discovering the gap at audit time.
Where to start
If you have one governed agent and a fleet forming around it, the sequence that works:
- Register every agent that can call a tool. Identity first — it is the precondition for everything else.
- Move policy enforcement to the gateway for the actions that matter most: money movement, data export, external communications.
- Put approval gates on the irreversible action classes, wherever they occur in a chain.
- Correlate the records so one workflow returns one trail.
- Drill the stop control at fleet level, not only per agent.
The agent readiness checklist covers the single-agent version of this in 36 checks. If you are evaluating platforms for the fleet-level layer, the control-plane scorecard scores build vs assemble vs buy per layer, and how Moring works shows the same four controls implemented as a product.
Where to go deeper
- What ungoverned agents actually cost prices the exposure this framework prevents, in four cost categories.
- The seven frameworks, mapped to four controls shows how the same controls satisfy NIST AI RMF, ISO 42001, the EU AI Act, SR 26-2, DORA, CPS 230, and the NAIC model.
- What FFIEC examiners are asking shows how the per-action record answers the examination questions these controls are built for.
- The board briefing translates this framework into the five questions directors should ask management.
Frequently asked questions
What is multi-agent governance?
Multi-agent governance is the set of controls that keeps a system of AI agents accountable when they act together: each agent carries its own identity, every delegated action is attributable to the agent and user that originated it, policies are enforced at the gateway rather than trusted to each agent's prompt, and the whole chain of actions leaves one evidence trail. It differs from single-agent governance in one decisive way: attribution and approval have to survive delegation between agents.
How is governing a fleet of agents different from governing one agent?
Three things change. First, attribution: when agent A asks agent B to act, the record must show the original user intent, not just B's decision. Second, policy consistency: the same action must be judged by the same policy no matter which agent in the chain requests it, which means enforcement sits outside the agents. Third, blast radius: one bad agent can instruct many, so the stop control has to work at the fleet and workflow level, not only per agent.
What controls matter most for multi-agent systems in regulated industries?
Four, in order: workload identity for every agent so actions are attributable; policy enforcement at the gateway so delegated actions hit the same controls; human approval gates on irreversible actions that survive delegation; and one correlated, tamper-evident record per action across the whole chain. Regulators do not examine agents one at a time — they ask who approved what, and the answer has to survive a chain of delegation.
Do multi-agent systems need human approval gates?
Yes, and the gate has to sit at the layer that sees the action, not inside any one agent's loop. When an agent delegates a high-impact action to another agent, the approval must be requested and recorded before execution, and the record must show who approved and what they were shown. Approval gates buried inside one agent's prompt do not hold when another agent initiates the request.
How do you audit a multi-agent system?
Reconstruct one workflow end to end: pick a completed action, then trace the full chain — original user request, each delegation, each policy decision with its version, each approval, each tool call, and the outcome. If any link in that chain is missing, the audit fails at that link. The test is not whether logs exist somewhere; it is whether one query returns the whole chain.
Governing Autonomous AI Agents in Production
Moring provides the runtime control layer, identity verification, human-in-the-loop approvals, and audit records needed to deploy production-grade AI agents safely.
About the author
Balaji Nagaraj KumarChief AI Officer, Moring AI, Moring AISweden
Leads platform and delivery engineering at Moring AI, turning each engagement into a substrate the customer’s own team operates at handoff. His background spans developer-platform leadership at Volvo Cars and engineering at CERN. He writes about governance for agents that take actions in production systems.
LinkedIn profile