Rollback is not always undo: recovering from agent actions

Key takeaways

• Reverting an agent version can stop a faulty version from causing further problems. It does not undo the records, messages or payments that version has already produced.
• Before launch, assess every write action for safe retries, compensation and irreversible effects.
• Make retries harmless with idempotency, and confirm that the receiving system actually enforces it.
• Design each compensating step in advance, with its own owner, approval and test, because compensation can fail too.
• Put irreversible actions behind an approval, a delay or a volume limit, and keep enough records to identify every action an agent version took.

Reverting an agent’s prompt, model or configuration changes what it does on its next run. It does not recall the messages the previous version already sent, or reverse the records it already wrote.

For example, suppose a prompt change causes an agent to send appointment reminders with the wrong clinic address. Reverting the prompt makes the next reminder correct. The customers who already received the wrong address still have it, and some may already be on their way. Recovery means a second set of actions: finding everyone who received a bad reminder, sending a correction, and perhaps calling the people whose appointments are within the hour. (The example is illustrative.)

That gap between reverting the agent and recovering from what it did is easy to miss in a release plan. Software teams are used to rollback meaning “go back to the last good version”. For an agent that acts on other systems, rollback stops future harm but does not erase past actions.

Assess what recovery means for each action

Distributed systems design already has a vocabulary for this. Microsoft’s description of the saga pattern separates compensable steps, which another step can undo, from a pivot step that acts as the point of no return, and from retryable steps that are idempotent so they can run again safely, meaning the receiving system handles duplicate requests safely and rejects the same key when used with different parameters. You can adapt the same idea to the tools your agent is allowed to call.

These properties can overlap: a booking operation can support both safe retries and cancellation. Assess them per action. The same agent might update an internal note (retry safely), create a calendar booking (compensate) and message a customer (hold before acting). Each lane gets different controls.

Make retries harmless before they happen

Retries are where many duplicate side effects come from. A tool call times out, and the agent or its framework does what any client does: it tries again. The trouble is that a timeout tells you nothing about whether the first attempt succeeded.

AWS describes one common answer in its Builders’ Library article on making retries safe with idempotent APIs. The caller sends a unique request identifier. If the service sees the same identifier again, it returns the original result instead of doing the work a second time. If the identifier arrives with different parameters, the service rejects it as a mismatch rather than guessing which request was meant.

Two practical points follow for agent tools. First, the key has to be generated once per intended action and reused on retries, not regenerated each attempt. Second, the receiving system has to enforce it. Sending a key creates no guarantee on its own, because safe retries depend on how the receiver recognizes and handles duplicate deliveries. If the booking system ignores the key, the retry still creates a second booking.

Design the reverse action before launch

For actions in the compensate lane, write the reverse step while you are designing the forward one. Microsoft’s guidance on the compensating transaction pattern is a useful reminder of why this is harder than it sounds:

• Compensation is specific to the business process. Cancelling a flight may not mean a full refund, so the reverse step carries its own rules.
• It does not necessarily return data to its original state, because other work may have changed the same records in the meantime.
• It can fail. The guidance recommends recording progress so compensation can resume, and making each compensating step idempotent so it can be repeated.
• Sometimes manual intervention is the only way to recover, and the system should raise an alert with enough detail for a person to act.

There is an agent-specific point too. A compensating step is often an outbound action in its own right. A cancellation notice is a customer message. A credit moves money. So the compensating step deserves the same treatment as the original action: a named owner, an approval rule, and a test in your evaluation set that proves it does what you expect.

Hold the actions you cannot take back

The compensating transaction guidance also recommends defining clear points of no return and arranging the workflow so that irreversible steps happen only after the critical checks have passed. For an agent, common controls for irreversible actions include:

Approval

A person confirms the action before it executes. This suits low-volume, high-consequence actions such as releasing a payment.

Delay

Actions queue for a short window before sending, so a bad batch can be stopped. This suits customer messages that are not time-critical.

Volume limits

The agent can take a limited number of these actions per period, so an error stays small while someone notices.

Which control fits depends on how much authority the agent already holds. Deciding which actions earn more independence over time, and which stay gated, is the core of graduated authority for operations agents.

Write the recovery runbook now

When something does go wrong, the order of work matters.

Stop new actions

Pause the agent or revoke the relevant tool. Stopping new work does not cancel an action already in progress, and rollback is a separate operation with its own prerequisites.

List what was touched

Use traces, audit records and downstream system records to identify every record, message and request the affected version produced or changed. Traces may be incomplete, so verify the list against the systems where the actions took effect. If your agent observability cannot answer “what did version X change between 09:00 and 11:00?”, that is the first gap to close.

Decide per record

For each affected item, choose: leave it, compensate it, or escalate it. A person decides; the agent can prepare the list.

Communicate

Tell affected customers and internal teams what happened and what you are doing, and keep incident updates clear about what is and is not yet resolved.

Verify and add a test

Confirm each compensation took effect, then add the failure to your evaluation set so the next version is tested against it.

A double booking after a retry

The runbook is easier to judge against a case. In this fictional one, a scheduling agent books follow-up appointments. During a slow afternoon, the booking system’s API times out on several requests, and the agent’s framework retries them. Some of the original requests had succeeded, so a number of customers now hold two appointments.

The operations lead pauses new bookings. The platform team pulls traces for the affected window and produces a list of bookings created twice for the same customer and time. The agent drafts a cancellation for each duplicate, and the operations lead reviews the list before any cancellation is sent, because a cancellation is itself a customer message. After the cleanup, the team changes the booking tool so each booking request carries a unique key that is reused on retry, and confirms with the booking system’s owners that duplicates are rejected. The duplicate case becomes a permanent task in the evaluation set.

Recovery checklist

• Every write action is assessed for retry safety, compensation and irreversible effects.
• Retryable writes carry an idempotency key, and the receiving system has been confirmed to enforce it.
• Each compensable action has a written compensating step, an owner and a test.
• Compensating steps are idempotent and record their progress so they can resume after a failure.
• Irreversible actions require approval, wait in a delay window, or run under a volume limit.
• The recovery process can identify affected records, messages and requests using traces, audit records and downstream system records where necessary.
• The recovery runbook names who pauses the agent, who decides per record and who communicates.
• Each incident becomes a new case in the evaluation set.

Sources

• Microsoft Azure Architecture Center: Compensating Transaction pattern. Vendor architecture guidance; principles apply beyond Azure.
• Microsoft Azure Architecture Center: Saga design pattern. Definitions of compensable, pivot and retryable transactions.
• AWS Builders’ Library: Making retries safe with idempotent APIs. Vendor engineering article on caller-provided request identifiers.

Frequently asked questions

About the author

Arya SharanDirector of Product Marketing & GTM, Moring AIChennai

Owns product marketing and go-to-market at Moring AI across positioning, demand and enablement. Works with regulated buyers on how governed AI agents are scoped, evaluated and bought. Writes about the operating questions that decide whether an agent reaches production.

LinkedIn profile →