Composite scenario, not a real company. Assembled from common industry patterns; every number is illustrative and its assumptions are listed at the end.
This case belongs to Multi-agent and event-driven AI architecture.
Context
The fraud operations team of a retail bank investigates flagged card transactions, about 600 a day. An investigation has six parts: collect evidence from three internal systems, write a summary with a recommended action, place a temporary hold on the card, open a case, tell the customer by text message, and wait for an investigator to confirm fraud (block the card, order a replacement) or clear it (release the hold). Analysts do this by hand in about 25 minutes per alert. The team wants a model to write the summary and a workflow to run the rest.
Problem statement
The pilot ran the steps as one agent loop in the order hold, text message, open case. When the case system timed out after the hold and the text, the customer had been told their card was on hold, the card was held, and no case existed. No investigator ever saw it.
About 0.5% of runs hit this, which is 600 x 0.005 = 3 a day. Each stayed unnoticed for about 2.5 days, so at any moment 3 x 2.5 = 7.5 cards were held with no case. The goal: every run ends either completed or fully undone, and any run that cannot be undone is visible to a person within 5 minutes.
Constraints
- The card platform and the case system are separate services reached only through APIs; there is no transaction that spans them.
- A text message cannot be unsent.
- A temporary hold may last at most 72 hours under the bank's policy.
- A person decides whether fraud is confirmed, never the model.
- A retried step must not hold or block twice.
Why the traditional approach fell short
A manual runbook keeps its progress in the analyst's head and a ticket note; an interruption leaves the half-done state described above. A fixed workflow engine runs the steps reliably but leaves the slowest part, the evidence summary, to people, because merchant descriptors, device data and call notes are unstructured. Rules-based fraud scoring flags the transaction but does not assemble a case. The pilot's agent loop with no stored state remembered what it had done only in its prompt, which is gone when the process dies.
Options considered
| Option | Idea | Verdict |
|---|---|---|
| A. Manual checklist (non-AI) | Analysts follow a runbook in the case tool | Kept as the fallback and the manual queue |
| B. Agent loop with tools, no persisted state | The model decides each next step | Rejected: nothing owns recovery, and the model can skip or repeat a step |
| C. Saga orchestrator, model only for the summary | Code runs the steps, stores state after each, and undoes completed steps if a later one fails | Accepted |
| D. Distributed transaction across both systems | Two-phase commit | Rejected: the two services expose only APIs, and a human wait of hours cannot sit inside a transaction |
The default is C: more code than B, but a vanished investigation becomes a recorded failure.
Solution architecture
Figure 1. Components of the investigation saga
How to read this: the Orchestrator is the only box that talks to everything, the model has no line to any tool server, and the dotted line is observability.
Figure 2. Runtime path with a failed case step and its compensation
How to read this: each tool call carries a key so a retry cannot repeat it, the first alt branch is the forward path, and the second is the undo path.
How AI solved it, step by step
One alert, end to end.
- An alert arrives on the Event bus. The Orchestrator creates investigation INV-77 and stores state STARTED.
- It reads evidence from the three systems with read-only tools.
- The model, which has no tools, summarises the evidence and recommends an action, citing transaction ids.
- The Orchestrator checks that every cited id exists in the evidence. A summary that cites an id that is not there is rejected and the case goes to the manual queue.
- It stores HOLD_PENDING and the hold command in one transaction, then places the hold with key INV-77-hold, and records HOLD_PLACED.
- It opens the case with key INV-77-case. This is the step most likely to fail, so it comes before anything irreversible.
- If the case step fails after three tries, the Orchestrator releases the hold, records COMPENSATED and puts the alert in the manual queue.
- If it succeeds, the Orchestrator sends the text message, now that a case exists, and asks the Approval service for a decision.
- The investigator decides. The Orchestrator applies the decision (block and replace, or release and close) and records the result.
The model made one decision: what to recommend. Every state transition belongs to code.
Under the hood
The microservices.io pattern page defines a saga as a reliably coordinated sequence of local transactions, where completed steps are undone by compensating transactions if the saga fails. In the orchestration style an orchestrator tells the participants which local transactions to execute, as in Figure 2. The page lists the costs: no automatic rollback, so a developer must design the compensating transactions, and no isolation between concurrent sagas. It also notes that a service must update its database and publish a message atomically without a distributed transaction, and offers the transactional outbox: store the message in the same transaction as the business change and have a relay send it. The relay might publish a message more than once, so consumers must be idempotent, which explains the keys in the diagram. Figure 3 is the state machine the store holds.
Figure 3. Saga states
How to read this: the right-hand column is the forward path, the two compensation states catch failures, and NeedsHuman is the only state that raises an alert for a stuck undo.
See Under the hood: Sagas and compensating actions.
What went wrong / failure modes
- The undo fails. The release call times out. The saga retries with backoff, then moves to NeedsHuman and alerts; it never silently stays in Compensating.
- The Orchestrator crashes mid-step. On restart it reads the last stored state and resends the command with the same key, so the hold is not placed twice.
- Two alerts on one card. A second saga could see the first one's hold half-way through. Rule: one active investigation per card; a second alert attaches to it.
- The text message cannot be undone. It is sent only after the case exists; if the case later proves wrong, the compensation is a correction message.
- A made-up transaction id in the summary. The check in step 4 rejects it.
- Instructions hidden in free text. Customer notes and merchant names are untrusted; the model has no tools, so it can only write a summary that a person reads.
- No decision arrives. After 48 hours the saga escalates to a supervisor; the 72-hour hold limit still applies.
Results
These are illustrative targets, not measurements. Cards held with no case fall from a standing 3 x 2.5 = 7.5 to about 3 x (5 / 1,440) = 0.01 on average, because a failed run is undone within minutes; stuck undos surface as NeedsHuman alerts. If analyst time per alert falls from 25 to 8 minutes, 600 x 17 = 10,200 minutes, or 170 analyst hours a day, move from evidence collection to decisions.
Tradeoffs and when NOT to use this approach
- Writing a compensating action per step is real work, and some steps (a message, a payment) have none. Put irreversible steps last.
- State and outbox storage add a dependency to operate.
- Compensation gives eventual consistency, not isolation: a hold may be visible for minutes before it is undone.
- What to do when no decision arrives is a policy choice: auto-releasing a hold protects customers, escalating protects against fraud.
- With one external side effect, a saga is overkill; retry with an idempotency key.
Design rules extracted
- Run multi-step actions as a saga: persist state after each step and define an undo for every step that has one.
- Put steps that cannot be undone last, after every step that can fail.
- Write state and the next command in one transaction, publish through an outbox, and give each command a key.
- A failed undo is an alert state, never a silent one.
- The model summarises and recommends; code owns transitions, and a person owns the irreversible decision.
Assumptions
- 600 alerts a day, a 0.5% partial-failure rate and 2.5 days before a customer calls are invented: 600 x 0.005 = 3 a day, 3 x 2.5 = 7.5 cards.
- Compensation within 5 minutes: 3 x 5 / 1,440 = 0.0104 cards on average (1,440 minutes in a day).
- The 25-minute and 8-minute analyst times are illustrative: 600 x (25 - 8) = 10,200 minutes = 170 hours.
- Three tries, the 48-hour escalation and the 72-hour hold policy are invented settings.
- The saga and outbox statements follow the microservices.io pattern pages, read on 2026-10-21; the bank, its systems and all figures are invented.
Last verified: 2026-10-22