Composite scenario, not a real company. Assembled from common industry patterns; every number is illustrative and its assumptions are listed at the end.
This case belongs to Diagramming AI systems.
Context
A telecom operator lets customers ask about a line on their bill and, if the charge looks wrong, open a dispute. The billing-dispute assistant runs in the customer app. A model reads the question, calls a billing lookup tool to fetch the charge details, explains them in plain language, and can call a second tool to file a dispute. The billing lookup tool reads from an existing billing system that was built for batch work and a few thousand human users. It was never sized for a chat front end.
The team has a container diagram and a working prototype. Before launch, the design review asks for a runtime sequence for the billing lookup, including what happens when the billing system is slow.
Problem statement
The review asks: when the billing system fails, how many calls does one customer question produce? Nobody can say, because the retry settings sit in three different places. Drawing them as a sequence answers it.
Figure 1. Runtime sequence as first designed, with retries in every layer
How to read this: each nested loop is one layer's retry policy, and the note at the bottom is the product of the three loops.
Each retry setting is reasonable on its own: the gateway retries by platform default, the Orchestrator retries tool calls because networks are unreliable, and the Tool server retries because its billing client does. No one decided that 27 was acceptable, because no one saw all three at once.
The measurable problem: with 3 attempts per layer and three layers, one failing call becomes 3 x 3 x 3 = 27 calls. The goal is to cap that factor at 2 and to keep the customer's worst-case wait under 20 seconds.
Constraints
- The billing system cannot be changed or scaled for this project.
- The customer-facing deadline is 20 seconds.
- The assistant must stay honest: a failed lookup must not turn into an invented explanation.
- The dispute tool creates records, so repeating it must not create duplicates.
Why the traditional approach fell short
The previous process was a call centre. Agents read the bill on a screen, so the billing system saw one human-paced request at a time. A rules-based bill-explainer answers a fixed list of charge types and misses the long tail of unusual lines, credits and proration. A self-service page shows the bill and nothing about why. The assistant fills that gap but changes the shape of the load: a model can call the same tool several times in one conversation, and the billing system must absorb every call.
Options considered
| Option | Idea | Verdict |
|---|---|---|
| A. Keep per-layer retries, lower the counts | Two attempts per layer instead of three | Rejected: 2 x 2 x 2 = 8 calls, still multiplicative, and still no owner |
| B. One retry owner, others fail fast | The Orchestrator is the only layer that retries, with backoff and a retry budget. The gateway and the Tool server make one attempt and return a clear overload status | Accepted |
| C. Cache and answer from a nightly extract | The assistant explains charges from a copy of the bill refreshed overnight, with no live call | Kept as the fallback when the billing system is down, labelled with the data's age. For many questions it is the simplest design and removes the dependency |
The default is B because it gives every failed call exactly one decision-maker. Option C is worth building anyway as the degraded mode.
Solution architecture
Figure 2. Retry ownership in the container view
How to read this: each box states its retry rule, solid arrows are live calls, and the dotted arrow is the degraded path.
Figure 3. Runtime sequence with a single retry owner
How to read this: the outer alt separates success from a failed first attempt, the nested blocks show the one budgeted retry and then the degraded answer if nothing came back, and the notes mark the decisions the Orchestrator makes.
How AI solved it, step by step
One customer question, end to end.
- The customer asks, "Why is line 3 charged twice this month?" The API gateway authenticates and forwards the question once. It does not retry this path.
- The Orchestrator builds the prompt and the model replies with a request for the charge-details tool.
- The Orchestrator checks the request and calls the Tool server with a 2-second timeout.
- The Tool server makes one attempt to the billing system.
- If the billing system answers, the Tool server returns the data and the model writes the explanation, citing the charge lines it was given.
- If the call fails, the Tool server returns an overload status. It does not retry.
- The Orchestrator checks its retry budget. If it allows, it waits 0.5 seconds and retries once. If the budget is spent or the retry fails, it reads the nightly copy and tells the customer how old it is.
- The Orchestrator publishes one audit event with the number of attempts and the outcome.
The decision that matters is in step 7, and it is made in one place by code the team owns.
Under the hood
Retries compose multiplicatively. If a layer makes up to a attempts and the layer below it makes up to b, a failing call at the bottom is made up to a x b times, and a third layer with c attempts gives a x b x c. The Google SRE book states the same effect, warning against "amplifying retries by issuing retries at multiple levels", with a case of three layers each allowing 3 retries (4 attempts) giving 4^3 = 64 attempts on the database for one user action. The AWS Builders' Library describes a five-deep stack with three retries at each layer, where independent retries raise the load on the database 243x, and says its best practice is to retry at a single point in the stack. The two sources count differently (the SRE book counts 4 attempts per layer, the AWS article states its factor without the arithmetic), so this page uses its own numbers. Both sources also point to limiting retries: the SRE book suggests a server-wide retry budget (its example is 60 retries per minute in a process) and returning a specific status when overloaded so that clients and other layers back off and do not retry, and the AWS article describes limiting retries locally with a token bucket. The sequence diagram is the tool that shows this: loops nest on the page the way they nest at runtime.
What went wrong / failure modes
- Timeouts stack. With a 2 s call timeout and 3 attempts per layer, the Tool server layer can take 3 x 2 = 6 s, the Orchestrator layer 3 x 6 = 18 s, and the gateway layer 3 x 18 = 54 s. The customer's 20 s deadline passes while the backend is still working on abandoned requests. Mitigation: one retry owner and a total timeout budget.
- The model asks for the same tool repeatedly. A loop of tool calls multiplies the load again. Mitigation: a step limit in the agent loop (see the state diagram in Diagramming AI systems).
- Retry budget starved. During a long outage retries are refused and users see the degraded mode, which is intended.
- Duplicate dispute records. A retried dispute call could file twice. Mitigation: an idempotency key on the dispute tool, reused across attempts.
- Stale fallback data. The nightly copy can be a day old. Mitigation: show the date in the answer and never present it as live.
Results
These are illustrative targets, not measurements. The worst-case calls to the billing system per customer question drop from 27 to 2. At an assumed 40 questions per second, an outage that made every call fail would offer 40 x 27 = 1,080 calls per second to the billing system in the first design, and 40 x 2 = 80 per second in the second. With a retry budget of 10% of calls, retries are capped at 4 per second, giving about 44 per second. The worst-case wait of one lookup falls to 2 + 0.5 + 2 = 4.5 seconds, well under the 20-second deadline. The qualitative result: the retry multiplication was found on a diagram before launch.
Tradeoffs and when NOT to use this approach
- A single retry owner means lower layers cannot recover from a blip on their own; the Orchestrator must be the layer that sees the overload status.
- A retry budget refuses retries during long incidents. That protects the billing system and costs some customers an answer, and the degraded mode must be good enough to carry them.
- If the billing system already handles high concurrency, extra machinery for one lookup may not be worth it. Measure its capacity first.
- Do not use cached data for questions where freshness is the point, for example "was my payment received today?".
Design rules extracted
- Retries have exactly one owner per call path. Every other layer fails fast and returns a clear overload status.
- Draw the sequence with the loops visible and multiply them. If the product is not a number you chose, fix the design.
- Cap retries globally with a budget, not only per request.
- Total timeouts must fit inside the customer's deadline, from the outermost layer in.
- Every action that creates a record carries an idempotency key that survives retries.
Assumptions
- 3 attempts per layer includes the first attempt, so 3 layers give 3 x 3 x 3 = 27 attempts. (The Google SRE example counts 3 retries as 4 attempts per layer.)
- The 2 s call timeout, 0.5 s backoff, 20 s customer deadline, 40 questions per second and the 10% retry budget are chosen for illustration. The 4 retries per second is 10% of 40.
- The timeout arithmetic assumes every attempt fails after the full 2 s, that each layer waits for the layer below to finish all its attempts, and that there is no backoff in the first design.
- In the second design the Tool server's own timeout to the billing system is shorter than the Orchestrator's 2 s, so it can return the overload status in time; the 4.5 s worst case is an upper bound.
- The operator, the billing system and the names of the layers' settings are invented.
Last verified: 2026-10-21