Composite scenario, not a real company. Assembled from common industry patterns; every number is illustrative and its assumptions are listed at the end.
This case belongs to Reliability and scaling for AI systems.
The billing dispute case is about retries multiplying down three layers. This case has one layer. The damage comes from timing, from ignoring the provider's wait hint, and from no tokens-per-minute headroom.
Context
- After each customer call, a worker asks a model to summarise the transcript into a care note.
- 200 workers share one provider account. The rate limits are set per organisation and shared with other teams.
- At 02:00 a scheduled backfill re-summarises 6,000 older calls for a new analytics field.
- Each task sends 2,500 uncached input tokens and receives 300 output tokens.
Problem statement
The provider measures requests per minute (RPM), input tokens per minute (ITPM) and output tokens per minute (OTPM). The tightest limit sets the pace.
| Limit | Value | Per task | Tasks per minute |
|---|---|---|---|
| ITPM | 400,000 | 2,500 | 160 |
| OTPM | 80,000 | 300 | 266.7 |
| RPM | 1,000 | 1 | 1,000 |
ITPM binds: 160 tasks per minute, 2.67 per second. Live summaries use 72 per minute, so headroom is 88 per minute, 1.47 per second.
At 02:00:00 all 200 workers send a request in the same second. The 429 responses return, and every worker retries after a fixed 1 s, ignoring the wait hint, up to 5 attempts. At most 1.47 x 5 = 7.3 tasks finish out of 1,000 attempts: over 99% of attempts are rejected, about 137 attempts per completed task, and live summaries stall.
Constraints
- The provider's limits are fixed, and ITPM is estimated at the start of each call, so the real use can differ slightly.
- Live summaries must keep flowing during the backfill.
- Results are written to a care-note store, so a repeated task must not create two notes.
Why the traditional approach fell short
Agent-typed notes are slow and uneven, and templates capture fields, not the story of the call. The batch scripts that did the old backfills used a fixed retry delay, which is harmless for one client and harmful for 200 started by one clock. A generic retry library with defaults has no idea what the provider's limit is.
Options considered
| Option | Why accepted or rejected |
|---|---|
| A. Non-AI scheduling: start the backfill with one worker, in the quiet hours | Kept as the stopgap. It removes the storm, but the next job with 200 workers brings it back |
| B. Ask for a higher limit | Rejected as the only fix: more headroom helps, but synchronised retries still waste calls |
| C. Jittered backoff alone | Rejected alone: 200 clients and 1.47 per second of headroom need a window of 136 s |
| D. Shared token bucket in the Model gateway, queue for waiting tasks, retry-after honoured with jitter, one retry owner | Accepted |
| E. The provider's batch interface for the backfill | Worth doing for any job that can wait, since the rate-limit page lists separate limits for it. Live traffic still needs D |
Solution architecture
Figure 1. The gateway paces calls and owns every retry
How to read this: only the gateway retries, and tasks that cannot be served wait in the queue instead of hammering the provider.
Figure 2. One task that meets a 429
How to read this: the first alt separates success from a 429, and the opt blocks are the two ways a rejected call ends.
How AI solved it, step by step
In the starting numbers, the burst of 25 at second 1 admits 10 and holds 15 back. Raise the capacity to 25 and watch the held count at second 1 fall to 0.
Under the hood
| Technique | What the primary source says | Design choice here |
|---|---|---|
| Backoff alone | In AWS's simulation it "helps a small amount, but doesn't solve the problem". The fix is to "add jitter" | Never use fixed or purely exponential delays for a fleet |
| Jitter | Randomness "to spread the retries around in time", because clients that back off to the same time overload again | Random extra on top of retry-after |
| Local token bucket | AWS limits retries "locally using a token bucket" | One shared bucket in the gateway |
| retry-after | The provider docs say to wait that many seconds, and earlier retries will fail | It is the floor of every wait |
Jitter spreads arrivals but only so far. Spreading 200 retries over a window of W seconds gives 200 / W per second: 100 at W = 2, 25 at 8 and 3.33 at 60, against 1.47 per second of headroom. A window of 136 s would be needed, so the bucket and the queue do the pacing and jitter breaks the lockstep. Mechanism: Under the hood: Backoff with jitter.
What went wrong / failure modes
| Failure | Designed mitigation |
|---|---|
| A 429 for the monthly spend cap has no retry-after and never clears | Detect the spend-limit error code, stop retrying, alert a human |
| The SDK's own retries (twice by default) stack on the gateway's | Set the client's retry count to 0 behind the gateway |
| The backfill ramps from zero to full in a second, which can trip acceleration limits | Ramp traffic gradually |
| A rate-limit 429 arrives without retry-after | Fall back to the jittered exponential schedule |
| Task sizes vary, so counting tasks hides token use | Meter estimated input tokens instead of tasks |
| A task is repeated after a requeue | Idempotency key on the care-note write |
Results
These are illustrative targets, not measurements.
| Measure | Fixed retry, no bucket | Target design |
|---|---|---|
| Attempts per completed task during the first wave | about 137 | at most 1.2 |
| Provider ITPM used at the limiter's maximum | uncontrolled | 120 x 2,500 = 300,000 (75%) |
| Backfill speed | stalled | 48 per minute, 6,000 tasks in 125 minutes |
| Live summaries during the backfill | stalled | unaffected |
Tradeoffs and when NOT to use this approach
- The 25% margin is unused capacity. Shrink it only with measurements of the estimation error.
- A central bucket is a shared component and a possible bottleneck, so run it replicated or keep its logic trivial.
- With one client and low volume, a plain exponential backoff is enough.
- Pacing makes the backfill slower. If it must finish sooner, buy more limit instead of spending retries.
Design rules extracted
- Size sustained load below the binding limit and keep the difference as retry headroom.
- Meter calls on the dimension that binds, usually input tokens per minute.
- Wait at least the provider's hint, then add a random extra.
- One component owns retries; everything else fails fast.
- Tasks that cannot be served wait in a queue, not in a retry loop.
Assumptions
- Limits 400,000 ITPM, 80,000 OTPM and 1,000 RPM; 400,000 / 2,500 = 160, 80,000 / 300 = 266.7, so ITPM binds. 160 / 60 = 2.67 per second.
- Live load 72 per minute = 180,000 ITPM (45%). Headroom 160 - 72 = 88 per minute = 1.47 per second.
- Fixed retries: 200 workers x 5 attempts = 1,000 attempts; 1.47 x 5 = 7.3 successes at best, so 99.3% rejected and 1,000 / 7.3 = 137 attempts per success.
- Limiter 2 per second = 120 per minute = 300,000 ITPM (75%) and 36,000 OTPM (45%). Backfill share 120 - 72 = 48 per minute = 0.8 per second; 6,000 / 48 = 125 minutes. The 2 s jitter window, 1.2 target and the telecom are invented.
- Window arithmetic: 200 / 2 = 100, 200 / 8 = 25, 200 / 60 = 3.33, 200 / 1.47 = 136 s.
- The provider is modelled as admitting only the headroom rate; a real token bucket would first pass a burst up to its capacity.
- References as of 2026-10-22: https://aws.amazon.com/blogs/architecture/exponential-backoff-and-jitter/ and https://builder.aws.com/content/3EumjoZascWd1oZiEgL8ORlv3qE/timeouts-retries-and-backoff-with-jitter (AWS Builders' Library), https://platform.claude.com/docs/en/api/rate-limits and https://platform.claude.com/docs/en/api/errors (provider documentation).
Last verified: 2026-10-22