Composite scenario, not a real company. Assembled from common industry patterns; every number is illustrative and its assumptions are listed at the end.
This case belongs to Reliability and scaling for AI systems.
Context
- A retailer's chat assistant answers order-status and product questions in the app and on the website.
- A background job drafts product descriptions and review summaries with the same model.
- All three call one LLM provider through one Model gateway, with one shared pool of 85 concurrent calls.
- A seasonal sale multiplies traffic, and the provider's response time slows at the same hour.
Problem statement
| Measure | Normal | Provider slow |
|---|---|---|
| Model latency per call | 2 s | 20 s |
| Offered load | 28 requests per second | 28 requests per second |
| Calls in flight needed (load x latency) | 56 | 560 |
| Completions per second the 85-slot pool allows | 42.5 | 4.25 |
At 20 s, at most 4.25 of the 28 requests per second can finish (15%), each after about 20 s. Order status, the question shoppers care about most, waits in the same line as description drafts.
Goal: answer every order-status request within 5 s even if the provider is down, and keep background work out of the shoppers' way.
Constraints
- Order data comes from the order service. The model only phrases it.
- Interactive deadline: 5 s. Background results are due within 24 h.
- The team cannot change the provider's capacity or its limits.
- A degraded answer must never claim more than the data shows.
Why the traditional approach fell short
A static order-tracking page works without a model, but cannot answer "will it arrive before Friday if I change the address?". Keyword FAQ search misses paraphrased product questions. A single shared pool sized for the average hides the real flaw: nothing separates a failing dependency from the features that barely need it.
Options considered
| Option | Why accepted or rejected |
|---|---|
| A. Non-AI only: order-tracking page and FAQ search for every request | Kept as the degraded mode. Exact and instant, but cannot answer open questions |
| B. Hot standby with a second provider | Rejected for now: every prompt must be re-evaluated per provider, and behaviour differs. Revisit if outages become frequent |
| C. Bigger shared pool | Rejected: 560 slots would be needed at 20 s, and the provider's own limits still apply |
| D. Bulkhead per feature, queue for background work, degraded mode, status banner | Accepted: failures stay inside one pool and shoppers always get an answer |
Solution architecture
Figure 1. Pools per feature, a queue for background work, and a degraded path
flowchart LR
accTitle: Bulkheads, queue and degraded path
accDescr: The Orchestrator sends order-status and product questions through separate concurrency-limited pools to the Model gateway. Background work goes through the Event bus to a Worker with its own pool. When the provider is slow, the Orchestrator answers from the Tool server and the Retriever without the model.
Client["Client app"] --> GW["API gateway"]
GW --> Orch["Orchestrator"]
subgraph Pools["Bulkheads: concurrency limits"]
B1["Order status: 30 slots"]
B2["Product Q and A: 45 slots"]
B3["Background: 10 slots"]
end
Orch --> B1
Orch --> B2
Orch -->|"enqueue"| EB["Event bus: background queue"]
EB --> Wk["Worker"]
Wk --> B3
B1 --> MG["Model gateway"]
B2 --> MG
B3 --> MG
MG --> LP["LLM provider"]
Orch -.->|"degraded: exact order data"| TS["Tool server"]
Orch -.->|"degraded: keyword search"| RT["Retriever"]
Orch --> TR["Trace store"]
How to read this: each pool is a bulkhead, solid arrows need the provider, and dotted arrows are the paths that still work without it.
Figure 2. One order-status request while the provider is slow
sequenceDiagram
accTitle: Order status request with a slow provider
accDescr: The Orchestrator reads exact order data first. If the order-status pool has a free slot and the breaker is closed, it asks the model to phrase the answer with a 5 second deadline. On a timeout, a full pool or an open breaker, it returns a template answer from the exact data with a status banner. Background work is queued and processed by a Worker in its own pool.
participant C as Client app
participant O as Orchestrator
participant T as Tool server
participant M as Model gateway
participant L as LLM provider
participant E as Event bus
participant W as Worker
C->>O: Where is my order
O->>T: read order status
T-->>O: exact order data
alt slot free and breaker closed
O->>M: phrase the answer, 5 s deadline
M->>L: request
alt answer within 5 s
L-->>M: text
M-->>O: text
O-->>C: phrased answer
else provider slow or error
Note over O,L: Timeout, no retry in this path
O-->>C: template answer plus status banner
end
else pool full or breaker open
O-->>C: template answer plus status banner at once
end
O-)E: enqueue description refresh
E-)W: next item at the paced rate
W->>M: background request from its own 10-slot pool
M->>L: request
How to read this: the outer alt is the decision to call the model at all, the inner alt is the deadline, and both failure branches end in the same template answer.
How AI solved it, step by step
Under the hood
Two ideas carry the design. Little's law links load, time and slots. Pools and queues keep one slow dependency from taking everything.
| Idea | What the primary source says | What it means here |
|---|---|---|
| Little's law | Average number in the system L equals arrival rate times average time in the system | 28 per second x 20 s = 560 calls in flight |
| Bulkhead | Isolate elements into pools so a failure in one does not stop the others. Consider combining with retry, circuit breaker and throttling | Each feature gets its own slots |
| Queue-based load leveling | A queue buffers tasks so the service works at its own pace. If the producer rate exceeds the consumer rate, the queue and latency grow | 10 slots drain 5 per second against 3 arriving |
Mechanism: Under the hood: Bulkheads and queue-based load leveling.
What went wrong / failure modes
| Failure | Designed mitigation |
|---|---|
| A pool is sized too small and rejects normal peaks | Size each pool at 1.5 to 1.7 times normal in-flight calls (here 20, 30 and 6 normal against 30, 45 and 10 slots) and alert on rejections |
| The background queue grows without bound | Cap the backlog at 20,000 items and let producers drop the lowest-priority refreshes; alert on queue age |
| Queue delivery repeats a message | Make the consumer idempotent, keyed by product id and description version |
| The breaker flaps between open and closed | Let one probe request through when the 20 s open period ends |
| The provider returns an overloaded or timeout error instead of being slow | Treat HTTP 529 and 504 like a timeout: degrade, do not retry in the interactive path |
| The template answer is stale | Build it only from live order data and show the banner |
Results
These are illustrative targets, not measurements.
| Measure | Shared pool | Target design |
|---|---|---|
| Order-status requests answered within 5 s while the provider is slow | 0 of 10 per second (each takes about 20 s) | 10 of 10 per second, with exact data and a banner |
| Background backlog after a 90-minute slowdown | Competes with shoppers | 13,500 items, cleared 112.5 minutes after recovery |
| Order-status pool in-flight calls at the 5 s cap | Shared with all features | At most 30, then fail fast |
Tradeoffs and when NOT to use this approach
- Bulkheads waste capacity: an idle pool cannot lend slots. Skip them for a system with one feature.
- Degraded mode is extra code to build and to test. Rehearse it, or it fails when it is needed.
- A queue cannot serve a caller who needs a live answer; use it only for work with a due time of hours.
- A second provider may beat this design when outages are frequent and prompts are simple.
Design rules extracted
- Give each feature its own concurrency limit, so one slow dependency cannot use every slot.
- Compute slots as load times latency, and plan for the slow case, not the normal one.
- Put work that is due in hours behind a queue with a backlog cap.
- Define the degraded answer before the incident, from data the model does not own.
- Fail fast when a pool is full: a rejected request that degrades beats one that waits.
Assumptions
- Offered load: order status 10, product Q and A 15 and background 3 requests per second, total 28.
- Normal latency 2 s gives 10 x 2 = 20, 15 x 2 = 30 and 3 x 2 = 6 calls in flight, 56 in total. At 20 s: 200 + 300 + 60 = 560.
- Shared pool 85 slots: 85 / 2 = 42.5 and 85 / 20 = 4.25 completions per second; 4.25 / 28 = 15%. Bulkheads 30 + 45 + 10 = 85.
- Interactive timeout 5 s; at 20 s latency every model call times out. Order status would need 10 x 5 = 50 slots, so the pool fills and later requests degrade at once.
- Background consumer: 10 slots / 2 s = 5 per second normally and 10 / 20 = 0.5 per second when slow. Backlog growth 3 - 0.5 = 2.5 per second x 5,400 s = 13,500 items. Drain rate 5 - 3 = 2 per second, so 13,500 / 2 = 6,750 s = 112.5 minutes.
- The breaker thresholds (5 timeouts in 30 s, 20 s open), the 20,000 backlog cap and the retailer are invented.
- Little's law: Little, "Little's Law as Viewed on Its 50th Anniversary", Operations Research 59(3), 2011. Bulkhead pattern: https://learn.microsoft.com/en-us/azure/architecture/patterns/bulkhead. Queue-based load leveling: https://learn.microsoft.com/en-us/azure/architecture/patterns/queue-based-load-leveling. Retry pattern, degraded functionality: https://learn.microsoft.com/en-us/azure/architecture/patterns/retry. Error codes as of 2026-10-22: https://platform.claude.com/docs/en/api/errors.
Last verified: 2026-10-22