Composite scenario, not a real company. Assembled from common industry patterns; every number is illustrative and its assumptions are listed at the end.
This case belongs to Models, routing and inference for AI Architects.
Context
An online retailer adds a support assistant to its app and website. Customers ask about order status, returns, delivery slots, damaged goods and payment problems. The assistant can call two tools: an order lookup and a return-label request. The budget was set from a pilot in which the system prompt was about 1,000 tokens. Before launch the prompt grew to 3,000 tokens (return policy text, tone rules, tool descriptions), traffic reached 80,000 messages a day instead of the planned 45,000, and every message still goes to the largest model the team had approved.
Problem statement
The forecast was 1,305 cost units (cu) a day. The first week's bill implies 3,920 cu a day, which is 3.0 times the forecast. The measurable goal: bring daily model cost to or below the original forecast of 1,305 cu at 80,000 messages a day, with quality on a 300-conversation review set no more than 2 percentage points below the largest-model-only baseline, and 95th percentile time to first text under 3 seconds.
Constraints
- Answers about returns and refunds must match current policy.
- Complaints, legal threats and payment disputes must not be answered by a weaker model.
- Customer data may only go to the approved provider, and there is no new training data.
Why the traditional approach fell short
A rules or FAQ deflector answers known questions instantly and almost for free, but it breaks on mixed messages ("my parcel is late and I want to change the address") and on anything that needs the customer's own order. A classic intent classifier needs labelled examples for every intent and fails quietly on new products and promotions. Human agents handle all of it, at the cost and queue time the assistant was meant to reduce.
Options considered
| Option | Verdict | Reason |
|---|---|---|
| A. Rules and FAQ deflector only (non-AI) | Kept as the first stage | Cheap and exact for about a quarter of messages; cannot handle the rest |
| B. Largest model for every message | Rejected | The current design; 3.0 times the forecast |
| C. Deflector, then a router (rules by risk, small model by default, escalation on low confidence), plus prefix caching | Accepted | Cuts the number of expensive calls and the price of the long prompt |
| D. One small model for everything | Rejected | Fails the rule that sensitive intents never go to a weaker model |
Solution architecture
Figure 1. Components of the routed support assistant
How to read this: the Orchestrator owns the routing decisions, the Model gateway owns the prompt prefix, and the arrow from the small tier back to the gateway is the escalation path.
Figure 2. One message, with the escalation path
How to read this: the outer alt separates the free path from the model path, and the opt block is the only place a message costs two model calls.
How AI solved it, step by step
- A customer writes, "My blender arrived cracked, can I return it?" The API gateway authenticates and passes it on.
- The Orchestrator looks for an FAQ match. There is none, because the message names an order.
- Rules check for sensitive intents (refund exceptions, complaints, legal, payment). A cracked item is a return, so the message goes to the small tier.
- The Model gateway builds the request with the 3,000-token prefix (policy, tone, tool descriptions) first and the order details and conversation last, so the prefix can be served from cache.
- The small tier answers, asks for the return-label tool, and reports a confidence score. The Orchestrator runs the tool.
- If the score is below the threshold, the same request is repeated on the large tier. This is the cascade idea from the FrugalGPT paper: a scoring function rates an answer, and the answer is returned if the score is above a threshold, otherwise the next model is queried.
- The Orchestrator traces the route, tier and cached token count; a sample goes to review.
Under the hood
The key step is prefix caching, in the mechanism described in Under the hood: Prompt caching and prefix stability. As of 2026-10-22, Anthropic's page says a cache covers the prompt in the order tools, system, messages, up to a marked block, and that hits "require 100 percent identical prompt segments". The default lifetime is 5 minutes, refreshed each time the entry is used, and prompts shorter than a per-model minimum are processed without caching and without an error. OpenAI's page says reuse requires the entire rendered prefix to match and advises stable content first, dynamic content last. Both report cached token counts in the response usage, which step 7 records. Neither promises a hit.
Cost then depends on the hit rate. With a write multiplier w, a read multiplier r and a hit rate h, the prefix costs h x r + (1 - h) x w times its normal price. Using the multipliers Anthropic lists for its default 5-minute cache (1.25 for a write, 0.1 for a read, with exceptions for some models), caching beats no caching only when h exceeds (1.25 - 1) / (1.25 - 0.1) = 21.7%. Below that, marking a prefix costs more than leaving it alone.
What went wrong / failure modes
- An unstable prefix. A timestamp or customer name at the top of the prompt changes every request, so the hit rate falls to zero. With every request paying the write multiplier, the day costs 1,356 cu, above the 1,176 cu of the routed design without caching. Mitigation: static content first, an alarm on the cached-token share, and a test that the first 3,000 tokens are identical across requests.
- A prefix below the minimum. Caching is silently skipped. Mitigation: check the model's minimum and alarm on zero cached tokens.
- A confidently wrong small tier. The score is itself a model output. Mitigation: calibrate the threshold on labelled conversations, keep sensitive intents on rules, and sample reviewed answers weekly.
- Escalation creep. At 40% escalation instead of 15% the day costs 916.2 cu, still under the forecast, but 52% of model messages use the large tier (31,200 of 60,000) and escalated ones wait for two calls. Mitigation: track the escalation rate.
- Quiet hours. Low overnight traffic lets the 5-minute lifetime lapse, so the first morning requests pay the write multiplier.
Results
These are illustrative targets, not measurements. Daily model cost at 80,000 messages:
| Design | Daily cost (cu) | Against the 3,920 baseline |
|---|---|---|
| Everything on the large tier | 3,920 | 100% |
| Plus FAQ deflector (25% of messages) | 2,940 | 75% |
| Plus router (32% of model messages on the large tier) | 1,176 | 30% |
| Plus prefix caching at a 90% hit rate | 610.8 | 15.6% |
The final design is 46.8% of the 1,305 cu forecast. At a 50% hit rate it costs 942 cu, still under the forecast. Quality and latency targets are untested until the review set is run.
Tradeoffs and when NOT to use this approach
- Practitioners disagree on whether one mid-tier model with caching is simpler and good enough; at lower volume it often is.
- Escalated messages take two model calls, so their latency is higher.
- At low volume the saving may not pay for evaluation and monitoring. Estimate it with the LLM cost estimator first.
- Do not route sensitive or regulated answers by confidence alone.
Design rules extracted
- Cost the request with the final production prompt, not the pilot prompt.
- Put stable text first and variable text last, and treat the prefix as an interface that tests protect.
- Route by risk with rules first, by confidence second, and send unknown cases up a tier.
- Track cache hit rate and escalation rate, each with a stated break-even.
- Keep the non-AI deflector for what it answers exactly.
Assumptions
Prices are invented cost units (cu) per million tokens, not any provider's prices. Large tier: 10 input, 40 output. Small tier: 1 input, 4 output.
- Pilot: 45,000 messages a day, 1,900 input tokens (1,000 prefix plus 900), 250 output. (1,900 x 10 + 250 x 40) / 1,000,000 = 0.029 cu; x 45,000 = 1,305 cu.
- Launch: 80,000 messages, 3,900 input, 250 output: 0.049 cu each; x 80,000 = 3,920 cu; 3,920 / 1,305 = 3.0.
- Deflector: 25% (20,000) answered with no model call, its own infrastructure cost excluded; 60,000 reach a model.
- Router: 20% (12,000) sensitive, to the large tier; 48,000 to the small tier first, of which 15% (7,200) escalate. Large calls 19,200 (32%), small calls 48,000. Without caching: 19,200 x 0.049 + 48,000 x 0.0049 = 940.8 + 235.2 = 1,176 cu.
- Caching: write 1.25 and read 0.1, the default 5-minute cache in Anthropic's documentation as of 2026-10-22 (other providers and models differ). Prefix cost multiplier = h x 0.1 + (1 - h) x 1.25. At h = 0.9: 0.215, so 3,000 prefix tokens count as 645 and a request has 1,545 input-equivalent tokens: large 0.02545 cu, small 0.002545 cu; 19,200 x 0.02545 + 48,000 x 0.002545 = 488.64 + 122.16 = 610.8 cu. At h = 0.5: 0.675, 2,925 input-equivalent tokens, 942 cu. At h = 0: 1.25, 4,650 input-equivalent tokens, 1,356 cu.
- The 900 history tokens are not cached; the two tiers keep separate caches (assumed: different models compute different key-value tensors, and the cited page does not say how caches are shared across models); the prefix is above the model's minimum.
- Escalation at 40%: 19,200 escalations, large calls 31,200 (52%), cost 31,200 x 0.02545 + 122.16 = 916.2 cu.
- The 15% escalation, 25% deflection, 20% sensitive share and 90% hit rate are invented, as are the retailer, tools and review set.
Last verified: 2026-10-22