Composite scenario, not a real company. Assembled from common industry patterns; every number is illustrative and its assumptions are listed at the end.
This case belongs to Cost and unit economics for AI Architects.
Context
A bank pilots an internal research assistant for credit analysts: it answers questions about a borrower by calling tools (filing extracts, covenant data, internal ratings). The pilot had 50 users. The rollout reaches 5,000 staff, and the model works in a loop of several calls per task.
Problem statement
The forecast scaled the pilot cost per task by the number of users. Production cost 19.7 times the forecast.
| Measure | Forecast | Production |
|---|---|---|
| Tasks per month | 440,000 | 880,000 |
| Cost per task (cu) | 0.046 | 0.453 |
| Monthly cost (cu) | 20,240 | 398,640 |
Goal: cost per task at or below 0.20 cu, a ceiling set by the business value of one task, with answer quality on a 200-task review set within 2 points of today.
Constraints
- Answers must cite the tool result they use.
- Customer data goes only to the approved provider.
- Analysts will not accept a visible drop in answer completeness.
Why the traditional approach fell short
A spreadsheet forecast (users x tasks x cost per task) treats a task as one call. Saved searches and report templates answer known questions but cannot follow a finding into a second source. Manual research is accurate and slow, which is what the pilot users wanted to escape.
Options considered
| Option | Why accepted or rejected |
|---|---|
| A. Search and report templates, no model (non-AI) | Rejected as the whole plan: it cannot synthesise across sources |
| B. Keep the agent as shipped and raise the budget | Rejected: 19.7 times the forecast, with no lever pulled |
| C. One planned call with fixed retrieval, no loop | Rejected: which data is needed depends on what the first result shows |
| D. Keep the loop, decompose the cost, fix the biggest term first, cap the steps | Accepted |
Solution architecture
Figure 1. Components with cost controls
How to read this: the Orchestrator owns the two controls, the step budget and the trimming; the Trace store is where cost per task becomes visible.
Figure 2. One task, with the budget exit
How to read this: the loop is where cost multiplies, and the two exits show the cap working.
How AI solved it, step by step
Under the hood
The cost of the loop comes from how it is built. Anthropic's tool-use page (as of 2026-10-22) describes the client-side loop: after a tool runs, the application sends "a new request containing the original messages, the assistant's response, and a user message with the tool_result blocks". Every call therefore re-sends everything before it.
In this case's own model, with a prefix of P tokens and h tokens added per step, n calls send n x P + h x n(n-1)/2 input tokens. Here P = 3,000 and h = 1,500.
| Calls (n) | Input tokens | Relative to 1 call |
|---|---|---|
| 1 | 3,000 | 1.0 |
| 6 | 40,500 | 13.5 |
| 12 | 135,000 | 45.0 |
Doubling the calls from 6 to 12 multiplies input by 3.3, because the second term grows with the square of n. Caching targets the first term: Anthropic's page says a cache write costs 1.25 times and a read 0.1 times the base input price for the default 5-minute lifetime, with some per-model exceptions, and that hits "require 100 percent identical prompt segments". The full mechanism is in Under the hood: Decomposing the cost of a request.
Set inputTokens to 27,000 (trimmed results) in the widget below and the cost falls from 0.453 to 0.318.
What went wrong / failure modes
| Failure | What the system does | Why |
|---|---|---|
| A task loops to 20 calls (3.61 cu, 8 times the average) | Stops at the call budget and returns a partial answer | A hard cap bounds the long tail |
| The prefix changes per request (a timestamp at the top) | Cache reads fall to zero; an alarm fires on the cached share | Hits need identical segments |
| Trimming removes a field the answer needs | The review set drops; the tool offers a "fetch more" call | Quality is measured with cost |
| Tasks per user rise from 8 to 12 | Monthly re-forecast with a range | Usage is the other multiplier |
Results
These are illustrative targets, not measurements.
| Design | Cost per task (cu) | Monthly cost (cu) | Times the forecast |
|---|---|---|---|
| Pilot-scaled forecast | 0.046 | 20,240 | 1.0 |
| As shipped | 0.453 | 398,640 | 19.7 |
| Results trimmed | 0.318 | 279,840 | 13.8 |
| Trimmed and prefix cached | 0.1905 | 167,640 | 8.3 |
The biggest term was history (49.7% of the task cost), so it went first. After trimming, the repeated instructions were the biggest (56.6%), so caching went second. The new cost per task, 0.1905, is under the 0.20 goal. The forecast is re-based on it: 8.3 times the old one, and the owner now accepts or rejects that number with the cost per task in view.
Tradeoffs and when NOT to use this approach
- A step cap cuts off tasks that need a long investigation; give those a separate, approved budget.
- Trimming costs engineering effort in every tool and can hide needed context.
- Caching helps only the stable prefix, and a hit is never guaranteed.
- Where usage stays small, these controls cost more than they save.
Design rules extracted
- Forecast cost per completed task (calls x growing context), never cost per call.
- Decompose before optimising: fix the biggest term, then measure again, because the biggest term changes.
- Bound every loop with a call budget and a token ceiling.
- Put stable text first and keep tool results small.
- Report cost per task weekly, tagged by feature and team.
Assumptions
Prices are invented cost units per million tokens: input 10, output 40 (a ratio of 4); cached read 0.1 times and write 1.25 times the input price, the documented multipliers used as ratios (as of 2026-10-22). No real price is used.
- Pilot task: 1 call, 3,000 input and 400 output tokens: (3,000 x 10 + 400 x 40) / 1,000,000 = 0.046 cu. Pilot: 50 users x 4 tasks x 22 days = 4,400 tasks, 202.4 cu. Forecast: 5,000 x 4 x 22 = 440,000 tasks x 0.046 = 20,240 cu.
- Production: 5,000 x 8 x 22 = 880,000 tasks. One task: 6 calls, prefix 3,000 each (18,000), history growth 1,500 x (0+1+2+3+4+5) = 22,500, so 40,500 input; output 5 x 150 + 450 = 1,200. Cost (40,500 x 10 + 1,200 x 40) / 1,000,000 = 0.453; x 880,000 = 398,640; / 20,240 = 19.7. Split: prefix 0.180 (39.7%), history 0.225 (49.7%), output 0.048 (10.6%).
- Trimmed: history 600 x 15 = 9,000; (27,000 x 10 + 48,000) / 1,000,000 = 0.318; x 880,000 = 279,840; prefix share 0.18 / 0.318 = 56.6%.
- Cached prefix, cold start each task: 3,000 x 1.25 + 5 x 3,000 x 0.1 = 5,250 token-equivalents; (5,250 + 9,000) x 10 + 48,000 = 190,500 / 1,000,000 = 0.1905; x 880,000 = 167,640; / 20,240 = 8.3. Anthropic's page lists minimum cacheable lengths from 512 to 4,096 tokens by model, so assume a model whose minimum is at most 3,000.
- A 20-call task: input 20 x 3,000 + 1,500 x 190 = 345,000; output 4,000; cost 3.61 cu, 8.0 times 0.453. Doubling calls: 12 x 3,000 + 1,500 x 66 = 135,000; 135,000 / 40,500 = 3.3.
- The bank, its tools and its users are invented.
Last verified: 2026-10-22