Composite scenario, not a real company. Assembled from common industry patterns; every number is illustrative and its assumptions are listed at the end.
This case belongs to Cost and unit economics for AI Architects.
Context
A legal services firm triages 40,000 inbound counterparty contracts a month for corporate clients. Each contract is marked "standard" (normal signature path) or "needs lawyer review". The 40,000 are contracts that did not match an approved template. Every contract currently goes to the largest approved model.
Problem statement
Always using the large model costs 2,880 cost units (cu) a month against a budget of 1,500.
Measurable goals:
- Monthly model cost at or below 1,500 cu.
- On a lawyer-labelled sample, the upper end of the 95% Wilson interval for the share of risky contracts wrongly marked standard is 4% or less.
- Every decision traced with tier, score and prompt version.
Constraints
- Lawyers decide on anything marked "needs review".
- Privileged documents go only to the approved provider.
- A change of model or prompt version needs a fresh calibration.
Why the traditional approach fell short
A clause library with keyword rules catches contracts written on the firm's own paper. Counterparty paper words the same clause many ways, and a heading such as "limitation of liability" appears in nearly every contract, so keyword flags fire too often and lawyers learn to ignore them. A classic classifier needs labelled examples per clause type and breaks when the law or a template changes.
Options considered
| Option | Why accepted or rejected |
|---|---|
| A. Clause library and keyword rules only (non-AI) | Rejected as the whole plan: too many false flags on counterparty paper |
| B. Large model for every contract | Rejected: 2,880 cu a month, 1.9 times the budget |
| C. Small model only | Rejected: it misses 38 of 300 risky contracts (12.7%) on the calibration set |
| D. Cascade: small model first, escalate below a calibrated threshold, with a break-even check | Accepted |
Solution architecture
Figure 1. Components of the cascade
How to read this: the Policy engine owns the threshold, and the arrow to the large tier is the only place a contract is paid for twice.
Figure 2. One contract, accepted or escalated
How to read this: the alt block is the cascade, and the opt block is unchanged from the old design.
How AI solved it, step by step
Under the hood
The cascade follows the FrugalGPT paper (arXiv 2305.05176): a scoring function rates an answer between 0 and 1; the answer is returned if the score is above a threshold, otherwise the next model is queried. The paper reports that its cascade matched the best single model with up to 98% cost reduction on its benchmarks; that is the paper's result, not a forecast here.
Every request pays the small call; a share e also pays the large one.
cascade cost per request = c_small + e x c_large
always-large cost = c_large
break-even escalation e* = 1 - c_small / c_large| Small / large cost ratio | Break-even escalation rate |
|---|---|
| 0.05 | 95% |
| 0.10 | 90% |
| 0.25 | 75% |
| 0.50 | 50% |
Try escalation rates in the widget; the starting values are per 1,000 contracts. The full derivation is in Under the hood: Cascade break-even.
What went wrong / failure modes
| Failure | What the system does | Why |
|---|---|---|
| The small tier is confidently wrong on a rare clause type | Indemnity and liability clauses always escalate by rule; weekly per-clause sample | A global score hides weak spots |
| Escalation drifts up (new counterparty paper) | Alarm at 45%; at 60% the cascade costs 50.4 per 1,000 | Cost and quality both move |
| Model or prompt version changes | Scores shift, so the threshold is recalibrated before release | The scorer learned the old outputs |
| Price ratio changes | Break-even recomputed; at ratio 0.5 and 52% escalation the cascade costs 73.44, more than always-large (72) | Break-even depends on both costs |
| Calibration sample too small | The interval is wide, so collect more labels before trusting it | 300 risky contracts is a floor |
Results
These are illustrative targets, not measurements. The calibration set holds 1,200 lawyer-labelled contracts, 300 of them risky. The gate: upper end of the 95% Wilson interval of missed risk at 4% or less.
| Threshold | Escalation | Cost per 1,000 (cu) | Risky contracts missed (of 300) | Upper bound | Gate |
|---|---|---|---|---|---|
| 0.70 | 18% | 20.16 | 21 | 10.5% | fail |
| 0.80 | 26% | 25.92 | 12 | 6.9% | fail |
| 0.90 | 35% | 32.40 | 5 | 3.8% | pass |
| 0.95 | 52% | 44.64 | 3 | 2.9% | pass |
The lowest passing threshold, 0.90, costs 32.4 cu per 1,000: 1,296 cu a month against 2,880 (a 55% saving) and inside the 1,500 budget. A higher threshold buys safety at a price: 0.95 is cheaper than always-large only because the small tier costs a tenth of the large.
Tradeoffs and when NOT to use this approach
- A cascade needs labelled data and a weekly review; with few labels, use the large model.
- Escalated contracts wait for two calls, so the slowest path is slower than always-large.
- Where a miss has legal consequences that no bound can cover, keep a lawyer or the large tier in the loop.
- If the small tier is not much cheaper, the break-even is low and the cascade may cost more.
- Practitioners differ on whether to fine-tune one small model instead.
Design rules extracted
- Compute the break-even escalation rate first: 1 - small cost / large cost.
- Choose the threshold from a labelled sample, with an interval, not a single percentage.
- Escalate high-stakes clause types by rule, whatever the score.
- Recompute the gate and the break-even whenever a model, prompt or price changes.
- Alarm on the escalation share: it moves both cost and quality.
Assumptions
Prices are invented cost units per million tokens: large 10 input and 40 output, small 1 and 4. No real price is used.
- Per contract 6,000 input and 300 output tokens. Large (6,000 x 10 + 300 x 40) / 1,000,000 = 0.072 cu; small (6,000 x 1 + 300 x 4) / 1,000,000 = 0.0072 cu; per 1,000: 72 and 7.2 (ratio 0.1).
- 40,000 contracts a month: always-large 40 x 72 = 2,880 cu (2,880 / 1,500 = 1.92). Cascade at 35%: 7.2 + 0.35 x 72 = 32.4 per 1,000; x 40 = 1,296 cu (45% of 2,880).
- Other thresholds: 7.2 + 0.18 x 72 = 20.16; 7.2 + 0.26 x 72 = 25.92; 7.2 + 0.52 x 72 = 44.64; at 60%: 7.2 + 0.6 x 72 = 50.4.
- The scorer's own cost is taken as zero here; the lesson's formula adds it. Break-even 1 - 7.2 / 72 = 90%. At ratio 0.5: small 36, cascade at 52% = 36 + 0.52 x 72 = 73.44 against 72.
- Calibration counts and escalation rates are invented. Wilson interval at z = 1.96 (the NIST e-Handbook's recommended interval for a proportion): 21/300 upper 10.46%; 12/300, 6.86%; 5/300, 3.84%; 3/300, 2.90%; 38/300 = 12.7%.
- The budget of 1,500 cu, the firm and its contracts are invented.
Last verified: 2026-10-22