Composite scenario, not a real company. Assembled from common industry patterns; every number is illustrative and its assumptions are listed at the end.
This case belongs to Models, routing and inference for AI Architects.
Context
A public body holds 2,000,000 documents (letters, forms and reports, already converted to text) that must be sorted into 20 categories so each reaches the right records team and retention schedule. The backlog must be cleared in 30 days. The first design put each document on a queue and called the largest approved model once per document, as if each were a customer waiting for an answer.
Problem statement
At 1,900 input tokens and a 30-token answer per document, the real-time design costs 40,400 cost units (cu), against an approved budget of 6,000 cu. Speed was never the issue: 2,000,000 documents in 30 days is 0.77 documents a second.
Measurable goals: model cost at or below 6,000 cu, backlog done in 30 days, fewer than 2% of documents sent to humans, and an error rate among machine-accepted documents whose 95% interval has an upper bound of 5% or less.
Constraints
- Documents contain personal data and may only go to the approved provider and region.
- Every decision must be traceable: document id, category, tier, confidence and prompt version.
- A person is the final authority on any document the machine is unsure about.
- Categories are fixed at 20; new categories need a policy decision, not a prompt edit.
Why the traditional approach fell short
Rules on metadata work for documents that arrive on forms with a type code, about 35% of the backlog, and that part stays rules. For free-text letters, keyword rules break because the same topic is written in different words and boilerplate is shared across categories. A classic machine-learning classifier needs thousands of labelled examples per category and retraining when categories change. Manual sorting at 3 minutes a document is 100,000 hours for the whole backlog.
Options considered
| Option | Verdict | Reason |
|---|---|---|
| A. Rules for form-coded documents, people for the rest (non-AI) | Rejected as the whole plan | The remaining 1,300,000 documents at 3 minutes each are 65,000 hours |
| B. Real-time call to the large model per document | Rejected | 40,400 cu, nothing in the process needs an instant answer |
| C. Rules first, then a batch cascade (small model, then large model for uncertain documents), then a human queue, with weekly human sampling | Accepted | Pays the large tier only for hard documents and uses asynchronous pricing |
Solution architecture
Figure 1. Components of the batch classification pipeline
How to read this: documents flow left to right and get more expensive attention only when the cheaper stage is unsure.
Figure 2. One batch, with expiry and the human path
How to read this: the loop at the top is waiting, the second loop sorts each result into one of three paths, and the document id ties every step together.
How AI solved it, step by step
- The Orchestrator classifies the 700,000 form-coded documents by rule. No model is involved.
- The Worker packs the other 1,300,000 into 26 batches of 50,000 requests, each with the label definitions and the document text, and submits them to the small tier.
- Each result carries a category and a confidence score. A score at or above the threshold with a valid category is accepted: 910,000 documents. The cascade idea comes from the FrugalGPT paper, where a scoring function rates an answer and the answer is returned if the score is above a threshold, otherwise the next model is queried.
- The 390,000 uncertain documents go to the large tier in 8 batches.
- Large-tier results below the threshold, or without a valid category (8%, 31,200 documents), go to the human queue.
- Every week the Eval service draws 400 accepted documents; reviewers mark each right or wrong.
- The result is a 95% Wilson interval for the error rate. If its upper bound is above 5%, the small-tier threshold is raised and the next week's uncertain share grows.
Under the hood
Two things are called batching. The API batch used here is a queue: the provider processes requests asynchronously, and the Anthropic and OpenAI batch documentation both state a 50% discount against synchronous calls, 24 hours as the maximum window, results in any order (match by custom id), and per-batch limits (as of 2026-10-22; limits differ, so the plan uses the smaller 50,000-request figure). Neither page explains how the discount is produced, so do not assume.
Inside a model server, batching has its own mechanism, set out in Under the hood: Continuous batching and paged KV memory. The Orca paper schedules at the granularity of a model iteration, not a request, so finished requests leave and new ones join. The PagedAttention paper observes that the memory holding each request's key-value cache limits how many requests fit, and that earlier systems stored only 20.4% to 38.2% of that memory as real token state. For the 13-billion-parameter model in that paper the cache costs 800 KB per token, so a 1,900-token document holds 1,556,480,000 bytes, about 1.56 GB. With roughly 30% of a 40 GB card available for cache (about 12 GB), 7 whole documents fit at once (7.7 by division). That is the paper's model, not the provider's, but it shows why short outputs help (the cache is mostly the 1,900 input tokens) and why server batching matters at 2 million documents.
What went wrong / failure modes
- Miscalibrated confidence. The small tier reports 0.95 on wrong answers. Mitigation: the weekly Wilson-interval gate, per category, and a threshold that moves.
- Expired or failed requests. Unfinished requests expire at the window's end. Mitigation: resubmit once by document id; writes are keyed by document id and prompt version, so a retry cannot duplicate.
- Results matched by position. Output order may differ from input. Mitigation: join on custom id only.
- Uncertain share grows. At 50% uncertain instead of 30%, cost is 7,878 cu, over budget. Mitigation: alarm on the uncertain share and review the label definitions first.
- Prompt edited mid-run. Two prompt versions in one backlog. Mitigation: record the version on every decision and re-run a sample when it changes.
Results
These are illustrative targets, not measurements.
| Design | Model cost (cu) | Share of real-time cost | Documents to humans |
|---|---|---|---|
| Real-time, large tier for all | 40,400 | 100% | Not designed |
| Rules, cascade, synchronous prices | 10,504 | 26% | 31,200 (1.56%) |
| Rules, cascade, batch prices | 5,252 | 13% | 31,200 (1.56%) |
The batch cascade meets the 6,000 cu budget and the 2% human share. Human review is 1,560 hours (31,200 documents at 3 minutes). In the first sample, 12 errors in 400 (3.0%) give a 95% interval of 1.72% to 5.17%: the upper bound is above 5%, so the gate fails and the threshold is raised. A later sample of 8 in 400 (2.0%) gives 1.02% to 3.90% and passes.
Tradeoffs and when NOT to use this approach
- Batch results arrive hours later. Do not use it when a person waits for the answer.
- The cascade runs two passes on uncertain documents; it only pays when the small tier handles most.
- Do not auto-accept where a wrong category has legal effect without more human sampling.
- At small volumes the pipeline and sampling effort may exceed the saving.
- Practitioners differ on whether to fine-tune a small model instead; that needs labelled data this body does not have.
Design rules extracted
- Ask whether anyone is waiting. If not, use asynchronous batch pricing.
- Send deterministic cases to rules before any model.
- Cascade by confidence, and calibrate the threshold against a human-labelled sample.
- Join batch results by id and make every write idempotent.
- Gate on an interval for the error rate, not on a single percentage.
Assumptions
Prices are invented cost units per million tokens: large 10 input and 40 output, small 1 and 4. Batch discount 50% (the documented figure, as of 2026-10-22, used as a ratio).
- 2,000,000 documents; 1,900 input tokens and 30 output tokens each. Large per document (1,900 x 10 + 30 x 40) / 1,000,000 = 0.0202 cu; small (1,900 x 1 + 30 x 4) / 1,000,000 = 0.00202 cu. Real-time large: 2,000,000 x 0.0202 = 40,400 cu. Rate: 2,000,000 / (30 x 86,400) = 0.77 per second.
- 35% rules: 700,000; model path 1,300,000. Small tier accepts 70%: 910,000; uncertain 390,000. Large tier leaves 8% for humans: 31,200 (1.56% of 2,000,000), at 3 minutes each = 1,560 hours. Manual: 2,000,000 x 3 / 60 = 100,000 hours; 1,300,000 x 3 / 60 = 65,000 hours.
- Batch cost: 1,300,000 x 0.00202 x 0.5 = 1,313; 390,000 x 0.0202 x 0.5 = 3,939; total 5,252. Without discount 2,626 + 7,878 = 10,504. If 50% are uncertain: 650,000 x 0.0202 x 0.5 = 6,565 plus 1,313 = 7,878.
- Batches of 50,000: 1,300,000 / 50,000 = 26 and 390,000 / 50,000 = 7.8, so 8.
- Wilson interval at z = 1.96 (the NIST e-Handbook's recommended interval for a proportion); sample sizes and error counts are invented.
- Cache arithmetic: 2 x 5,120 x 40 x 2 bytes = 819,200 bytes per token; x 1,900 = 1,556,480,000; 12,000,000,000 / 1,556,480,000 = 7.7.
- The agency, its categories and its documents are invented.
Last verified: 2026-10-22