Composite scenario, not a real company. Assembled from common industry patterns; every number is illustrative and its assumptions are listed at the end.
This case belongs to Cost and unit economics for AI Architects.
Context
A distributor enriches 240,000 shipment and supplier records every night: the model reads free-text notes and returns a part category, a hazard flag and a delay reason. Planners need the result by 06:00. The job starts at 22:00 and calls the real-time interface with many parallel workers, on the same account and rate limit as the planners' live assistant.
Problem statement
| Measure | Real-time job | Goal |
|---|---|---|
| Finish time | 08:48 | Before 06:00 |
| Live assistant limited | 06:00 to 08:48 | Never |
| Cost per night (cu) | 1,425.6 | 60% of that or less (855.4) |
The cause is arithmetic: 240,000 records x 1,620 tokens is 388,800,000 tokens. At a shared limit of 600,000 tokens per minute, that takes 648 minutes (10.8 hours).
Constraints
- Planners start at 06:00 and need the full set.
- The live assistant must keep its rate-limit headroom.
- A record's result is written once per prompt version; a rerun must not duplicate.
- Records go only to the approved provider.
Why the traditional approach fell short
Catalogue lookups and regular expressions handle records that carry a part code; the free-text notes ("pallet 4 wet, dock closed till Tue") do not follow a pattern. A classic classifier needs labelled examples for each delay reason and breaks when the reasons change. Planners reading notes cannot cover 240,000 a night.
Options considered
| Option | Why accepted or rejected |
|---|---|
| A. Catalogue lookup and rules only (non-AI) | Rejected as the whole plan: free-text notes; kept upstream for coded records |
| B. Stay real-time, throttle to 240,000 tokens a minute | Rejected: 388,800,000 / 240,000 = 1,620 minutes, 27 hours |
| C. Stay real-time, buy a higher limit | Rejected: fixes the clock, not the cost (still 1,425.6 cu) |
| D. Batch interface with a window, a cutoff before the deadline and a capped real-time fallback | Accepted |
Solution architecture
Figure 1. Two paths to the provider
How to read this: the nightly bulk goes through the top path; the real-time path carries live questions and a small capped fallback.
Figure 2. One night, with cutoff and fallback
How to read this: the loop is waiting, the alt block sorts each result, and the opt block is the only real-time work.
How AI solved it, step by step
Under the hood
A batch interface trades time for price. Both providers' pages state the terms as of 2026-10-22; the plan uses the stricter figure of each.
| Term | Anthropic Message Batches | OpenAI Batch API |
|---|---|---|
| Price | 50% of standard | 50% discount versus synchronous |
| Window | Batches expire if not complete within 24 hours; most finish within 1 hour | Completes within 24 hours, "often more quickly" |
| Limit per batch | 100,000 requests or 256 MB | 50,000 requests, 200 MB |
| Result order | Any order; match by custom_id | May not match input; match by custom_id |
| Unfinished at window end | Expired, not billed | Written to the error file as batch_expired; completed ones billed |
| Cancel | May contain partial results | Cancelling can take up to 10 minutes |
| Rate limits | Limits on batch requests and queued requests | "Batch API rate limits are separate from existing per-model rate limits" |
Neither page explains how the discount is produced. The design cost is N x c x b + retries + fallback, with b = 0.5. The full comparison is in Under the hood: Batch versus real-time pricing models.
What went wrong / failure modes
| Failure | What the system does | Why |
|---|---|---|
| Batches run slower than usual | The 05:00 cutoff fires and the capped fallback runs | The deadline is 06:00, not the provider window |
| Results matched by position | Join on record id only | Order is not guaranteed |
| A rerun writes twice | Upsert keyed by record id and prompt version | Retries must be idempotent |
| Fallback crowds out live users | Capped at 100,000 tokens a minute | One sixth of the shared limit |
| Spend overshoots the alarm | Alarm on cost per night, not a hard stop | Anthropic's page says batches may slightly exceed the spend limit |
Results
These are illustrative targets, not measurements.
| Design | Finish | Live assistant limited | Cost per night (cu) | Share |
|---|---|---|---|---|
| Real-time, unthrottled | 08:48 | 06:00 to 08:48 | 1,425.6 | 100% |
| Real-time, throttled | 01:00, two days later | No | 1,425.6 | 100% |
| Batch with cutoff and fallback | About 05:31 | No | 731.3 | 51.3% |
The batch design costs 712.8 (batches) + 7.1 (retry batch) + 11.4 (fallback) cu. Over 30 nights that is 21,940 cu instead of 42,768, a saving of 20,828 cu.
Tradeoffs and when NOT to use this approach
- Results arrive hours later: do not batch anything a person or an event is waiting on.
- The worst case is the full window, so the cutoff and fallback are part of the design.
- Discount and limits are provider terms that change; recheck them.
- Small volumes may not justify the extra state and monitoring.
Design rules extracted
- Ask who is waiting; if nobody, use the batch window.
- Set the cutoff before the business deadline and define what happens to stragglers.
- Join by id and write idempotently.
- Cap any real-time fallback so it cannot crowd out live traffic.
- Trace succeeded, errored, expired and minutes taken for every run.
Assumptions
Prices are invented cost units per million tokens: input 3, output 12; the batch discount is the documented 50% (as of 2026-10-22), used as a ratio.
- 240,000 records, 1,500 input and 120 output tokens each. Per record (1,500 x 3 + 120 x 12) / 1,000,000 = 0.00594 cu; per night 240,000 x 0.00594 = 1,425.6; 30 nights 42,768.
- Tokens per night 240,000 x 1,620 = 388,800,000. At 600,000 a minute (an invented shared limit, not a provider figure): 648 minutes = 10.8 hours; 22:00 + 10.8 h = 08:48. At 240,000: 1,620 minutes = 27 hours. Over 8 hours the job needs 810,000 a minute.
- Batch: 240,000 x 0.00594 x 0.5 = 712.8; errored 1% = 2,400 x 0.00594 x 0.5 = 7.128; stragglers 0.8% = 1,920 x 0.00594 = 11.4048 at full price; total 731.3328, 51.3% of 1,425.6. Thirty nights 21,940. Batch size 240,000 / 12 = 20,000; at about 4 characters a token (a rule of thumb) 1,500 tokens is near 6 KB, so a batch is near 120 MB, under both size limits.
- Fallback: 1,920 x 1,620 = 3,110,400 tokens / 100,000 = 31.1 minutes.
- The distributor, its records and its limits are invented.
Last verified: 2026-10-22