Cost and Unit Economics for AI Systems

Build versus buy: the cost view

How to read this: the table lists lines to price, not winners; fill it with your own figures and total each column over 36 months.

Do not let token cost stand in for total cost: page 2 and the contract search case show why. For self-hosting versus an API the arithmetic reduces to one ratio. Page 4 gives cost per million tokens = hourly cost / (million tokens per hour at full load x utilisation). With r = self-hosted cost per million tokens at 100% utilisation divided by the API cost per million tokens, self-hosting breaks even at utilisation r.

Operating staff, evaluation and on-call add a fixed monthly cost that raises these thresholds, so measure with your own traffic.

Case files

Three composite case files apply this page. All are illustrative scenarios, not real companies.

• Pilot to production: a cost shock. Problem: a 50-user pilot became 5,000 users and an agent loop multiplied the calls. Pattern: decompose the cost per successful task, then fix the largest term first.
• Batch versus real-time: a redesign. Problem: a nightly enrichment job ran on the real-time path. Pattern: batch with a window, a cutoff for stragglers and retries on the batch.
• Cascade break-even in legal review. Problem: a legal-review triage used the large model for every document. Pattern: a cascade with a calibrated threshold and a break-even check.

Under the hood: Decomposing the cost of a request

Intuition

A bill is a sum of terms that scale with different things: tokens in, tokens out, calls, people. In a single call the output usually weighs most. In an agent loop the input does, because every call re-sends the history.

Mechanism

Figure 5. One task, four model calls, growing input

How to read this: the loop repeats the whole history, so cost grows faster than the number of calls.

1. The provider's Messages interface is stateless: Anthropic's documentation says you send the full conversational history each time, and its tool-use loop sends the earlier messages plus the tool result in each new request (as of 2026-10-22, Messages, tool use).
2. Calls 1 to 3 each end in a tool call and call 4 is the answer. Input per call is the previous input plus the previous output plus the tool result: 3,000 + 300 + 200 = 3,500.
3. Without caching every input token is fresh. With caching, assume each call reads the previous call's prefix (read at 0.1) and writes only the new 500 tokens (write at 1.25).
4. Output weighs 4 per token in the placeholder rates.

Add output, 4 x 300 x 4 = 4,800 token-equivalents. The totals are 19,800 uncached and 11,475 cached, a 42.0% saving. Input is 75.8% of the uncached total. Growth alone costs 17.9% more than four flat 3,000-token calls with the same output (16,800).

Use the widget to see the single-call case, where output dominates.

The start values give 0.0031 CU per request, 3.1 CU per 1,000: fresh input 1,000, cached input 100, output 2,000 token-equivalents, so output is 64.5% of the cost. Set cachedShare to 0 and the cost per 1,000 rises to 4; set it to 0.9 and it falls to 2.38. The widget bills cached tokens at the cached rate and ignores the write premium; the next section adds it.

What this means for architects

Under the hood: Cache hit-rate math

Intuition

A cache write costs more than a normal read, and a read costs far less. So caching a prefix saves money only when enough requests reuse it before it expires. The question is not "is caching on" but "what is the hit rate".

Mechanism

Figure 6. Life of a cached prefix

How to read this: only the self-loop on Warm is cheap, and both arrows back to Cold cost a full write.

Provider terms as of 2026-10-22 (page 4 explains the mechanism):

• Anthropic documents 5-minute cache writes at 1.25 times the base input rate and reads at 0.1 times, lower read multipliers on some models, a default 5-minute lifetime refreshed at no extra cost on each use, and 2 times for writes with the 1-hour option (prompt caching).
• OpenAI says caching is on by default for supported models, discounts cached input (its page says up to 95%) and reports cached_tokens in usage (prompt caching).
• Check your own provider and model: the numbers below use 1.25 and 0.1 as illustrative inputs.

1. Let P be the stable prefix tokens, V the variable tokens, w the write multiplier, r the read multiplier and h the hit rate.
2. A hit costs rP + V. A miss writes the prefix, so it costs wP + V.
3. Expected input cost per request = V + P x [h r + (1 - h) w]. Uncached it is P + V.
4. Caching wins when h r + (1 - h) w < 1, so the break-even hit rate is h* = (w - 1) / (w - r) = 0.25 / 1.15 = 21.7% for w = 1.25 and r = 0.1.
5. With random arrivals at lambda per minute and lifetime T minutes, the chance that the next request finds a warm entry is about 1 - e^(-lambda T). This is this page's own independent-arrivals model, not a provider statement: it ignores routing across machines.

With P = V = 1,000 and output weighing 2,000 token-equivalents:

With T = 5 minutes, break-even needs about one request every 20 minutes (lambda = 0.049). At 0.1 requests a minute the modelled hit rate is 39%, at 0.5 it is 92%.

Try the request-level view. Set the write multiplier to 2: on which request does caching start to win?

At the start values the cached total after two requests is 1.35 prefix-lengths against 2 uncached, so break-even is request 2. With a write multiplier of 2 it moves to request 3 (2.2 against 3).

What this means for architects

Under the hood: Batch versus real-time pricing models

Intuition

A batch interface lets the provider run your work when capacity is free, in exchange for a discount and a loose deadline. You give up latency and order; you get a lower cost ratio. The design question is what happens to the work that does not finish in time.

Mechanism

Figure 7. A nightly batch with a cutoff and a real-time fallback

How to read this: the alt block is the part a happy-path design leaves out, and the last arrow is safe to repeat because the write is keyed.

Documented as of 2026-10-22, expressed as ratios and limits only:

Anthropic's page adds that processing can be slowed by demand, that prompt-caching discounts stack with the batch discount but cache hits in batches are best effort (it says typical hit rates range from 30 to 98 percent), and that the 1-hour cache lifetime is worth considering for shared context. OpenAI says batches use a separate rate-limit pool from standard limits.

1. This page's own model: let b be the batch cost ratio (0.5) and f the share of items not finished by the cutoff and rerun in real time at ratio 1.
2. Effective ratio for the batchable work = (1 - f) b + f = b + (1 - b) f. This assumes unfinished requests are not billed in the batch: both providers document that for expired requests, and Anthropic for cancelled requests not yet sent. Check how yours bills requests in progress when you cancel.
3. Stack caching: input cost relative to real-time uncached = b x (V + P x [h r + (1 - h) w]) / (P + V), with the same assumptions as before (P = V = 1,000, w = 1.25, r = 0.1).

How to read this: the first table prices the stragglers, the second prices the cache inside the batch, using the 30% and 98% ends of the range Anthropic documents.

What this means for architects

Under the hood: Cascade break-even

Intuition

A cascade sends every request to a cheap model first and pays for the large model only when the answer is not trusted. So each request pays for the small attempt and the scorer, and the escalated share pays again. It wins only when escalations are rare enough.

Mechanism

Figure 8. What each path of a cascade pays

How to read this: the two branches are the two prices a request can pay, and the escalation rate decides the mix.

FrugalGPT (Chen, Zaharia and Zou) defines a cascade with a generation scoring function that returns a reliability score in [0, 1] for a query and an answer, and a router that queries models in sequence and returns the first answer whose score passes that model's threshold. Its abstract reports matching the best individual model with up to 98% cost reduction (2023 APIs), or higher accuracy at equal cost. The 98% is its best dataset: Table 3 shows 98%, 73% and 59% on its three.

The paper names these limits:

• Labelled examples from a similar distribution are needed.
• Learning the cascade takes resources, which the authors treat as one-time.
• In its error analysis, an unsure score forced calls down the whole chain, which it calls an open problem.
• It lists latency among factors left for future work.

Their scorer is a small DistilBERT regression model, described as cheaper than the language models, with its cost not quantified.

1. Let c_s, c_g and c_l be the cost per request of the small model, the scorer and the large model, and f the escalation rate.
2. Cascade cost per request = c_s + c_g + f x c_l. Always-large = c_l.
3. Break-even escalation rate f* = 1 - (c_s + c_g) / c_l.
4. With rework (this page's own extension, not from FrugalGPT): let a be the share of requests wrongly accepted from the small model, e the large model's own wrong share and E the cost of one undetected error in the same units. Then f* = 1 - (c_s + c_g) / c_l - (a - e) x E / c_l.

How to read this: each row adds one cost the row above left out, and the break-even escalation rate falls with each.

At f* = 78% the adjusted cascade costs 0.12 + 0.78 + 0.2 = 1.10, equal to always-large. Raising the threshold lowers a and raises f, so the threshold is the dial: set it on labelled data (see page 8).

Try it: set the escalation rate to 0.9 and watch the cascade cost reach the always-large cost.

The widget omits the scorer and the rework term, so its break-even (90%) is the upper bound; the table above shows how the other terms pull it down.

What this means for architects

Decision framework

Figure 9. Which term to attack first

How to read this: measure first, then take the branch for the term that is largest or cheapest to reduce.

Comparison of the main options

Failure scenarios to design for

Worked example

Start from the decomposition of a task and apply four levers in order.

Assumptions: this is a composite scenario and every figure is illustrative, in placeholder cost units.

• 1,000 task attempts, 3 model calls per task.
• Each call has 1,000 fresh and 1,000 cached input tokens and 500 output tokens, at 1, 0.1 and 4 CU per million tokens. The prefix write premium is ignored (hit rate near 100%).
• Retrieval 0.002 CU per task; two tool calls at 0.0005 CU each; infrastructure 1.5 CU per 1,000 tasks.
• Retries repeat 8% of model and tool work; reviewers check 2% of tasks at 0.15 CU each; 92% of attempts succeed.

Cost per 1,000 successful tasks = 17.624 / 0.92 = 19.16 CU. A forecast of calls times cost per call gives 3 x 3.1 = 9.3, so the full figure is 2.06 times higher.

Output is the largest term (34%) but routing is the biggest lever (25.6% against 11.0%), because it applies to all model cost. The steps assume the 92% success rate holds. If the final design succeeds 88% of the time, the cost is 11.62 and the cumulative saving falls to 39.3%: a four-point quality loss gives back 2.7 points.

A cost guard you can run

The guard prices a call in placeholder units, computes cost per success and decides allow, degrade or reject from the worst-case reservation of Figure 4. The checks at the bottom were executed and pass.

from dataclasses import dataclass

@dataclass
class Rates:  # placeholder cost units per million tokens, never real prices
    fresh_in: float = 1.0
    cached_in: float = 0.1
    out: float = 4.0

def call_cost(r: Rates, fresh: int, cached: int, out: int) -> float:
    return (fresh * r.fresh_in + cached * r.cached_in + out * r.out) / 1_000_000

def cost_per_success(total_cost: float, successes: int) -> float:
    if successes == 0:
        return float("inf")
    return total_cost / successes

def decide(spent: float, limit: float, worst_case: float, soft: float = 0.8) -> str:
    """Reserve the worst case (estimated input + max output) before the call."""
    if spent + worst_case > limit:
        return "reject"
    if spent + worst_case > soft * limit:
        return "degrade"  # smaller model, shorter max output, no optional tools
    return "allow"

worst = call_cost(Rates(), 1000, 1000, 500)  # 0.0031 units
assert abs(worst - 0.0031) < 1e-12
assert cost_per_success(3000, 800) == 3.75
assert decide(10.0, 100.0, worst) == "allow"
assert decide(79.999, 100.0, worst) == "degrade"
assert decide(99.999, 100.0, worst) == "reject"

Read it as the three jobs of the gateway's cost layer: price, divide by successes, and gate on the worst case. A production version would settle the real usage after the call, release the reservation and write the tagged usage event.

Common mistakes

• Forecasting calls times cost per call. It omits retries, review and tails; forecast cost per successful task.
• Comparing models on cost per call. Compare cost per success, with the success rate measured on the golden set.
• Caching a prefix that is reused less than break-even. Compute the hit rate first, and skip one-off prompts.
• Batching work with a deadline shorter than the window. Add a cutoff and a real-time fallback, or do not batch.
• Copying a cascade threshold from a paper. Set it on your labelled data for a target false-accept rate.
• Checking the budget once a month. Use a soft limit, a hard limit and per-task caps enforced at the gateway.

Try it

Time: 25 to 40 minutes.

1. Take one feature and write its eight cost terms from the table in Core concepts, with your own usage counts (calls per task, tokens per call, review share). Use placeholder units or the cost estimator for real rates.
2. Compute cost per successful task with a stated success rate, and compare it with calls times cost per call.
3. Compute the break-even cache hit rate for your provider's documented multipliers, and the hit rate your traffic would give.
4. Apply two levers from the ladder in order, recompute, and state what you would check on the golden set before shipping each.

Done when you have a one-page table with the eight terms, cost per successful task before and after two levers, the break-even hit rate or escalation rate for the lever you chose, and the golden-set check that must pass.

Key takeaways

• Budget cost per successful task: failed attempts, retries and reviewers are paid for.
• Decompose before you optimise. Output dominates a single call and input dominates an agent loop.
• Caching wins above h* = (w - 1) / (w - r); batching gives a fixed ratio b + (1 - b) f with a cutoff; a cascade wins below f* = 1 - (small + scorer) / large.
• Cost grows by multiplication, and the tail of long loops needs a cap.
• Budgets need a soft limit, a hard limit, per-task caps and a defined degraded mode.
• Choose levers by measured term and break-even, cheapest and most reversible first, and prove quality on the golden set.
• Provider multipliers, windows and discounts change: read them from config with a review date.

Self-check

Further reading

• Chen, Zaharia and Zou, FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance, for the cost model and the cascade.
• Anthropic, Batch processing and Prompt caching, as of 2026-10-22.
• OpenAI, Batch API guide and Prompt caching, as of 2026-10-22.
• The site's LLM cost estimator and LLM token budgeting for real figures.

Last verified: 2026-10-22

Next

Next is Security and threat modeling for AI systems, which covers trust boundaries, prompt injection, tool abuse and what to log safely.