Models, Routing and Inference for AI Architects

Mechanism

Sennrich, Haddow and Birch adapted byte pair encoding (BPE) to words: start from characters plus an end-of-word symbol, then repeatedly replace the most frequent adjacent pair with one new symbol. The vocabulary size is the initial symbols plus the number of merges, which the paper calls the only hyperparameter. At use time the merges are applied in learned order, so any word can be encoded. Running the code the paper prints on its toy dictionary (low 5, lower 2, newest 6, widest 3; the figure caption names a slightly different dictionary) gives the first merges (e, s) with count 9, (es, t) 9, (est, end-of-word) 9, (l, o) 7 and (lo, w) 7; the unseen word "lowest" then encodes as the pieces low and est. Production tokenizers build on the same idea: the tiktoken README, for example, describes BPE as reversible and lossless, working on arbitrary text, and compressing text.

Figure 5. Where token counts come from

How to read this: the loop happens once, before training, and the lower half happens on every request.

Tokenizers differ by model, and the primary sources show by how much. The Llama 3 paper reports a 128K-token vocabulary (100K from the tiktoken tokenizer plus 28K for non-English languages) and an English compression of 3.94 characters per token against 3.17 for the Llama 2 tokenizer. For 4,000 English characters that is 4,000 / 3.17 = 1,262 tokens against 4,000 / 3.94 = 1,015, so the older tokenizer needs 24% more. Anthropic's token-counting page says models from Claude 4.7 on use a newer tokenizer that produces approximately 30% more tokens for the same input than earlier models, depending on content (as of 2026-10-22). The context window limits input plus output together: Anthropic's page defines it as all the text a model can reference when generating a response, including the response itself.

What this means for architects

Count tokens with the target model's own tokenizer or the provider's counting endpoint, never with words times a ratio and never with counts measured on another model. The site's token counter uses 4 characters per token and is a planning estimate only. Budgets and cost forecasts are in tokens, so a model upgrade is a budget event even if the prompt text is unchanged: pin model versions, recount, and make the Model gateway enforce the limit it computed. Expect more tokens per character for code, numbers and non-English text, and measure your own content.

Under the hood: Attention and the KV cache

Intuition

To choose the next token, the model lets the current position look back at every earlier position and weigh them by relevance. What it looks up in each earlier position (its key and its value) never changes once computed. So the server keeps them instead of recomputing them for every new token. That store is the KV cache. It grows by a fixed amount per token, and it is memory, not arithmetic, that runs out first.

Mechanism

The Transformer paper defines attention as softmax(Q K^T / sqrt(d_k)) V: each query is compared with all keys, the scores are normalised, and they weight the values. In a decoder, masking stops a position from attending to later positions. vLLM's paper describes the consequence at serving time: in the generation phase the key and value vectors of earlier positions were cached at previous iterations, so only the new token's key and value are computed.

The cache size per token follows from the model's shape: bytes per token = 2 (key and value) x layers x KV heads x head dimension x bytes per value. The PagedAttention paper works the example for the 13B-parameter OPT model: 2 x 5,120 (hidden size) x 40 (layers) x 2 (bytes per FP16 value) = 819,200 bytes, which is the paper's 800 KB per token (reading 1 KB as 1,024 bytes), and up to 1.6 GB for a 2,048-token sequence (1.68 x 10^9 bytes).

Grouped-query attention (GQA) shrinks the cache by sharing keys and values among groups of query heads; multi-query attention (MQA) is the extreme with one KV head. The Llama 3 paper states that its models use GQA with 8 key-value heads and lists the dimensions below; the head dimension is the model dimension divided by the number of attention heads, which gives 128 in all three.

For the 8B model, 2 x 32 x 8 x 128 x 2 = 131,072 bytes. With 32 KV heads (plain multi-head attention) it would be 4 times larger, 512 KiB per token. These are per sequence: ten concurrent 8,192-token conversations on the 8B model hold 10 GiB of cache on top of the weights.

Figure 6. What is computed once and what is kept

How to read this: the first block happens once, the loop repeats for every output token, and the cache is the only thing that carries information between steps.

FlashAttention is a separate optimisation: an exact attention algorithm that tiles the computation to cut reads and writes between GPU high-bandwidth memory and on-chip SRAM. It makes attention faster and less memory-hungry; it does not remove the need to store keys and values.

What this means for architects

The cache turns context length into memory. Capacity for a self-hosted model is (GPU memory minus weights) / (KV bytes per token x tokens per request). In the vLLM paper's setup (a 13B model on a 40 GB A100) about 65% of memory holds weights and close to 30% is left for the cache: about 12 GB, or 12 x 10^9 / 819,200 = 14,648 tokens in total, roughly seven full 2,048-token sequences. That is why long contexts cut concurrency, and why a model's maximum window is not the context you can afford on every request. Fewer KV heads means cheaper long context, a selection criterion next to price and quality. With an API the same arithmetic appears as the provider's limits and long-input pricing, which are provider-specific.

Under the hood: Prefill and decode

Intuition

Reading the prompt is done for all tokens at once; writing the answer is done one token at a time. The first is a large batch of arithmetic that a GPU does well. The second repeats a small amount of arithmetic per token, but each step must stream the model's weights from memory again, so it is limited by memory bandwidth and not by compute.

Mechanism

The vLLM paper describes a prompt phase that takes the whole prompt, parallelises over its tokens, and produces the first new token, then a generation phase that cannot be parallelised because each token depends on the previous one: it uses matrix-vector operations, underuses the GPU, is memory-bound, and accounts for most of a single request's latency. The Sarathi-Serve paper states it as prefill compute-bound, decode memory-bound, and measures on Mistral-7B on an A100 that the linear operations for one decode token cost nearly the same as for 128 prefill tokens. DistServe defines TTFT as the duration of prefill and TPOT as the average time per token after the first.

Figure 7. Anatomy of one streamed request

How to read this: the first block before the loop is paid once and grows with prompt length, while the loop is paid per output token.

Two bounds make this concrete (all inputs are stated assumptions). The decode floor: if decode reads all weights per token, time per token is at least weight bytes divided by memory bandwidth. An 8B-parameter model in 16-bit weights is about 16 GB; at the 2,039 GB/s NVIDIA lists for the A100 80GB SXM, 16 x 10^9 / 2,039 x 10^9 = 7.85 ms per token, so 400 tokens take at least 3.1 s (ignoring the KV cache and overheads, so real systems are slower). And TTFT: with an assumed prefill rate of 10,000 tokens per second, a 9,000-token prompt prefills in 0.9 s, or 3,000 / 10,000 = 0.3 s if 6,000 tokens come from a prefix cache. With TPOT 25 ms and 400 output tokens the totals are 0.9 + 399 x 0.025 = 10.875 s and 10.275 s: caching moved TTFT and the total by the same 0.6 s, because the 399 decode steps dominate.

What this means for architects

An output token costs more latency than an input token: each is a sequential step, while input tokens are processed in parallel. Shortening answers (a tighter format, a schema without prose, a sane max_tokens) is the strongest lever on total time, and thinking tokens are decode time too. Shortening or caching the prompt moves TTFT, which an interactive user feels first. Set separate objectives for TTFT and TPOT and alert on their 95th percentiles: the Sarathi-Serve authors note that under load scheduling delays raise TTFT, and DistServe shows that adding a prefill to a batch of decodes slows both. Batch jobs care about throughput and can ignore TTFT.

Under the hood: Continuous batching and paged KV memory

Intuition

Because decode is memory-bound, running many requests together shares each weight read across all of them, which is why serving systems batch. Two practical problems follow. Requests finish at different times, so a fixed batch wastes slots. And each request's cache grows by an unknown amount, so reserving space for the worst case wastes memory. Continuous batching solves the first, paged memory the second.

Mechanism

The Orca paper (OSDI 2022) contrasts request-level scheduling, where the engine keeps a batch fixed until every request in it finishes, with iteration-level scheduling, where the scheduler runs one model iteration at a time. Finished requests return immediately and a new request starts after the current iteration, not after the whole batch. The authors report 36.9 times the throughput of NVIDIA FasterTransformer at the same latency on GPT-3 175B. Toy arithmetic shows the waste removed: four requests with output lengths 10, 50, 200 and 30 in a fixed batch occupy 4 x 200 = 800 token-steps but need 10 + 50 + 200 + 30 = 290, a utilisation of 36.25%.

The vLLM paper attacks memory. Earlier systems pre-allocated a contiguous chunk per request for its maximum length, and the paper measures that only 20.4% to 38.2% of KV-cache memory held real token states. PagedAttention stores the cache in fixed-size blocks (16 tokens by default), keeps a block table from logical to physical blocks, allocates a block only when the last one fills, and so wastes at most part of one block per request. Blocks can be shared with reference counts and copy-on-write. When memory runs out, vLLM preempts whole requests (earliest arrivals served first) and recovers them by swapping to CPU memory or recomputing. The paper reports 2 to 4 times the throughput of FasterTransformer and Orca at similar latency.

Worked numbers for the OPT-13B cache at 819,200 bytes per token: a 100-token sequence needs ceil(100 / 16) = 7 blocks, which is 112 slots, 12 of them unused (12 x 819,200 = 9.8 MB). Reserving the 2,048-token maximum instead holds 1.68 GB for a request that uses 100 tokens, 95.1% of it idle.

Figure 8. Life of a request in a continuous-batching server

How to read this: each box is a state, arrows carry the condition, and the loop on Running is the per-iteration step that continuous batching makes possible.

What this means for architects

Latency under load is queueing plus a slower iteration as the batch grows, so a provider that is fine at noon can show a long TTFT at peak with no change on your side. For a self-hosted server the KV block pool is the capacity limit: concurrent requests times average tokens times bytes per token must fit, and a long max_tokens is not free because it raises preemption risk. Provider batch endpoints are a separate product feature: Anthropic's Message Batches API processes requests asynchronously, most batches finish within an hour, batches expire if processing does not complete within 24 hours, and its page states a 50% cost reduction (as of 2026-10-22, batch processing). The document-classification case file uses this trade of delay for cost.

Under the hood: Quantization

Intuition

Weights are stored as 16-bit numbers. Store them in 8 or 4 bits and the model needs a half or a quarter of the memory, and since decode is limited by how many bytes it reads per token, it can also run faster. The price is rounding error, which must be small enough not to hurt your task.

Mechanism

Weight memory is parameters x bits / 8. For an 8B-parameter model: 16 bits is 16 GB, 8 bits is 8 GB, 4 bits is 4 GB, plus small per-group scales whose size depends on the scheme. At the 2,039 GB/s used above, the decode floor falls from 7.85 ms to 3.92 ms to 1.96 ms per token. That is a ceiling for weight-bound decode, not a forecast.

LLM.int8() (Dettmers et al.) quantizes the matrix multiplications of feed-forward and attention projection layers to 8 bits, halving inference memory with no loss in their tests; it multiplies the emergent outlier dimensions that ruin naive quantization in 16 bits while more than 99.9% of values are multiplied in 8 bits. GPTQ (Frantar et al.) is a one-shot post-training method using approximate second-order information that compresses weights to 3 or 4 bits, quantizing a 175B-parameter model in about four GPU hours with negligible accuracy degradation in their measurements. They report per-token speedups at batch size 1 for 3-bit OPT-175B of about 3.25 times (one A100, against FP16 on 5 GPUs) and 4.5 times (two A6000, against FP16 on 8 GPUs). They explain that dequantizing costs extra compute but the kernel reads far less memory, and they note that LLM.int8() reduces memory but has the same runtime as the FP16 baseline and that they do not quantize activations.

Figure 9. Offline quantization and online use

How to read this: the top block is paid once and may change quality, the loop is where the memory saving shows up as speed.

What this means for architects

A quantized model is a different model. Papers measure perplexity and standard benchmarks, not your task, so a quantized variant needs its own run on your golden set before it replaces anything, and the accuracy cost varies with model size, bit width and scheme. For self-hosting, smaller weights leave more memory for the KV cache, which raises concurrency, so quantization is a capacity lever as well as a latency lever. For API models, whether and how a provider quantizes is generally not documented; treat any silent behaviour change as something your regression gate must catch.

Under the hood: Speculative decoding

Intuition

A small draft model guesses the next few tokens cheaply. The large model then checks all guesses in a single pass, keeps the ones it would have produced itself and replaces the first wrong one. Because decode is memory-bound, checking several positions in one pass costs little more than generating one token, so accepted guesses are nearly free.

Mechanism

Leviathan, Kalman and Matias (Google) and Chen et al. (DeepMind) independently published the method. With draft distribution q and target distribution p, a drafted token is kept with probability min(1, p/q); on the first rejection a replacement is sampled from the normalised positive part of p - q. Both papers prove that the output distribution is identical to sampling from the target alone (Leviathan et al., Appendix A.1; Chen et al., Theorem 1), and both state that each target pass yields at least one and at most gamma + 1 tokens (gamma, called K by Chen et al., is the number of guesses).

Only Leviathan et al. derive the speed formulas below; the one theorem in Chen et al. is the distribution proof, and their speed results are measurements. Let alpha be the average probability that a guess is accepted; Leviathan et al. assume it is the same at each position (independent and identically distributed), which makes the tokens per pass a capped geometric variable with expectation E = (1 - alpha^(gamma+1)) / (1 - alpha) (their Equation 1). Their Theorem 3.8 gives the expected wall-time improvement E / (gamma x c + 1), where c is the time of one draft run divided by that of one target run, assuming gamma + 1 target evaluations can run in parallel without raising wall time. With alpha = 0.8 and gamma = 4, E = (1 - 0.8^5) / 0.2 = 3.36 tokens per pass; at c = 0.05 the improvement is 3.36 / 1.2 = 2.80 times, at c = 0.2 it is 3.36 / 1.8 = 1.87 times. Their Theorem 3.11 gives the cost in arithmetic: operations per token rise by (gamma x c-hat + gamma + 1) / E, 1.55 times here (c-hat = 0.05 is the draft's arithmetic cost ratio). Measured: speedups of 2.3 to 3.4 times on T5-XXL, which the paper summarises as 2X-3X (Leviathan et al.), and 2 to 2.5 times on Chinchilla 70B in a distributed setup at batch size 1 (Chen et al.), who also find that the gain plateaus or regresses as the number of guesses grows.

Figure 10. One speculative step with four guesses

How to read this: the target model runs once per step, and the number of tokens emitted depends on how many guesses it accepts.

What this means for architects

Speculative decoding is mostly a server-side optimisation, and with an API a provider may or may not use it: that is not visible to you. Where you run the server it reduces latency without changing the output distribution, but it needs spare compute (the paper says it does not help where additional compute is not available), so its gain shrinks as batches grow and it can reduce throughput at high load. It needs a draft model that agrees often with the target and shares its tokenizer; the acceptance rate alpha is workload-dependent and must be measured on your traffic, since the gain collapses as alpha falls.

Under the hood: Prompt caching and prefix stability

Intuition

The cached state of a token depends on every token before it, so two prompts can share cached work only for the part where they are identical from the very first token. The vLLM paper makes the point directly: the same tokens can have different KV cache at different positions. Put the part that never changes first and the part that always changes last, and most of the work is reused. Change one early byte and everything after it is recomputed.

Mechanism

How providers implement this is mostly undocumented; the contract is documented. OpenAI's page says the cache stores key-value tensors, not tokens, that reuse requires the entire rendered prefix to match, and that cached state lives on individual machines, where traffic above 15 requests per minute can lead to overflow routing. Anthropic's page says the cache covers the prompt in the order tools, system, messages, up to the block marked with cache_control (up to 4 breakpoints, or one automatic breakpoint that moves forward), that a hit needs 100 percent identical segments, and that the default lifetime is 5 minutes, refreshed on each use, with a 1-hour option at extra cost. A change at one level invalidates it and all later levels, so editing a tool definition invalidates tools, system and messages. An entry becomes available only after the first response begins, so parallel requests that must share a prefix should wait for the first to start. Usage splits input as cache_read_input_tokens + cache_creation_input_tokens + input_tokens. Minimum length is model-specific: as of 2026-10-22 Anthropic lists 512 to 4,096 tokens by model, and OpenAI lists 1,024 for its newest models and says earlier ones vary with request settings. Shorter prompts are processed without caching and without an error.

Figure 11. A prefix cache across three requests

How to read this: the cache is inside the provider and not directly visible, so the usage fields in each reply are the evidence of a hit or a miss.

Worked example with the multipliers Anthropic's page states as of 2026-10-22 (writes 1.25 times the base input rate for the 5-minute lifetime, reads 0.1 times; some models use other read multipliers, so check the page for yours). A 6,000-token prefix sent in 10 requests within the lifetime costs 10 x 6,000 = 60,000 token-equivalents uncached, against 1.25 x 6,000 + 9 x 0.1 x 6,000 = 7,500 + 5,400 = 12,900 cached, which is 78.5% less. The break-even is the second request: cached cost for n requests is 1.25 + 0.1 (n - 1) prefix-lengths, which is 1.35 for n = 2 against 2 uncached.

What this means for architects

Prefix stability is an architectural property, set by prompt layout and release practice. Order the request from most stable to least: tool definitions, system prompt, shared reference text, retrieved passages, then the user turn. Keep timestamps, request ids and user names out of the start, serialise tool schemas in a fixed order, and do not reorder tools per request. Every prompt release is a one-time cache rewrite, so ship prompt changes in batches. Compare request rate with lifetime: 1,000,000 requests a month is about 23 a minute on average, which keeps a 5-minute entry warm by day and may let it expire at night. Alert on the cache-read share in the usage fields, since a silent layout change shows up only there. Per-tenant data early in the prompt gives each tenant its own prefix and defeats sharing. For privacy, read the provider's data terms: OpenAI says cached key-value tensors may be stored in GPU-local storage.

Under the hood: Streaming responses

Intuition

Instead of holding the answer until the model finishes, the provider sends tokens as they are produced. The total time is unchanged, but the user sees text after the time to first token and can start reading, and the interface can react (stop, cancel) before the end.

Mechanism

Provider streams use server-sent events (SSE), defined in the WHATWG HTML Standard. The response has the media type text/event-stream and is always UTF-8: events separated by blank lines, each made of lines with a field name and a value. data carries the payload (several data lines are joined with newlines), event names the type, id sets the last event id and retry the reconnection time. A line starting with a colon is a comment, and the standard suggests one about every 15 seconds so proxies do not drop an idle connection. A browser EventSource reconnects after a drop and sends a Last-Event-ID header; a status other than 200 or a wrong content type fails the connection with no reconnect, and HTTP 204 tells the client to stop. An incomplete trailing event is discarded.

Anthropic's streaming documentation (as of 2026-10-22) builds on this: a message_start event, content blocks each with a start, deltas and a stop, message_delta events with cumulative usage, a final message_stop, and ping events anywhere. Tool-call arguments stream as partial JSON strings parsed only after the block stops. Errors can arrive inside the stream, for example an overloaded_error that would be HTTP 529 without streaming. The page describes resuming an interrupted stream by capturing the partial response and sending a continuation request, and says tool-use and thinking blocks cannot be partially recovered.

Figure 12. A stream through the system, with a failure after the first token

How to read this: the same arrows carry every delta from the provider to the screen, and the else branch shows the case in which the HTTP status can no longer signal the error.

What this means for architects

Streaming moves a failure into the middle of a response. Once headers are sent the status is 200, so errors must travel in-band and the client must show a partial answer as incomplete. Do not retry silently after the first token: shown text cannot be unsaid, and a retried tool call may run twice. Output checks that ran on a complete answer must now run on partial text or after it, so decide per intent which replies may stream; the in-product assistant case makes this decision. Every hop (gateway, reverse proxy, CDN) must pass the stream through unbuffered, and idle timeouts must allow gaps between tokens. JSON cannot be parsed until complete. Usage arrives at the end, and whether closing the connection stops generation and billing is provider-specific, so test it.

Decision framework

1. If requests have a type that is known before the call (endpoint, UI surface, tier, intent), then route with rules first, because rules are free, deterministic and testable; revisit when a review of misrouted requests shows that difficulty is not visible in the metadata.
2. If cheap answers can be scored reliably and a slower tail on escalated requests is acceptable, then use a cascade, otherwise a classifier router, because a cascade pays for two calls on every escalation and is only as good as its scorer; revisit when the measured escalation rate approaches the break-even rate (1 minus the small-to-large cost ratio).
3. If the prompt has a stable prefix above the provider's minimum and traffic keeps it warm, then lay the prompt out for prefix caching before adding routing, because it cuts cost and TTFT without changing any answer; revisit when prompt releases are more frequent than the cache lifetime.
4. If an answer depends on the user, the tenant, the time or changing data, then do not use a semantic cache for it and key any exact cache on all of those inputs, because a wrong hit is an incident and not a miss; revisit when the content versioning is strong enough to expire entries exactly.
5. If the user is waiting and the answer is longer than a sentence, then stream, and set separate objectives for time to first token and per-token time; if no one is waiting, do not stream and prefer batch execution, because streaming adds failure modes and buys nothing offline; revisit when the output checks can run on partial text.
6. If data residency, version pinning or sustained high utilisation is a hard requirement, then evaluate self-hosting with an honest operations cost, otherwise stay on an API behind a Model gateway, because the gateway keeps the exit open; revisit when volume or regulation changes.

Comparison of the main options

Failure scenarios to design for

1. The router sends a hard request to the small model and nobody notices. Defined behaviour: every call logs route, reason and confidence; a sample of small-model answers per intent is reviewed weekly; tool errors, schema failures and thumbs-down escalate the next turn to the large model; the golden set has a slice per route.
2. The cascade scorer accepts a wrong answer (false accept). Defined behaviour: the threshold is set on labelled data for a target false-accept rate, intents with costly errors are on the always-large rule list, and a rise in the share of low-confidence answers raises an alert because it signals drift.
3. A semantic or exact cache returns another tenant's answer or an answer from an old policy. Defined behaviour: tenant and content version are part of the key, entries expire when the content version changes, hits are logged with the matched query, and cross-tenant probes are part of the test suite.
4. The prefix cache goes cold after a release, a traffic gap or a layout change. Defined behaviour: the gateway tracks the cache-read share per route, a drop below a set share alerts, releases include one warm-up request before traffic, and capacity is sized for the uncached case.
5. A model upgrade changes the token count and requests exceed their budget. Defined behaviour: model versions are pinned, the gateway counts with the target model's tokenizer, over-budget requests are trimmed by the stated policy or rejected with a defined error, and a version change reruns the budget table.
6. A stream fails after the first token. Defined behaviour: the partial text is kept and marked incomplete, no tool call is executed from a partial response, the user is offered an explicit retry, and the usage record notes the interruption.
7. Structured output is truncated by the token limit. Defined behaviour: the stop reason is checked before parsing; on a token-limit stop the request is retried once with a higher limit or sent to review, never parsed as if complete.

Worked example

A support assistant sends 1,000,000 requests a month. Cost is measured in large-model input-token equivalents (LTE) so that no price appears; use the LLM cost estimator with real prices.

Assumptions: this is a composite scenario and every figure is illustrative. Each request has a 6,000-token stable prefix, 300 variable input tokens and 250 output tokens. An output token weighs 4 LTE. A small model costs 0.1 of the large model per token. Prefix-cache writes weigh 1.25 and reads 0.1 (as stated for Anthropic as of 2026-10-22; check your provider), with a 90% hit rate on every model. In the cascade, every request goes to the small model first and 25% escalate, so an escalated request pays for the small attempt and then the large call.

Three checks keep this honest. The cascade costs 0.1 + f of the large-only cost for escalation rate f, so it stops paying at f = 0.9, and the real saving depends on the measured f and on quality at the threshold. The perfect classifier is an upper bound no real router reaches; a real one adds a small call and makes errors. And the 90% hit rate is an assumption: the large model sees about 5.8 requests a minute on average (0.25 x 23.1), so its prefix can expire in quiet hours. Read the real rate from the usage fields. Output cost is untouched by caching and driven by output length, which is why the weight of 4 per output token matters as much as the cache.

A routing policy you can run

The policy below combines a rule layer with a cascade. The scorer (confidence) must be validated on a golden set before its threshold means anything, and the models are passed in as functions so the policy can be tested without a provider. The block was executed with stub models for the four paths: a confident small answer, a low-confidence escalation, a small-model timeout and an always-large intent.

import logging
from dataclasses import dataclass
from typing import Callable

log = logging.getLogger("model-gateway")

@dataclass
class Answer:
    text: str
    confidence: float

Model = Callable[[str], Answer]
ALWAYS_LARGE = {"refund_dispute", "complaint", "legal_threat"}
THRESHOLD = 0.80

def route(intent: str, prompt: str, small: Model, large: Model) -> Answer:
    if intent in ALWAYS_LARGE:
        log.info("route=large reason=rule intent=%s", intent)
        return large(prompt)
    try:
        first = small(prompt)
    except TimeoutError:
        log.warning("route=large reason=small_timeout")
        return large(prompt)
    if first.confidence >= THRESHOLD:
        log.info("route=small confidence=%.2f", first.confidence)
        return first
    log.info("route=large reason=low_confidence confidence=%.2f", first.confidence)
    return large(prompt)

Read it as three layers. The set ALWAYS_LARGE is the rule layer: intents with costly errors never reach the small model. The try block is the cascade, with the timeout as an escalation trigger. Every branch logs the route and the reason, which is what makes the policy auditable. A production version would also pass the remaining deadline to the large call (Figure 3), count tokens before either call, and emit the usage event of Figure 1.

Common mistakes

• Routing on a confidence number nobody validated. A model's self-reported or heuristic score is not calibrated by default; set the threshold on labelled data and re-check it after every model or prompt change.
• Putting a timestamp, request id or user name at the top of the prompt. It makes every request's prefix unique and the prefix cache never hits.
• Counting tokens as words times 1.3, or with another model's tokenizer. Counts are model-specific and change with model versions; use the target model's counter.
• Semantic-caching answers that depend on the user's data. The cache returns a plausible answer to the wrong person or an old answer to the right one.
• Tracking only total latency. Time to first token and time per token have different causes and different fixes, and one number hides which one regressed.
• Streaming everything, including batch jobs and tool calls. Streaming adds partial-failure handling and breaks output checks that need the whole answer; use it where a person is reading.

Try it

Time: 25 to 40 minutes.

1. Pick a model whose layers, KV heads and head dimension are published (for example one of the Llama 3 sizes in the table above) and compute its KV bytes per token and per 32,000-token request.
2. Paste a real prompt (about 500 words) into the site's token counter, note the estimated tokens per word, then count the same prompt with a provider's token-counting endpoint for two different models if you have access, and record the difference.
3. Write a latency budget for one feature: an assumed prefill rate, TTFT, TPOT and output length, then compute total time with and without streaming and with and without a cached prefix.
4. List your prompt's parts in cache order (tools, system, reference text, retrieved text, user turn), mark every part that changes per request, and check that the stable part is first and is longer than your provider's minimum cacheable length.

Done when you have a one-page note with the KV bytes per token, the measured tokens per word and its model-to-model difference, the latency split with its arithmetic, and a prompt order in which no per-request value appears before the stable prefix.

Key takeaways

• A model gateway owns budgets, routing, caching, provider retries and usage records; the Orchestrator owns prompts and decisions.
• Route with rules where the request type is known, a classifier where data supports it, and a cascade only with a validated scorer; the cascade break-even is 1 minus the cost ratio.
• Tokens are model-specific. Count with the target model's tokenizer, and treat a model upgrade as a budget event.
• KV-cache bytes per token = 2 x layers x KV heads x head dimension x bytes per value; it turns context length into memory and limits concurrency.
• Prefill is parallel and compute-bound, decode is sequential and memory-bound, so output tokens cost more latency than input tokens; set separate objectives for TTFT and TPOT.
• Prefix caching works only on an identical start of the prompt: put stable content first, keep per-request values out of the top, and watch the cache-read share.
• Streaming improves perceived latency and moves failures into the middle of a response; design for partial output.

Self-check

Further reading

• Kwon et al., Efficient Memory Management for Large Language Model Serving with PagedAttention (SOSP 2023), and Yu et al., Orca: A Distributed Serving System for Transformer-Based Generative Models (OSDI 2022).
• Leviathan et al., Fast Inference from Transformers via Speculative Decoding, and Chen et al., Accelerating Large Language Model Decoding with Speculative Sampling.
• Chen et al., FrugalGPT, and Ong et al., RouteLLM, for cascades and learned routers.
• Provider documentation: Anthropic prompt caching and OpenAI prompt caching.
• WHATWG, HTML Standard: Server-sent events.

Last verified: 2026-10-22

Next

Next is Data and retrieval architecture for AI systems, which covers how documents are ingested, chunked, indexed and retrieved safely.