Composite scenario, not a real company. Assembled from common industry patterns; every number is illustrative and its assumptions are listed at the end.
This case belongs to Models, routing and inference for AI Architects.
Context
A B2B project-management product adds an assistant panel to every page. Users ask it to summarise the board they are looking at, explain a field, or draft a status update. On each message the Orchestrator sends the system instructions (2,000 tokens), the descriptions of all 12 tools (3,000 tokens) and the current page (up to 9,000 tokens). The reply returns in one piece, and the panel shows a spinner until it arrives. Support tickets say the assistant "hangs".
Problem statement
Two request profiles are enough to see the problem. A typical request has 8,000 input tokens and a 400-token answer. A long request, which stands in for the slow tail, has 14,000 input tokens and an 800-token answer. Modelled with the rates under Assumptions, the user waits 15.63 seconds for the typical answer and 30.17 seconds for the long one, with nothing on screen until then.
Measurable goal: first visible text within 2 seconds for both profiles, and a typical complete answer within 10 seconds.
Constraints
- The page context cannot be dropped, only trimmed, and answer quality cannot be traded for speed without a check.
- Replies used to be checked in full before display. Streaming shows text before that check finishes, so the team must decide which intents may stream.
- The platform team can change the Orchestrator and the Model gateway, not the provider.
Why the traditional approach fell short
Help centre search and field tooltips answer almost instantly, which sets the user's expectation for every other help feature. They return the same article to everyone and cannot see the board, so they cannot summarise it. Report templates give a fixed status update that cannot adapt to what changed this week. The assistant covers that gap, but a 15 to 30 second wait loses against a search box.
Options considered
| Option | Verdict | Reason |
|---|---|---|
| A. Non-AI contextual help: tooltips, help search and a status-update template for the most common first questions | Kept for definitional questions | Instant and free of model risk; cannot see the board |
| B. Stream the reply and change nothing else | Rejected as the whole answer | First text arrives at 2.33 s for the typical request but 3.53 s for the long one, and the full answer is as slow as before |
| C. Stream, shrink the prefix and page context, cache the stable prefix, shorten answers, and send first-turn intents to a smaller model | Accepted | Targets both time to first text and the tail |
Default choice: C, because B improves only how the wait feels. The tail comes from input size and output length.
Solution architecture
Figure 1. The original request, as a timeline of one non-streamed call
How to read this: the notes carry the time budget, and the only message the user sees is the last arrow.
Figure 2. Components after the redesign
How to read this: the Orchestrator decides what goes into the prompt and which tier answers, and the reply flows back along the same path as a stream.
Figure 3. Runtime path with streaming and a mid-stream failure
How to read this: the loop is the stream, and the alt block is the case where the call fails after the user has already seen text.
How AI solved it, step by step
- A user asks, "Summarise this board." The Orchestrator classifies the intent as a first-turn summary.
- It selects only the tool descriptions that intent needs (1,000 tokens instead of 3,000) and trims the page context to at most 6,000 tokens.
- It builds the prompt with instructions and tool descriptions first and the page content and question last, so the stable 3,000-token prefix can be cached.
- A rule sends first-turn intents of this kind to the small tier, and everything else to the large tier. Unknown intents go to the large tier.
- The system prompt asks for a short answer, about 250 tokens for a typical request.
- The call is made with streaming on, because summaries and field explanations of the user's own board are flagged as safe to show before the check finishes. The first text is forwarded to the client as it arrives. Intents that must be checked first are not streamed.
- The Orchestrator records time to first text and total time, and the content check runs on the finished text and is logged.
Under the hood
The mechanism is the split between prefill and decode, in Under the hood: Prefill and decode. The PagedAttention paper describes the prompt phase as taking the whole prompt at once and being parallelisable, and the generation phase as producing the remaining tokens one at a time, which "severely underutilizes GPU computation" and is "responsible for most portion of the latency of a single request". The DistServe paper names the matching metrics: time to first token for prefill and time per output token for decoding. Anthropic's latency documentation (as of 2026-10-22) recommends a faster model, fewer prompt and output tokens, and streaming.
That gives a budget model: time to first text is about overhead + uncached input / prefill rate + one decode step, and the total is overhead + uncached input / prefill rate + output / decode rate. This is a planning model, not a documented provider formula, and the cache term is an assumption: OpenAI's documentation says caching reduces processing before the response starts but does not guarantee lower latency. Measure the rates on your own traffic.
What went wrong / failure modes
- Streaming fails midway. An error can arrive as an event inside a stream that has already started, as the Anthropic streaming documentation describes. Mitigation: keep the partial text, mark it incomplete and offer a retry. This design does not resume a generation.
- Cold-cache requests set the tail. The first request after the cache lapses pays the full prefill. For the long profile on the large tier this is 2.53 s, above the 2 s target, until the cache is warm. Mitigation: measure time to first text by cache state, not only overall.
- Idle connections. Anthropic's documentation notes delays between events while tool input is produced, and the WHATWG standard says legacy proxy servers are known to drop HTTP connections after a short timeout and suggests a comment line every 15 seconds or so. Mitigation: heartbeat comments and a client-side stall timer.
- Unvetted streamed text. Text is shown before the reply has been checked. Mitigation: do not stream replies that must be validated first (see guardrails).
- Wrong tier. A summary on the small tier can be weaker. Mitigation: unknown intents go up; review sampled answers per tier.
Results
These are illustrative targets from the budget model, not measurements. Seconds to first text and to a complete answer:
| Step | Typical: first text | Typical: complete | Long: first text | Long: complete |
|---|---|---|---|---|
| Original, not streamed (user sees the end) | 15.63 | 15.63 | 30.17 | 30.17 |
| Stream | 2.33 | 15.63 | 3.53 | 30.17 |
| Shrink prefix and context, shorter answers | 1.93 | 10.23 | 2.53 | 19.17 |
| Cache the 3,000-token prefix | 1.33 | 9.63 | 1.93 | 18.57 |
| Small tier for first-turn intents | 0.91 | 3.68 | 1.11 | 6.66 |
With 60% of typical requests on the small tier and 40% on the large tier, the mean time to first text is 1.08 s and the mean complete answer is 6.06 s. Both profiles meet the 2 s goal on the large tier, the long one only just (1.93 s), and its complete answer there is still 18.57 s.
Tradeoffs and when NOT to use this approach
- Streaming improves the feel of the wait, not the total time, and complicates the client (partial rendering, retries).
- Shorter answers are faster but may omit detail users want. Practitioners differ on the trade; test it with users.
- Do not stream output that code will parse or that must be validated before display.
- Do not add a small tier if the intent classifier is less reliable than the latency problem is costly.
- For background jobs with no user waiting, latency work is wasted.
Design rules extracted
- Write the latency budget as stages (overhead, prefill, decode) before choosing fixes, and measure the tail with a long profile.
- Cut input tokens that every request pays for: select tools by intent and cap page context.
- Put the stable prefix first so it can be cached, and track time to first text by cache state.
- Stream user-facing replies, and design the mid-stream failure path on day one.
- Keep a non-AI path for fixed answers.
Assumptions
All numbers are illustrative. Model: time to first text = 0.7 + uncached input / prefill rate + 1 / decode rate; complete = 0.7 + uncached input / prefill rate + output / decode rate.
- Overhead 0.7 s (0.3 s network, 0.4 s context assembly). Large tier: prefill 5,000 tokens/s, decode 30 tokens/s. Small tier: prefill 15,000 tokens/s, decode 90 tokens/s.
- Typical request: 2,000 instruction + 3,000 tool + 3,000 page tokens = 8,000 in, 400 out. Long: 5,000 prefix + 9,000 page = 14,000 in, 800 out.
- Not streamed, typical: 0.7 + 8,000 / 5,000 + 400 / 30 = 0.7 + 1.6 + 13.33 = 15.63 s. Long: 0.7 + 2.8 + 26.67 = 30.17 s. Streamed first text: 0.7 + 1.6 + 0.033 = 2.33 s and 0.7 + 2.8 + 0.033 = 3.53 s.
- Shrink step: tool descriptions 3,000 to 1,000 tokens, long-profile page context capped from 9,000 to 6,000, answers 400 to 250 and 800 to 500 tokens. Typical 6,000 in, long 9,000 in.
- Cache step: the 3,000-token prefix adds no prefill time (assumed). Large tier, typical: 0.7 + 3,000 / 5,000 + 0.033 = 1.33 s, complete 0.7 + 0.6 + 8.33 = 9.63 s. Long: 0.7 + 1.2 + 0.033 = 1.93 s, complete 0.7 + 1.2 + 16.67 = 18.57 s. Cold cache, long: 0.7 + 9,000 / 5,000 + 0.033 = 2.53 s.
- Small tier, typical: 0.7 + 0.2 + 0.011 = 0.91 s, complete 0.7 + 0.2 + 2.78 = 3.68 s. Long: 0.7 + 0.4 + 0.011 = 1.11 s, complete 0.7 + 0.4 + 5.56 = 6.66 s. Means: 0.6 x 0.91 + 0.4 x 1.33 = 1.08 s; 0.6 x 3.68 + 0.4 x 9.63 = 6.06 s.
- The 60/40 split, token counts and rates are invented, as are the product, tools and users.
Last verified: 2026-10-22