What you'll learn: three patterns for turning a request into something a system can act on (structured extraction, classification and routing, and a deterministic filter plus a model that only explains): when to use each, how it fails, how to evaluate it and what it costs.
At a glance
Figure 1. Where the three patterns sit in the Order and Policy Assistant
How to read this: the Orchestrator extracts fields and picks a route; the order route fetches facts in code and sends only those to the model.
- Extraction turns free text into fields; a field stays empty when the text is silent.
- Routing sends one input to one of a few handlers; include "other" and "unsure".
- Filter plus explain keeps facts in code; the model only words them.
Scenario
Composite scenario, not a real company. At Lumen Industrial Supply, a fictional distributor, an agent types: "Can the customer still change the address on order 48213?" The assistant must find the order number and intent, fetch the status, apply a policy and explain.
What you will be able to do
- Choose between extraction, routing and filter plus explain for a requirement.
- Say how each pattern fails and how to measure it.
- Design route labels with "other" and "unsure".
Prerequisites
- The seven-step design method.
- AI Architect models and routing for the architecture.
Core ideas
Pick a pattern from the requirement
Figure 2. Which understanding pattern fits
How to read this: ask the questions in order.
| Extraction | Routing | Filter plus explain | |
|---|---|---|---|
| Model does | Reads text into fields | Picks one label | Words the facts |
| Main risk | Invented values | Wrong label | Facts altered |
| Check with | Field match | Confusion table | Reply equals facts |
Pattern 1: Structured extraction
- When to use: free text must become known fields.
- When not to: the data already arrives structured, or no fixed schema fits.
- Structure: the "extract" step in the Orchestrator of Figure 1.
- Failure modes: an invented value for a missing field; a value from the wrong sentence.
- How to evaluate: hand-label real messages; exact match per field; count invented values.
- Cost profile: one short call; retries add calls.
- Minimal build: a schema with optional fields, "leave empty if unstated", a validator, one retry, then a clear error (Structured outputs).
- Used in the Order and Policy Assistant: yes: order id and question type, with the id checked against the order system.
Pattern 2: Classification and routing
- When to use: one entry point, a few handlers, distinct categories.
- When not to: one handler only, labels humans disagree on, or a button or keyword rule works as well.
- Structure: the "route" step in the Orchestrator of Figure 1.
- Failure modes: a confident wrong label; real cases missing from the label set.
- How to evaluate: a confusion table on labelled messages, weighting costly errors such as a combined question sent down the order-only route; track the "unsure" share.
- Cost profile: one short call, or none if rules decide (routing policies).
- Minimal build: three to five labels including "other" and "unsure"; the classifier returns only the label; "unsure" asks a question or goes to a person; log every label.
- Used in the Order and Policy Assistant: yes, with labels order, policy, both and other.
Pattern 3: Deterministic filter plus explanation
- When to use: the answer is a fact or shortlist code can fetch or compute exactly, and people need it in plain words.
- When not to: the explanation needs facts code did not fetch, or the field alone will do.
- Structure: the Order system to Model gateway path in Figure 1.
- Failure modes: the model adds a fact or rewords a number wrongly; code sends too much (personal data) into the prompt.
- How to evaluate: code checks every number, id and status in the reply against the fetched value; a person reads a sample.
- Cost profile: one call with a small input, because only filtered facts go in.
- Minimal build: code fetches only the needed fields; the instruction says to explain only those; a check compares reply and facts; on a mismatch, show the facts alone.
- Used in the Order and Policy Assistant: yes, for order status.
In the interview: "When would you not use a model?" A strong answer names what code does exactly (fetch, filter, dates, totals) and where a model helps: reading and wording text.
Worked example
Assumptions: illustrative order 48213, status allocated; the delivery date is in the order system.
| Agent message | Extract | Route | Code does | Model does |
|---|---|---|---|---|
| "Can the customer still change the address on order 48213?" | order 48213, change address | both | Fetch status allocated |
Explain with P-ORD-04 cited |
| "Opened pump delivered 20 days ago, can they return it?" | return, opened, 20 days | policy | Compute days since delivery | Explain P-RET-01 v3 (30 days, 15 percent fee if opened and not defective) |
| "What is a good lunch place?" | nothing found | other | Fixed out-of-scope message | Nothing |
Spot the flaw. A team gives the model the delivery date and today's date and asks: "Is this order inside the return window? Answer yes or no, then explain." What is the flaw?
Answer
Counting days and comparing with 30 is exact, so it belongs in code; a model can miscount. Give the model the result to explain.