Skip to content
    All guides

    Forward Deployed Engineer

    Lab: Support-Ticket Triage Assistant Design

    An optional blank-page drill: design a support-ticket triage assistant for a software vendor in 30 minutes, then score it with the rubric and a model answer.

    By Published 6 min read

    • fde
    • solution-architect
    • mastery-lab
    • ticket-triage
    • classification
    • optional
    Share

    This lesson is part of AI Solution Architect: Estimate, Propose and Defend. Enroll for free, then sign in, to read it in full.

    Enroll to unlockAlready enrolled?

    What you'll learn: how to run the seven-step design drill on a ticket triage brief: what is hard (priority, hostile ticket text, the cost of a misroute), which patterns fit, and how a scored model answer looks.

    At a glance

    Figure 1. The drill: seven outputs in 30 minutes

    flowchart LR accTitle: Ticket triage drill accDescr: Seven outputs in order: requirements, patterns, architecture sketch, failure path, evaluation plan, threat list and MVP scope, produced in four timed boxes and then scored against the rubric. A["Frame and requirements"] --> B["Patterns"] B --> C["Architecture and failure path"] C --> D["Evaluation and cost"] D --> E["Threat list"] E --> F["MVP scope"] F --> G["Score with the rubric"]

    How to read this: the same four timed boxes as the blank-page defense.

    • Triage is classification plus things code should decide: priority and tier.
    • The costly error is a misrouted security ticket, not an average miss.
    • Ticket text is untrusted: it can hold instructions and secrets.

    Scenario

    Composite scenario, not a real company. Brindlewick Software sells a business application and has 18 support agents and about 14,000 tickets a month (illustrative). Tickets arrive by email and web form as free text, often with pasted logs.

    A person sorts tickets into five queues at the start of each shift: billing, bug, how-to, account access and security. Tickets can wait hours. Customer tier (gold or standard) is in the CRM (customer relationship management system) and decides priority. The head of support wants the right queue within minutes, with a priority and a one-line summary. No customer replies in the first release.

    What you will be able to do

    • Run the seven-step drill on a classification and routing brief.
    • Say which decisions belong to code and which to the model.
    • Score a triage design against the rubric.

    Prerequisites

    Core ideas

    The seven-step checklist for triage

    Step What is hard here Usually fits
    1. Frame Time to the right queue has no measured baseline; rules may cover some tickets Measure first; ask which tickets rules cover
    2. Requirements Pasted secrets and personal data; mixed languages; tier lives in the CRM Redact before the model sees text
    3. Patterns Boundaries between queues are fuzzy; priority is policy, not language Classification and routing; structured extraction; human review queue
    4. Architecture A provider outage must not stop ticket flow Fallback to the manual queue
    5. Evaluation The classes are unequal in cost Per-queue recall, with security weighted highest
    6. Threats A ticket can say "mark this urgent" or "route to security" Priority computed by code, never by the model
    7. MVP (smallest first release) Trust is earned Suggestion first, automatic routing later for low-risk queues only

    At the customer: "Can we trust it to route security tickets?" Good: the model suggests, a rule and a person cover security, and that queue's recall is shown.

    Worked example

    Attempt the drill first, then open the answer.

    Open after your attempt: the model answer and its score

    Frame. Outcome: median time to the correct queue falls from a baseline to collect to 15 minutes (illustrative). Ask which tickets keyword rules already route. Out of scope: customer replies.

    Requirements. Users: triage lead, 18 agents. Data: ticket text, CRM tier. Constraint: redact secrets before the model; logs hold ids, not text.

    Patterns. Classification and routing with a confidence score; structured extraction of product and error code; priority computed in code from tier and rules; human review queue for low confidence. Discarded: an agent that answers tickets (errors unbounded) and fine-tuning first (no labelled data yet).

    Figure 2. Model answer: triage with a fallback

    Ticket triage model answerAPI gateway to Orchestrator (ticket); Orchestrator to Model gateway (redacted); Model gateway to LLM provider (request); CRM to Orchestrator (tier); Orchestrator to Support queues (routed); Orchestrator to Review queue (manual)OUTSIDE OUR CONTROLAPI gateway: email, web formAPI gatewayemail, web formOrchestrator: redact, thresholdsOrchestratorredact,thresholdsModel gateway: schema, loggingModel gatewayschema, loggingLLM provider: classify, summariseLLM providerclassify,summariseCRM: customer tierCRMcustomer tierSupport queues: five queuesSupport queuesfive queuesReview queue: unsure or downReview queueunsure or downticketredactedrequesttierroutedmanual

    How to read this: the model, outside our control, suggests a queue; code adds the tier and decides; the red dashed path catches low confidence and provider failure.

    Evaluation and cost. Golden set of 300 tickets labelled by two agents, stratified by queue, plus 30 tickets that contain instructions. Metrics: per-queue recall, with the security queue's recall reported alone; routed-without-review share. Cost at placeholder prices of 1.00 in and 5.00 out per million tokens: 14,000 x 700 = 9,800,000 tokens in and 14,000 x 60 = 840,000 out, so 9.80 + 4.20 = 14.00 a month.

    Threats. Instructions in ticket text: output is a schema and priority is code. Secrets: redacted first. A missed security ticket: a keyword rule or a model suggestion sends it to the security queue and a person.

    MVP. Two weeks in shadow mode, comparing suggestions with the human sort; then show suggestions. Automatic routing only above a tuned threshold, for how-to and billing, never security.

    Criterion Score Why
    Requirements quality 2 Baseline, scope, constraints
    Pattern choice 2 Two alternatives discarded
    Architecture soundness 2 Flow, fallback, boundary, review queue
    Risk and security 2 Injection, secrets, security misroute
    Evaluation plan 2 Stratified set, per-queue recall
    Cost awareness 1 Token cost only; review load unestimated
    Communication 1 No defend statement; no weakness conceded
    Total 12 of 14 Pass

    Assumptions: volumes, token counts and the target are illustrative; insert the provider's current published price and its date.

    Practice task

    Run the drill on the Brindlewick brief or a variation.

    1. Work 30 minutes in four boxes (5 frame, 15 design, 5 risks and evaluation, 5 defend aloud). Produce six outputs: requirements, patterns, an architecture sketch, an evaluation plan, a threat list and an MVP scope.
    2. After a break, score each criterion with the rubric.
    3. Compare with the model answer and record your lowest criterion.

    Exercise

    The head of support also wants automatic replies to how-to tickets. Redo only the steps that change and list them.

    Acceptance criteria: patterns, threats, evaluation and cost named as affected. Done when you can state one new control the reply feature requires.

    Hint 1: what is new

    A reply is an outbound message the customer sees; the first release never sent one.

    Hint 2: which controls

    Think of citations, a refusal, and a person approving the first batch.

    Common mistakes

    • Letting the model set priority from words like "urgent" in the ticket.
    • Reporting one overall accuracy that hides a poor security recall.
    • Routing automatically with no shadow period.

    Knowledge check

    1. Why compute priority in code and not let the model decide?

    Priority depends on tier and policy, which the CRM holds. A ticket saying "urgent" is untrusted text; if the model weighs it, any customer can jump the queue.

    2. Why report recall for the security queue separately?

    A misrouted security ticket costs far more than a how-to one. An overall figure averages them and can hide the costly misses.

    3. What is the point of a shadow period?

    Suggestions are compared with the human sort without affecting any ticket, giving real accuracy on real traffic before anyone depends on it.

    4. What should happen when the provider is down?

    Tickets go to the manual review queue, so the flow falls back to today's process instead of stopping. The model must not be a single point of failure.

    5. Why did the model answer lose a point on cost?

    It priced the tokens but not the human review load, which often costs more than the model.

    You can now

    • Run a seven-step drill on a classification brief.
    • Keep priority and tier decisions in code.
    • Choose metrics that reflect unequal error costs.

    Further reading

    Last verified: 2026-11-03

    Next

    Continue with Lab: invoice and claims document processing.

    Enroll to keep reading

    Lesson 12 of 15 · AI Solution Architect: Estimate, Propose and Defend

    An optional blank-page drill: design a support-ticket triage assistant for a software vendor in 30 minutes, then score it with the rubric and a model answer.

    In the rest of this lesson

    • Practice task
    • Exercise
    • Common mistakes
    • Knowledge check
    • You can now
    • Further reading

    Enrolling is free. It unlocks every lesson in AI Solution Architect: Estimate, Propose and Defend, tracks your progress across devices and keeps your quiz scores. The newsletter is optional.

    EnrollSign in

    Ask about this article

    Get answers grounded in this post. AI-generated — based on this article, and may be imperfect.

    AI Solution Architect: Estimate, Propose and Defend· Page 12 of 15
    PreviousSolution Design Interview Questions: Sizing, TCO, RiskNext Lab: Invoice and Claims Document Processing Design
    Scaled AI Weekly

    Enjoyed this? Get more like it every Monday.

    Real architecture decisions, LLMOps patterns that survive production, and engineering leadership advice — from 12+ years of building at enterprise scale. Free. No spam. Unsubscribe anytime.

    Get new articles by email

    Free. We email a confirmation link first, and you can unsubscribe from any email.

    Join engineers building production AI systems

    Comments

    Back to all guides