Everything above was checked by running the reference build's pipeline with the mock provider. The UI was component-tested, but the reference build's own report says it was not driven in a browser against the running API. Rehearse in the UI, and keep the curl commands from the contract page ready as a fallback.
What the evaluation scene shows
Show the strategy rows from eval/results/latest.md, copied from the harness output.
Read it in one sentence per row. The filter inside the query removes all five leaks, excluding superseded documents removes the two stale citations, and flagging conflicts catches both contradictory-spec questions. Retrieval quality does not differ across strategies: doc recall@3 is 94.2% in every row.
Then show the miss. Question q36 asks whether Lumen sells a white-label version of its monitoring software with the customer's logo. It should be refused, but the service asks clarifying questions, because most of the words appear somewhere in the public documents and the coverage check scores 0.71 against a 0.6 threshold. Nothing partner-only leaks. Naming the miss yourself earns more trust than waiting for someone to find it.
Limits you must say out loud
If you did not tune against your own golden set, say so truthfully for your build. If you did, say that.
The ten-minute interview version
The parts add up to 1:00 + 2:00 + 3:00 + 2:30 + 1:00 + 0:30 = 10:00.
In the interview: you will usually be interrupted. Let it happen. A presentation that survives three interruptions and picks up where it left off shows more than a perfect monologue, and the buffer is for exactly that.
Likely questions and answer outlines
These teach how to reason. Do not memorise them; your answers should come from your build.
Why not let the model answer straight from the catalog?
- Code, not the model, must decide which SKUs exist and whether a citation is real.
- The model proposes drafts; the pipeline keeps a draft only if the SKU is real and the cited chunk names the product.
- That is what makes "no citation, no claim" enforceable and measurable.
How do you know it works?
- Name the golden set: 36 questions, including 9 expected refusals and 4 expected clarifications.
- Quote the numbers from the output file, then state the in-sample caveat.
- Say what you would add: questions written by people who did not build it, and a held-out set that you never tune against.
The provider is a mock. What did you actually learn?
- The mock proves the contracts, the validation, the limits and the evaluation harness, because those do not depend on the model.
- It cannot prove answer quality or prompt robustness. A real model needs its own run, and the compose prompt in particular was only ever answered by the mock.
- Offer the next step: run the same golden set with your own key and compare.
How do you stop partner data leaking?
- Identity comes from a verified token, never from the body or the model.
- The visibility filter runs inside the retrieval query and the tools, so partner-only chunks never reach the context.
- Evidence: the leak count is 5 for the baseline and 0 for the filtered strategies, and security tests cover the roles.
What breaks first in production?
- State lives in one process: idempotency keys and approvals are lost on restart and not shared across instances.
- Keyword search misses paraphrases, and the refusal check is a word-coverage heuristic that can be wrong in both directions.
- Authentication is a development token, not an identity provider.
Why keyword search and not embeddings?
- Start with the simplest thing you can measure. Keyword search is explainable and the failures are visible in the output.
- Name when you would change: when misses come from paraphrase, and the evaluation shows it.
- Say that embeddings need their own evaluation, and that you would compare against the keyword baseline, not replace it on faith.
How would this fit the customer's systems?
- Point to your integration note: where the service sits, how identity passes through, what happens on a timeout.
- Say what you would need from the customer: the catalog source of truth, a refresh path for documents, and an identity provider.
- Mention the rollout path, starting with a pilot group of sales engineers.
Walk me through a failure.
- Use q36. State the expected result, the actual result, the cause (the coverage score of 0.71 against 0.6) and what you would try.
- Say why you did not simply change the threshold: the same threshold was tuned while looking at the other questions, so moving it for one question risks breaking another.
Rehearsal checklist
A demo that works once is not rehearsed. Run it three times with a timer, and check these before each run.
- The service is started fresh, so no idempotency key from an earlier run replays an old answer.
- You know which role the UI is set to before you paste each request.
- The cited file is already open in a second window, scrolled to the sentence.
eval/results/latest.md was produced by a run you can name, and you re-ran the harness after your last code change.- You have one sentence ready for each limit in the table above.
- You have decided in advance which scene you will drop if you are running two minutes late. The usual choice is the conflict scene.
Time each scene, not just the total. If the headline request takes 2 minutes instead of 75 seconds, the problem is the scene, and you can fix it by pointing at three things instead of seven.
Explain it back. In three sentences, tell a sales director why a refusal is a feature. A good answer says what the refusal prevents (an invented product in front of a customer), what it costs (an unanswered question), and how you know it fires when it should (the six unanswerable golden questions and the one known miss).
Backup plan
What not to claim
- That the evaluation is an accuracy estimate.
- That the real provider works, unless you ran it.
- That this build is production-ready.
- That finishing it leads to a job. It is evidence of a skill, and you still have to show the skill.
Last verified: 2026-11-02
Next
This is the last capstone page. To rehearse a design defence under pressure, continue to Architecture review board simulation in the AI Solution Architect course.
To check yourself, take the interview and career practice quiz and then the longer scenario-based practice exam; every wrong answer links to the lesson section to revise. The build modules have their own quiz and exam.