RAG and Agent Interview Questions: Retrieval to Approval

What you'll learn: how to answer four RAG and agentic workflow questions (debugging a retrieval miss, proving access control and reading an evaluation honestly, designing a bounded tool loop with human approval, and deciding when not to use an agent) using real behaviour from a tested reference service.

Interviewers use these questions to separate people who have wired a retriever to a model from people who have had to explain a wrong answer to a customer. They listen for where you look first, what you measure, and which decisions you keep in code.

The examples come from the course capstone for the fictional Lumen Industrial Supply. Its retrieval is hand-written BM25 keyword search, with no embeddings and no vector store. Its default provider is a rule-based mock, not a model. Its evaluation is in-sample: 36 questions written by the person who tuned the pipeline.

Figure 1. The capstone's request path and where each question applies

flowchart LR accTitle: Request path through the discovery pipeline accDescr: The request carries a signed identity, which gives the allowed visibilities. A bounded loop of three read-only tools runs, then evidence is retrieved by keyword search with the filter inside the query, trimmed to a token budget, drafted by the model and checked in code before the answer, refusal or clarifying question is returned. A["Request and signed identity"] --> B["Q2: role gives allowed visibilities"] B --> W["Q3: bounded loop of three read-only tools"] W --> C["Q1: keyword search with the filter inside the query"] C --> D["Context budget keeps the best chunks"] D --> E["Model drafts recommendations"] E --> F["Code keeps real SKUs and real citations"] F --> G["Answer, refusal or question"]

How to read this: follow the arrows left to right; Q4 asks whether the loop in the middle needs a model at all.

How to use this page

Answer aloud for four minutes, then compare. Every code excerpt is copied from a file in capstone/fde-product-discovery-copilot and checked against it; the numbers come from eval/results/latest.md or from tests you can run.

For the design background, see data and retrieval, agents, tools and MCP and evaluation.

Questions

Q1. A sales engineer says the assistant ignored the humidity guidance that is in your documents. How do you find out why?

What it assesses. Whether you debug retrieval systematically instead of blaming the model, and whether you know what keyword search can and cannot do. The interviewer wants a place to look first and a way to confirm the fix.

Answer framework.

Reproduce with the exact question and the same user role.
Place the failure: not in the corpus, in the corpus but not retrieved, retrieved but cut by the budget, or retrieved but ignored.
Inspect the chunks: boundaries, headings, size, and the words they share with the query.
Find the rank of the chunk you wanted, and what beat it.
Change one thing, re-run the whole golden set, and compare before and after.

Strong answer example. The answer reads evidence, not intuition.

I would print the chunk ids that reached the model. For the example request (vibration sensors for 40 pumps in a humid plant, OPC UA), the context holds the VibraSense datasheet, the EdgeLink gateway datasheet and the OPC UA compatibility note. The humidity FAQ is absent. So the failure is the second kind: in the corpus, not retrieved.

Then the rank. In a test probe (the request plus the two candidate SKUs, as the pipeline builds it), the best FAQ chunk ("What rating do I need for a humid plant?") ranked 11th among the chunks this user may see, and the compatibility note's "Checking an existing setup" section ranked first. The request has more words about OPC UA than about humidity, and BM25, which scores by shared words, rewards the topic with more words.

The evaluation agrees. Under the production strategy, q01 scores 66.7% document recall at 3, and q02 and q07 score 50%, because one or two documents dominate the evidence. The recommended SKUs for q01 are still right, so the answer is correct but its evidence is thin.

Chunk size is not the cause. The window setting is 120 words, but sections are short and a chunk never crosses a heading, so the 174 chunks from 30 documents average about 32 words.

Verified: capstone/fde-product-discovery-copilot/app/retrieval/chunking.py

def chunk_document(doc: Document, size: int = 120, overlap: int = 20) -> list[Chunk]:
    if size <= 0 or not 0 <= overlap < size:
        raise ValueError("need size > 0 and 0 <= overlap < size")
    chunks: list[Chunk] = []
    for heading, lines in split_sections(doc.body):
        for window in _windows(lines, size, overlap):
            chunks.append(
                Chunk(doc.doc_id, len(chunks), doc.title, heading, "\n".join(window),
                      doc.skus, doc.visibility, doc.status)
            )
    return chunks

Verified: capstone/fde-product-discovery-copilot/lessons/REF-rag-agents/test_ref_rag_agents_checks.py

def test_chunks_on_this_corpus_are_far_smaller_than_the_chunk_size_setting() -> None:
    sizes = [len(c.text.split()) for c in SERVICES.store.chunks.values()]
    assert len(SERVICES.store.documents) == 30 and len(sizes) == 174
    assert max(sizes) <= SETTINGS.chunk_words
    assert 31 < sum(sizes) / len(sizes) < 33

Verified: capstone/fde-product-discovery-copilot/lessons/REF-rag-agents/test_ref_rag_agents_checks.py

def test_the_best_humidity_faq_chunk_ranks_eleventh_for_the_example_request() -> None:
    query = f"{PUMPS} LUM-VS-100 LUM-GW-200"  # the pipeline appends the candidate SKUs to the request text
    hits = SERVICES.store.search(query, frozenset({"public"}), 12, exclude_superseded=True)
    refs = [h.chunk.ref for h in hits]
    assert refs[0] == "compat-opcua#5"
    assert refs.index("faq-humid-environments#1") == 10

Fix	What it changes	Risk
Search once per stated requirement (environment, protocol)	Each topic gets its own ranking	More code, more queries
Raise `top_k`	More chunks reach the model	The token budget drops some, and cost rises
Add embeddings	Catches paraphrases with no shared words	A new store to run and a new way to be wrong

My default is one search per requirement, then a re-run of all 36 questions. I would not start with embeddings: the evidence for this question says the miss is ranking, not paraphrase. I would change my mind if the failing questions shared no words with the documents.

Common mistakes.

Blaming the model before looking at the retrieved chunks.
Tuning chunk size with no measurement.
Fixing one question and not re-running the others.
Calling a keyword miss a prompt problem.

Follow-up questions.

How do you tell a paraphrase miss from a ranking miss?
What do you change if the right chunk is retrieved but cut by the context budget?
Two documents state different specs. What should the answer do?

Scoring rubric.

Criterion	Weak	Acceptable	Strong
Diagnosis	Blames the model	Checks what was retrieved	Places the failure in one of four stages with evidence
Evidence	Opinion	Reads a few chunks	Reads chunk ids and the rank of the wanted chunk
Fix choice	Larger chunks or a bigger model	A plausible change	A change matched to the cause, with the alternative and when it would win
Verification	Retests the one case	Retests a few	Re-runs the full set and checks for regressions

Q2. You added a visibility filter to retrieval. How do you prove a public user can never see partner-only content, and what does your evaluation tell the customer?

What it assesses. Whether you enforce access control where the data is selected, test it directly, and report a number with its limits. A strong candidate also says what the metric cannot see.

Answer framework.

Take identity from a verified token, never from the request body or the model.
Apply the filter inside the query, before ranking, and in every tool.
Make "hidden" and "missing" look the same.
Test the control directly, then measure leaks on questions designed to provoke them.
Report the number with its sample size, who wrote the set, and what it does not cover.

Strong answer example. Start with where the filter sits.

The index receives a predicate and applies it before scoring. The top results are chosen only among allowed chunks, and the ranking statistics ignore hidden ones. A filter applied after the search would return fewer results than asked for and would let hidden documents influence the order.

Verified: capstone/fde-product-discovery-copilot/app/retrieval/filters.py

def chunk_filter(
    chunks: Mapping[str, Chunk],
    visibilities: Collection[str],
    exclude_superseded: bool = False,
    skus: Collection[str] | None = None,
) -> Callable[[str], bool]:
    wanted = set(skus) if skus else None

    def allowed(chunk_ref: str) -> bool:
        c = chunks[chunk_ref]
        return (
            c.visibility in visibilities
            and not (exclude_superseded and c.status == "superseded")
            and (wanted is None or bool(wanted.intersection(c.skus)))
        )

    return allowed

Verified: capstone/fde-product-discovery-copilot/tests/test_retrieval.py

def test_bm25_where_filter_runs_before_ranking() -> None:
    items = [(f"doc{i}", "pump sensor") for i in range(5)] + [("secret", "pump sensor pump sensor pump")]
    idx = BM25Index(items)
    assert idx.search("pump", 1)[0][0] == "secret"
    hits = idx.search("pump", 3, where=lambda i: i != "secret")
    assert len(hits) == 3 and all(h[0] != "secret" for h in hits)  # still k results, none hidden

The evaluation compares three strategies. The baseline has no filter and leaked partner-only content on 5 of 36 questions (q08, q10, q12, q33 and q34). Strategies A and B leaked on none. A test pins the same behaviour.

Verified: capstone/fde-product-discovery-copilot/tests/test_discovery.py

def test_without_the_filter_the_baseline_leaks_partner_content(services: Services) -> None:
    q = "What is the partner price for the LUM-VS-900?"
    leaked = run_discovery(q, PUBLIC, services, BASELINE)
    assert any(ref.startswith("partner-programme-guide") for ref in leaked.context_refs)
    assert not run_discovery(q, PUBLIC, services, STRATEGY_A).context_refs

To the customer I would say: zero leaks on 36 questions, all asked as a public user, by the author who also built the filter. That is a regression check, not a proof. It does not cover a second tenant, a person whose access was just removed, or a document with the wrong flag. Here visibility is one flag per document; in production it would come from the source system and lag behind it, so I would test a revoked user.

One more honesty point. Groundedness is 100% under all three strategies, because the mock composes answers only from the context it is given. A metric that cannot fail cannot separate options with this provider; it would be informative with a real model.

Common mistakes.

Filtering after retrieval, or relying on "do not reveal partner data" in the prompt.
Reading the role from the request body.
Evaluating on questions that never try to leak anything.
Reporting "100%" with no sample size.

Follow-up questions.

A user loses access at 10:00. When does the index reflect that?
How would you test isolation between three tenants?
Can a leak happen through a citation, a clarifying question or an error message?

Scoring rubric.

Criterion	Weak	Acceptable	Strong
Enforcement point	In the prompt	After retrieval	Inside the query and in every tool, with identity from a token
Testing	Manual spot check	A unit test	Unit test plus adversarial questions and a count of leaks
Evaluation reading	Quotes the best number	Notes the sample is small	States who wrote the set, what is not covered, and which metric cannot discriminate
Production gap	None	Mentions roles	Names permission sync, revocation lag and tenant isolation

Q3. Design an agent that can look up products and prepare a quote draft. What limits do you put on it, and where does a human step in?

What it assesses. Whether you keep authority in code. The interviewer looks for read-only versus state-changing actions, argument validation, a bounded loop, and an approval that cannot be reused for different arguments.

Answer framework.

List the actions and split them: read-only, or anything that changes state.
Give the model only read-only tools, and validate every argument against a schema in code.
Bound the loop: step count, time and repeated calls. A tool failure becomes an observation, not a crash.
Route the state-changing action through an approval gate bound to the exact arguments.
Check permissions in code, and log each step.

Strong answer example. The model gets three read-only tools: search the catalogue, read a product record, check protocol support. The registry refuses to register anything else, and a request for an unregistered tool, including create_quote_draft, returns an error to the model.

Verified: capstone/fde-product-discovery-copilot/app/workflow/tools.py

    def register(self, tool: Tool) -> None:
        if not tool.read_only:
            raise ValueError("only read-only tools may be registered; use the approval gate for actions")
        self._tools[tool.name] = tool

Verified: capstone/fde-product-discovery-copilot/tests/test_workflow.py

def test_runner_cannot_be_talked_into_an_unregistered_tool(ctx: ToolContext) -> None:
    evil = json_text({"action": "tool", "tool": "create_quote_draft", "args": {"sku": "LUM-VS-100", "quantity": 1, "customer_ref": "x"}})
    result = WorkflowRunner(ScriptedProvider([evil, json_text({"action": "final"})]), default_registry(), ctx).run(Requirements())
    assert result.observations[0]["ok"] is False and "Unknown tool" in result.observations[0]["error"]

The loop is bounded by default settings of 14 steps and 5 seconds, and a repeated identical call is skipped and recorded. The time limit is checked between steps, so it cannot interrupt a slow call. A real deployment needs a client timeout as well.

For the quote draft, the model may only ask. The gate hashes the action and arguments, allows one use, and needs a second person with the right role to approve. If the arguments change after approval, execution fails.

Verified: capstone/fde-product-discovery-copilot/tests/test_workflow.py

def test_approval_is_bound_to_exact_arguments_and_single_use() -> None:
    gate = ApprovalGate()
    approval = gate.request(ARGS, ALICE)
    gate.approve(approval.approval_id, BOB)
    with pytest.raises(ApprovalError, match="not the ones"):
        gate.execute(approval.approval_id, ARGS.model_copy(update={"quantity": 400}))
    gate.execute(approval.approval_id, ARGS)
    with pytest.raises(ApprovalError, match="used"):
        gate.execute(approval.approval_id, ARGS)
    assert len(gate.quote_drafts) == 1

Limits of the reference build: approvals are in memory and per process, and create_quote_draft is a library function behind the gate, not an HTTP endpoint.

Common mistakes.

Putting permissions in the prompt instead of the tool.
One broad tool that can run anything.
Approving a description instead of the exact arguments.
No step or time bound, so a confused model loops until the budget ends.

Follow-up questions.

Approvers click "approve" without reading. What changes?
Which actions could skip approval, and who decides?
How would approval work if the approver answers an hour later?

Scoring rubric.

Criterion	Weak	Acceptable	Strong
Action split	One tool list	Marks some tools read-only	Read-only tools for the model, state change only through a gate
Enforcement	Prompt instructions	Checks in the handler	Schema validation and role checks in code
Bounds	None	Step limit	Steps, time, duplicates, and failures as observations, plus the known gap
Approval	A yes/no flag	A second person	Bound to exact arguments, single use, requester cannot approve

Q4. A customer asks for "an agent" for product discovery. When would you say no, and what would you build instead?

What it assesses. Judgment about autonomy. The interviewer wants a decision backed by call counts, cost, risk and testability, not a preference for or against agents.

Answer framework.

Ask whether the steps are known in advance. If they are, write the workflow in code.
Count model calls per request and what grows with each step.
List what can go wrong at each step and who carries the risk.
Ask whether you can list the paths well enough to test them.
Choose the least autonomy that meets the requirement, and say what would justify more.

Strong answer example. Measure first. In the capstone, the example request makes 9 model calls: 1 to extract requirements, 7 to plan the loop (six tool calls, then "final") and 1 to write the answer.

Verified: capstone/fde-product-discovery-copilot/lessons/REF-rag-agents/test_ref_rag_agents_checks.py

def test_one_request_makes_nine_model_calls_and_seven_of_them_plan_the_tool_loop() -> None:
    counting = CountingProvider()
    svc = Services(SETTINGS, SERVICES.catalog, SERVICES.store, counting)
    run = run_discovery(PUMPS, PUBLIC, svc, STRATEGY_B)
    assert dict(counting.calls) == {"extract_requirements": 1, "plan_next_step": 7, "compose_answer": 1}
    assert run.response.usage.llm_calls == 9 and run.response.usage.workflow_steps == 6
    plan_share = counting.tokens_in["plan_next_step"] / run.response.usage.input_tokens
    assert 0.75 < plan_share < 0.80  # each planning call re-sends every earlier observation

The planning calls carry 2,162 of the 2,792 input tokens, about 77%, because each one re-sends every earlier observation. At the placeholder prices of 1 and 5 EUR per million input and output tokens, the request costs (2,792 x 1 + 296 x 5) / 1,000,000 = 0.004272 EUR. If code ran the same six tool calls from a fixed script, only extraction and writing would call a model: 630 input and 166 output tokens, or (630 x 1 + 166 x 5) / 1,000,000 = 0.00146 EUR, about 34% of the cost.

Illustrative: token counts are the mock's estimate (1 word is about 1.3 tokens) and the prices are placeholders. Fill in your provider's published prices.

The honest caveat: the mock plans by script, so this says nothing about how a real model would choose. A real model varies the path and the call count, and that variation is the price of agency. I would say no when the sequence is fixed, a step is regulated, or the customer cannot tolerate variance. I would say yes when the next step depends on what the last result says and the paths are too many to write down, with the bounds from Q3.

Common mistakes.

Building an agent because the customer used the word.
Quoting a price per call and forgetting the number of calls.
Saying "agents are unreliable" with no measurement.
Testing only the happy path of a loop whose path varies.

Follow-up questions.

What evidence would make you move one step from workflow to agent?
How do you test a path that changes between runs?
How do you cap cost per request?

Scoring rubric.

Criterion	Weak	Acceptable	Strong
Fit judgment	Always yes or always no	Considers the task	States the condition under which each answer is right
Cost evidence	None	Per-call price	Calls per request, token growth and a computed comparison
Risk	Not mentioned	General worry	Per-step failure and who carries it
Alternative	None	"A script"	A workflow with named model calls and a path to more autonomy

Last verified: 2026-11-02

How to practise

Run ./.venv/Scripts/python -m pytest lessons/REF-rag-agents -q in the capstone folder and read each test name as a claim.
Open eval/results/latest.md, pick one of the five questions listed as not behaving as expected, and say aloud which stage failed.
Redo the Q4 cost arithmetic with your provider's published prices and explain the difference.
Describe the approval gate to a friend in three sentences without saying "human in the loop".
Score yourself with each rubric and repeat the criterion you scored lowest.

Continue to Interview questions: security and reliability, which covers prompt injection, excessive agency, tenant leakage, retries and idempotency, and audit logging.

RAG and Agent Interview Questions: Retrieval to Approval

How to use this page

Questions

Q1. A sales engineer says the assistant ignored the humidity guidance that is in your documents. How do you find out why?

Q2. You added a visibility filter to retrieval. How do you prove a public user can never see partner-only content, and what does your evaluation tell the customer?

Q3. Design an agent that can look up products and prepare a quote draft. What limits do you put on it, and where does a human step in?

Q4. A customer asks for "an agent" for product discovery. When would you say no, and what would you build instead?

How to practise

Next

Enroll to keep reading

Ask about this article

Enjoyed this? Get more like it
every Monday.

Comments

How to use this page

Questions

Q1. A sales engineer says the assistant ignored the humidity guidance that is in your documents. How do you find out why?

Q2. You added a visibility filter to retrieval. How do you prove a public user can never see partner-only content, and what does your evaluation tell the customer?

Q3. Design an agent that can look up products and prepare a quote draft. What limits do you put on it, and where does a human step in?

Q4. A customer asks for "an agent" for product discovery. When would you say no, and what would you build instead?

How to practise

Next

Enroll to keep reading

Ask about this article

Enjoyed this? Get more like it every Monday.

Comments

Enjoyed this? Get more like it
every Monday.