Skip to content
    All guides

    Forward Deployed Engineer

    Lab: Internal Knowledge Assistant With Access Control

    An optional blank-page drill: design an internal knowledge assistant that respects per-user permissions in 30 minutes, then score it against the rubric.

    By Published 6 min read

    • fde
    • solution-architect
    • mastery-lab
    • knowledge-assistant
    • access-control
    • optional
    Share

    This lesson is part of AI Solution Architect: Estimate, Propose and Defend. Enroll for free, then sign in, to read it in full.

    Enroll to unlockAlready enrolled?

    What you'll learn: how to run the seven-step design drill on an internal knowledge assistant: why permissions must be enforced inside retrieval, how leaks happen even when the answer is safe, and how a scored answer looks.

    At a glance

    Figure 1. The drill: seven outputs in 30 minutes

    flowchart LR accTitle: Knowledge assistant drill accDescr: Seven outputs in order: requirements, patterns, architecture sketch, failure path, evaluation plan, threat list and MVP scope, produced in four timed boxes and then scored against the rubric. A["Frame and requirements"] --> B["Patterns"] B --> C["Architecture and failure path"] C --> D["Evaluation and cost"] D --> E["Threat list"] E --> F["MVP scope"] F --> G["Score with the rubric"]

    How to read this: the same four timed boxes as the blank-page defense.

    • The search must filter by the asking user's permissions before any text reaches the model.
    • Titles, citations and "no access" messages can leak that a document exists.
    • Permission changes must reach the index.
    • Evaluation needs negative tests: questions that must return nothing.

    Scenario

    Composite scenario, not a real company. Marlow Engineering Group has about 2,000 employees and 30,000 documents in a wiki, shared drives and a ticket system, with permissions by group. HR, finance and legal spaces are restricted. Staff hunt for procedures or ask colleagues.

    Leadership wants an assistant that answers questions in plain words and cites the documents. Each person must see answers built only from documents they could open themselves. About 20,000 questions a month are expected (illustrative).

    What you will be able to do

    • Run the seven-step drill on an access-controlled retrieval brief.
    • Place the permission check where it cannot be bypassed.
    • Write negative tests for access.

    Prerequisites

    Core ideas

    The seven-step checklist for a knowledge assistant

    Step What is hard here Usually fits
    1. Frame "Find answers faster" has no baseline; some questions need a person Measure search time today; define a refusal
    2. Requirements Group-based permissions in several source systems; HR, finance and legal spaces Identity from single sign-on (SSO); per-document access lists
    3. Patterns Answers must be grounded and permitted Retrieval with citations; hybrid retrieval; AI gateway
    4. Architecture A prompt that says "do not reveal" is not a control Filter in the search query, from the user's groups
    5. Evaluation Quality and leaks are different tests Question set plus negative access tests
    6. Threats Stale permissions; cached answers; injection in a document; leaking existence Per-user cache keys; refusal wording that reveals nothing
    7. MVP (smallest first release) One low-risk space first A department wiki, read-only, a small pilot

    At the customer: "Can it show me something I should not see?" Good: permissions are applied in the search itself, negative-test results are shown, and the delay for a revoked right is stated.

    Worked example

    Attempt the drill first, then open the answer.

    Open after your attempt: the model answer and its score

    Frame. Outcome: time to find a procedure (collect the baseline) and the share of answers staff rate useful. Out of scope: HR, finance, legal content. Question: how fast must a removed permission take effect.

    Requirements. Users: all staff, SSO identity. Data: wiki and shared drives, each document with an access list. Constraint: no document, title or snippet is shown to someone who cannot open it.

    Patterns. Retrieval with citations over a keyword and vector index, access lists stored with each chunk, and a model gateway. Discarded: a shared index filtered by prompt after retrieval (a leak waiting for a bug) and one index per user (cost and sync grow with staff).

    Figure 2. Model answer: permission filter inside the search

    Knowledge assistant model answerClient app to API gateway (question); API gateway to Orchestrator (groups); Orchestrator to Retriever (query); Retriever to Client app (no answer); Retriever to Vector index (filtered); Orchestrator to Model gateway (permitted); Model gateway to LLM provider (request)OUTSIDE OUR CONTROLClient app: SSO sign-inClient appSSO sign-inAPI gateway: identity, groupsAPI gatewayidentity, groupsOrchestrator: build queryOrchestratorbuild queryRetriever: filter by user groupsRetrieverfilter by usergroupsVector index: chunks with access listsVector indexchunks withaccess listsModel gateway: logging, no textModel gatewaylogging, no textLLM provider: answer, citeLLM provideranswer, citequestiongroupsqueryno answerfilteredpermittedrequest

    How to read this: the user's groups travel with the query, the index returns only permitted chunks, and only those reach the model, which sits outside our control; no permitted chunk means a plain refusal.

    Evaluation and cost. 150 questions across three roles, including 60 negative tests where the user must get no information; target zero leaks. Also groundedness and refusal rate, plus a revoke test: remove a right, then ask again after the sync window. Cost at placeholder prices of 1.00 in and 5.00 out per million tokens: 20,000 x 3,500 = 70,000,000 tokens in and 20,000 x 300 = 6,000,000 out, so 70.00 + 30.00 = 100.00 a month. Indexing is priced separately.

    Threats. Stale permissions: lists synced on a short schedule. Cached answers: keys include the permission set. Injection in a document: text is data, the model has no tools. Leaking existence: refusals say "no answer found", never "restricted".

    MVP. The engineering wiki only, read-only, 50 pilot users, a negative-test run before each release.

    Defend. "Permissions live in the search query, not the prompt. Weakness: the sync delay for revoked rights, which I would measure before widening access."

    Criterion Score Why
    Requirements quality 2 Baseline, scope, revoke question
    Pattern choice 2 Two alternatives discarded
    Architecture soundness 2 Filter inside retrieval, refusal path, boundary
    Risk and security 2 Stale rights, cache, existence leak
    Evaluation plan 2 Negative tests, revoke test
    Cost awareness 1 Tokens priced; indexing and support effort not estimated
    Communication 2 Decision first, weakness conceded
    Total 13 of 14 Pass

    Assumptions: headcount, volumes and token counts are illustrative; insert the provider's current published price and its date.

    Practice task

    Run the drill on the Marlow brief or a variation.

    1. Work 30 minutes in four boxes (5 frame, 15 design, 5 risks and evaluation, 5 defend aloud). Produce six outputs: requirements, patterns, an architecture sketch, an evaluation plan, a threat list and an MVP scope.
    2. After a break, score each criterion with the rubric.
    3. Compare with the model answer and record your lowest criterion.

    Exercise

    Leadership asks to add the HR space. List what changes before it can be allowed.

    Acceptance criteria: you cover access lists, tests, logging and who approves. Done when you can state the negative test that must pass before HR content is indexed.

    Hint 1: what is different about HR

    A leak costs more and the content holds personal data, so the bar for tests and logging is higher.

    Hint 2: the gate

    Set a release gate: zero leaks on HR negative tests, owned by someone outside the build team.

    Common mistakes

    • Filtering results after retrieval, or asking the prompt to hide things.
    • Testing only that good questions work, never that forbidden ones fail.
    • Showing a "this document is restricted" message that confirms it exists.

    Knowledge check

    1. Why must the permission filter sit inside the search query?

    Text that is never retrieved cannot reach the model or the answer. A later filter, or an instruction in the prompt, depends on every later step working perfectly.

    2. How can an answer leak even when its text is safe?

    Citations, titles, snippets or a "restricted" message can reveal that a document exists. Refusal wording and citations must be limited to what the user may open.

    3. What does a revoke test check?

    It removes a user's right, waits the stated sync window and asks again, proving the change reaches the index. Without it, stale permissions can stay open unnoticed.

    4. Why include the permission set in a cache key?

    Two users asking the same question may be allowed different documents. A shared cache could give one user an answer built from another's documents.

    5. Why start with one department wiki?

    It limits the cost of a mistake while the controls and tests are proven. Widening later is a tested step.

    You can now

    • Run a seven-step drill on an access-controlled assistant.
    • Place the permission check inside retrieval.
    • Write negative tests and a revoke test.

    Further reading

    Last verified: 2026-11-03

    Next

    Continue with Lab: agentic order-exception handler.

    Enroll to keep reading

    Lesson 14 of 15 · AI Solution Architect: Estimate, Propose and Defend

    An optional blank-page drill: design an internal knowledge assistant that respects per-user permissions in 30 minutes, then score it against the rubric.

    In the rest of this lesson

    • Practice task
    • Exercise
    • Common mistakes
    • Knowledge check
    • You can now
    • Further reading

    Enrolling is free. It unlocks every lesson in AI Solution Architect: Estimate, Propose and Defend, tracks your progress across devices and keeps your quiz scores. The newsletter is optional.

    EnrollSign in

    Ask about this article

    Get answers grounded in this post. AI-generated — based on this article, and may be imperfect.

    AI Solution Architect: Estimate, Propose and Defend· Page 14 of 15
    PreviousLab: Invoice and Claims Document Processing DesignNext Lab: Agentic Order-Exception Handler Design
    Scaled AI Weekly

    Enjoyed this? Get more like it every Monday.

    Real architecture decisions, LLMOps patterns that survive production, and engineering leadership advice — from 12+ years of building at enterprise scale. Free. No spam. Unsubscribe anytime.

    Get new articles by email

    Free. We email a confirmation link first, and you can unsubscribe from any email.

    Join engineers building production AI systems

    Comments

    Back to all guides