Threat-model an AI system: trust boundaries, the OWASP LLM Top 10 (2026), prompt injection chains, confused deputy, guardrails, sandboxing and log redaction.
What you'll learn: how to find where text you did not write reaches a component that acts on it, which OWASP risks that creates, and which controls limit the damage: scoped tools, privilege separation, approvals, egress rules, output handling and clean logs. Defensive only: every example is illustrative and non-functional.
At a glance
Figure 1. Trust boundaries and the six entry points of an AI system
How to read this: arrows labelled E1 to E6 are the entry points that cross a trust boundary, the other arrows are normal traffic inside your zone, and the flowing dashed arrow is asynchronous.
Each entry point is where text you did not write reaches something that acts on it.
The model cannot reliably tell instructions from data: treat its output as a request, not a command.
Limit consequences structurally: scoped credentials, no unneeded tools, egress rules, approval for irreversible actions.
Classifiers and prompt rules lower the odds. No source cited here claims a defence that fully solves prompt injection.
Traces are a data store: redact before storage, set retention, restrict reads.
Report what a security test can claim, not that the risk is gone.
The real-world problem
Composite scenario, not a real company. An assistant summarises each employee's inbox and can send replies on their behalf. An outside sender writes a message that contains text addressed to the assistant instead of the reader. The assistant processes it like every other message. The design question is not whether that message can be detected, but what the assistant can do if it is not.
After this page you will be able to
Draw the trust boundaries of an AI system, number its entry points and name the control at each.
Name the ten items of the OWASP Top 10 for LLM Applications (2026) and place them on the design.
Explain, from the sources, why prompt injection works and why no single filter is enough.
Choose among scoped credentials, approval gates, egress allow-lists, privilege separation and redaction for a given attack chain.
State what a red-team result can and cannot claim.
Prerequisites and cross-links
Diagramming AI systems: the trust-boundary view (its Figure 10) is the base of Figure 1 here.
How to read this: each row is one thing to protect, who might go after it, which crossing they use and what a failure costs.
The OWASP Threat Modeling Cheat Sheet asks four questions: "What are we working on?", "What can go wrong?", "What are we going to do about it?" and "Did we do a good enough job?" Figure 1 and this table answer the first two.
The OWASP Top 10 for LLM Applications (2026)
This page uses the 2026 edition (Version 2026, dated 3 August 2026 on OWASP's resource page). OWASP's list page still showed the 2025 list on 2026-10-22, so the 2025 ID stays as a second column. The first three columns are OWASP's. The last two are this course's mapping.
How to read this: find the entry point in Figure 1, read across to the risk, and follow the last column for the control.
Check yourself: a third-party MCP server's tool description is entry point E3. Which two items in the table apply?
LLM01:2026 Prompt Injection, because the description is text the model reads. LLM04:2026 Supply Chain, because a third-party tool package is a supply-chain component (OWASP files MCP servers and tool registries under ASI04 in its Agentic Top 10).
Direct and indirect injection
Direct
Indirect
Who writes the text
The primary user
A third party who controls a resource the system reads
Who is harmed
Mostly the operator
Often the user, not the attacker
Entry point
E1
E2, E3
Illustrative, non-functional example
A user asks the assistant to disregard its rules
An email contains text addressed to the assistant
First controls
Policy engine, scoped tools, abuse monitoring
Mark content as data, no tools in the reading step, approval for actions
How to read this: the columns differ in who holds the pen and who pays, not in how the model processes the text.
NIST AI 100-2 E2025 (March 2025) says a direct prompting attack arises when the attacker is the primary user, and that indirect injection is enabled by "resource control": the attacker modifies resources the model ingests at runtime.
Check yourself: an outsider emails a customer, and the customer's assistant acts on the email. Direct or indirect, and who is harmed?
Indirect. The attacker is a third party who controls a resource (the email), not the user. The customer, who is the primary user, is the one harmed.
The confused deputy
Element
Hardy's compiler (1988)
An AI agent
The deputy
A compiler licensed to write in its own directory
The Orchestrator holding a service or user credential
Who supplies the input
The invoker, who named the output file
A model whose context includes outside text
The confusion
The compiler's own authority served the invoker's request
A tool call runs with real authority for a request the text chose
The fix
A capability that names the file and authorises the write
A scoped credential per request, no authority in model-written values
How to read this: the left column is Norm Hardy's account of a billing-file incident at a timesharing company, and the right column is the same shape in an agent.
The agent is the same deputy with a harder problem: the party supplying the input is a model that outside text can steer.
Exfiltration paths
Figure 2. Three ways data leaves, and where each is closed
How to read this: P1, P2 and P3 are the three paths, the red box is outside your control, and the green boxes on the way are where a control goes.
Path
How data leaves
Control
P1 image or link
A client fetches a URL that carries data (NIST names markdown image rendering)
A send or HTTP tool whose destination the model chooses (NIST: forwarding email to an attacker's inbox)
Egress allow-list, fixed recipients, approval for new destinations
P3 hidden in text
The design-patterns paper describes encoding data in a reply with non-printing characters
Keep the data out of that step, normalise output, show reviewers the exact content
How to read this: each row is one path from Figure 2, the mechanism a source documents and the control that closes it.
Check yourself: the reading step has no send tool and no network access. Which of P1, P2 and P3 remain?
P1 and P3. Both travel in the answer text to the Client app, so they need output controls. P2 needs a tool or a network path, which that step does not have.
Excessive agency
Root cause (OWASP LLM03:2026)
What it looks like
OWASP prevention items
Excessive functionality
A tool the task never needs, left over from a trial
Minimize tools and their functionality, avoid open-ended tools
Excessive permissions
A tool that can delete when the task only reads
Minimize tool permissions, execute tools in the user's context
Excessive autonomy
A high-impact action with no independent check
Require user approval, complete mediation
How to read this: the left column is the cause OWASP names, the right column the prevention items it lists.
OWASP defines the risk as damaging actions in response to unexpected, ambiguous or manipulated model output, "regardless of what is causing the LLM to malfunction". Two further options "will not prevent Excessive Agency but can limit the level of damage caused": monitoring tool use and rate limiting. OWASP says the entry "climbed to third" in 2026.
How to read this: follow the arrows for the order of a request, and the two shaded zones group layers by what they can promise.
What the fetched sources say about the limit, as of 2026-10-22:
Design-patterns paper: detection filters "raise the bar for attackers" but "remain fundamentally heuristic and cannot guarantee prevention of all attacks".
NIST: current mitigations do not offer full protection, so designers may assume injection is possible when a model sees untrusted input.
OWASP LLM01:2026: "no reliable prevention mechanism exists today" (the 2025 text said it is unclear whether foolproof prevention exists).
CaMeL authors, asked whether prompt injection is solved: "No".
Spotlighting (Hines et al., arXiv 2403.14720): attack success cut from over 50% to below 2% in the authors' GPT-family experiments, a measured gain and not a guarantee.
Pick the control that stops each case.
Which control stops this? (interactive when JavaScript is on)
A retrieved web page makes the model write an image link whose address carries chat data, and the client fetches it on render.
A stronger system prompt
Block or rewrite remote images and links where output is rendered
A smaller context window
Log redaction
Answer
Block or rewrite remote images and links where output is rendered. The leak happens when the client fetches the URL, so the control belongs where output is rendered.
A ticket's text steers the model to pass another tenant's id as a tool argument.
Ask the model to double-check the id
Remove the tenant parameter and use a credential scoped to the session tenant
Add an input classifier
Shorten log retention
Answer
Remove the tenant parameter and use a credential scoped to the session tenant. Authority must not live in a value the model writes. A scoped credential makes the wrong id unreachable even if the model is steered.
A document tells an agent with an HTTP tool to post internal data to an unknown host.
An egress allow-list on the tool's network path
Spotlighting the document
Redacting the logs
Lowering the temperature
Answer
An egress allow-list on the tool's network path. An allow-list limits where data can go whatever the model decides. Spotlighting lowers the odds of steering but guarantees nothing.
Sandboxing, logs and supply chain
Area
Risk
Control
Code execution
Generated code touches files and networks it should not
A sandbox with only the connections it needs (design-patterns paper), a short-lived credential, only the tools the step needs
Traces and logs
Prompts, answers and tool arguments hold personal data and secrets
Redact before storage; OWASP's Logging Cheat Sheet says tokens, passwords, keys and sensitive personal data should usually not be logged directly
Models and dependencies
OWASP LLM04:2026: "the risks extend to third-party pre-trained models, datasets, and model artifacts"
Supplier due diligence, provenance checks with signing and hashes, an SBOM, a patching policy
How to read this: each row is an area outside the model that still needs a control, with the source's recommendation.
On MCP, the specification (revision 2026-07-28, the one its "latest" tools page links to as of 2026-10-22) says there should always be a human in the loop able to deny tool invocations, and that clients must consider tool annotations untrusted unless they come from trusted servers. Servers must validate inputs, apply access controls, rate limit and sanitise outputs.
Case files
EchoLeak, Indirect Injection in an AI Assistant. Problem: an arXiv case study describes CVE-2025-32711 as a zero-click prompt injection vulnerability in Microsoft 365 Copilot; per its abstract, one crafted email could enable remote data exfiltration. Pattern: defence across trust boundaries. Public case.
Email Summariser Injection, Trust Boundaries. Problem: a hostile message tries to steer an assistant that summarises inbound email. Pattern: trust boundaries, no tools in the summarising step. Composite.
PII in Traces and Logs, Redaction Design. Problem: prompts and answers with personal data land in traces. Pattern: log redaction with retention and access control. Composite.
Under the hood: Why prompt injection works
Intuition
A model reads one input made of your rules, the user's message, retrieved text and tool results. Roles such as system, user and tool result exist in the API, but how much weight the model gives each is learned behaviour, not an enforced boundary. Text in any slot can influence what comes out.
Mechanism
Greshake et al. (arXiv 2302.12173) argue that LLM-integrated applications "blur the line between data and instructions". NIST AI 100-2 E2025 says the same from the defender's side: because models combine the data and instruction channels, attackers can use the data channel to affect system operations.
Figure 4. Four sources, one context window
How to read this: four authors feed one input, and the only decision point for an action is after the output, in your code.
The Orchestrator assembles the prompt from four kinds of source.
The provider presents the model one input, with roles as signals the model was trained to weigh.
The model generates output, which text anywhere in the input can influence.
The Orchestrator reads the output and may run a tool request.
Segment
Tokens
Written by
Share
System prompt
400
The developer
6.7%
User message
60
The user
1.0%
5 retrieved passages (5 x 800)
4,000
Third parties
67.1%
Tool results
1,500
Other systems
25.2%
Total
5,960
100%
How to read this: illustrative sizes for one request. The developer wrote 400 of 5,960 tokens (6.7%), so 93.3% of the input came from someone else.
Assumptions: the token counts are invented for illustration. The share column is tokens divided by 5,960.
What this means for architects
What changes
What you do
A prompt cannot enforce the split between rules and data
Put authority in code outside the model: Policy engine, scoped credentials, approval
Every source added to the context is an entry point
Add it to Figure 1 with an owner and a control before it ships
Roles and delimiters help but are learned, not enforced
Use them (as of 2026-10-22, OpenAI's guidance keeps untrusted inputs out of developer messages and Anthropic's puts third-party content in tool results) and still assume they can fail
Larger contexts carry more third-party text
Budget what may enter the context: top-k, tool output size, attachments
Under the hood: Direct and indirect injection attack chains
Intuition
In a direct attack the user is the attacker. In an indirect attack someone plants text where the system will read it later, and the victim is whoever runs the assistant. Both chains end the same way: an action, or an output that leaves.
Mechanism
Greshake et al. list four ways to deliver an injection.
Delivery
What it means in the paper
Where it enters Figure 1
Passive
Text placed in public sources the system retrieves, such as a web page
E2
Active
Text sent to the system, for example by email
E2, E3
User-driven
The user is tricked into pasting the text
E1
Hidden
Multi-stage payloads, images, encoded text
Any of the above
How to read this: each row is a delivery method from the paper and the entry point it uses, which tells you where a control must sit.
Figure 5. Direct attempt, stopped at the Policy engine
sequenceDiagram
accTitle: Direct injection attempt stopped by the Policy engine
accDescr: A user sends a message that asks the assistant to ignore its rules and use a tool outside the user's rights. The model replies with a tool request. The Orchestrator treats it as a request and asks the Policy engine. The Policy engine denies it, the Orchestrator returns a refusal and publishes a security event.
autonumber
participant U as User
participant C as Client app
participant O as Orchestrator
participant L as LLM provider
participant P as Policy engine
participant E as Event bus
U->>C: message asking the assistant to ignore its rules
C->>O: message plus verified identity
O->>L: prompt
L-->>O: tool request outside the user's rights
Note over O: The model output is a request, not a command
O->>P: may this user call this tool with these arguments?
P-->>O: deny
O-)E: security event with user and tool
O-->>C: refusal
How to read this: time runs downward, the note marks the decision, and the deny at step 6 ends the chain.
Figure 6. Indirect chain, with the defended path shown
sequenceDiagram
accTitle: Indirect injection chain with defences
accDescr: An outside sender puts a message in the mailbox. The user asks for a summary. The Orchestrator reads the mailbox in a step that has no send tool and labels the bodies as external data. The model returns a summary and possibly a send request. If a send is requested the Policy engine denies it and a security event is published. The Orchestrator strips links and images and returns plain text, and the Client app does not fetch remote images.
autonumber
participant X as Outside sender
participant B as Mailbox
participant U as User
participant O as Orchestrator
participant L as LLM provider
participant P as Policy engine
participant E as Event bus
participant C as Client app
X->>B: email with text addressed to the assistant
U->>O: summarise my mail
O->>B: read messages in a step with no send tool
B-->>O: message bodies
Note over O: Bodies are labelled as external data
O->>L: prompt with labelled data
L-->>O: summary and possibly a send request
alt a send is requested
O->>P: may the summarising step send?
P-->>O: deny
O-)E: security event, suspected injection
end
Note over O: Strip links and images from the summary
O-->>C: plain-text summary
Note over C: Remote images are not fetched
How to read this: the alt block is the injected path, and the three notes are the controls that do not depend on detecting the email.
Step through the same chain with and without defences.
An indirect-injection chain, with and without defences (interactive when JavaScript is on)
No defences
Text is planted. An outside sender emails text addressed to the assistant, not to the human reader.
A normal request. The user asks the assistant to summarise today's mail. The request is legitimate.
Text enters the context. The mail tool returns the message. Its text lands next to the system prompt with no enforced boundary.
The model follows it. The model emits a tool request, such as a send, that the user never asked for.
The tool runs with real authority. The Orchestrator runs it with the user's mailbox credential. The Tool server cannot tell who really asked.
Data leaves. A message goes to an outside address, or a link in the reply is fetched. Nobody approved it.
With defences
Text is planted. The same email arrives. A filter may flag it, but the design does not depend on that.
A normal request. The same request. The summarising step has a read tool and no send tool.
Content is labelled as data. The body is returned as external data with its source stated, so it is marked untrusted.
The model may still be steered. At worst it writes a misleading summary or asks for a send, which cannot run.
Structural controls act. The Policy engine denies the send. Output checks strip links and images.
The attempt is visible. A security event is logged. Defences limit consequences, they do not make the model immune.
Check yourself: in Figure 6, which controls still work if the model is fully convinced by the email?
The Policy engine deny, the missing send tool, link and image stripping and the refusal to fetch remote images. None depends on the model recognising the email.
What this means for architects
What changes
What you do
Direct injection harms the operator, indirect often the user (NIST)
Abuse monitoring for E1, an isolated reading step for E2 and E3
Delivery can be passive, active, user-driven or hidden
Inventory every channel in Figure 1, including pasted text
Multi-stage or encoded payloads are harder for a first-stage filter to catch (Greshake et al.)
Never make a filter the only control
The chain needs every stage
Place controls at several stages, so one miss is not a breach
Under the hood: The confused deputy and tool abuse
Intuition
Hardy's compiler had authority from two sources and could not say which one a request should use. An agent holds a credential and acts on text it cannot attribute. The mistake is the same: authority used for the wrong principal.
Mechanism
Norm Hardy's paper "The Confused Deputy (or why capabilities might have been invented)" describes a compiler at a timesharing company. The compiler could write files in its own directory, which also held the billing file. A user named the billing file as the compiler's debugging output file.
Figure 7. Hardy's compiler, step by step
sequenceDiagram
accTitle: The confused deputy in Hardy's compiler example
accDescr: A user runs the compiler and names the billing file as the output file. The compiler asks the operating system to open it for writing. The operating system allows the write because the compiler's licence covers its home directory. The debugging output overwrites the billing data.
autonumber
participant U as User
participant K as Compiler (the deputy)
participant S as Operating system
participant F as Billing file
U->>K: run the compiler with the billing file as the output file
Note over K: The compiler holds a licence for its own directory
K->>S: open the billing file for writing
Note over S: Allowed, because the compiler's licence covers that directory
S->>F: debugging output is written
Note over F: The billing data is overwritten
How to read this: the compiler is correct in isolation, and the failure is that its own authority was used for the invoker's request.
The deputy holds authority of its own (the directory licence).
A caller supplies an input that names a resource (the output file).
The system checks the deputy's authority, not the caller's intent, and allows the action.
Hardy's fix is a capability: "The capability both identifies the file and authorizes the compiler to write there."
Design
Who chooses the tenant for a tool call
Tenants the model can reach
Tenant id is a tool parameter
The model
400
Tenant comes from the session, with a scoped credential
The gateway and the Policy engine
1
How to read this: the same tool, two designs, with 400 tenants as in the helpdesk composite case.
Assumptions: 400 tenants, as in the helpdesk case file. In the first design a steered model can name any of them. In the second it has no parameter to name one.
What this means for architects
What changes
What you do
Authority in a model-written argument is attacker-influenced
Remove the parameter; take tenant and user from the verified session
One shared service credential is ambient authority
Issue a short-lived credential per request, scoped to one tenant and one tool
Tools that act outward widen the blast radius
Add an egress allow-list and approval for new destinations
MCP clients relay model-chosen inputs
Show tool inputs before the call, as the specification asks, and log tool use
Under the hood: Guardrail classifier pipelines
Intuition
A guardrail is a small check in front of or behind the model. It can be rules, or a classifier model, that returns allow, block or uncertain. It is cheap relative to the main model, and it is wrong some of the time.
Mechanism
Llama Guard, described in an arXiv paper (2312.06674), is one published example: an instruction-tuned model that uses a safety-risk taxonomy to classify both user prompts and model responses. Anthropic's documentation (as of 2026-10-22) describes a similar pattern for tool output: pass the raw output to a small classifier call, use structured output so the verdict is a parseable boolean, and treat a refusal from the screening model as a harmful verdict.
Figure 8. A checkpoint with a fail-closed path
flowchart TD
accTitle: A guardrail checkpoint with verdict mapping and a fail-closed path
accDescr: Text to check goes through cheap rules and then a classifier model. A verdict maps to allow when clear, block when flagged, or escalate when uncertain. If the classifier times out or refuses, the checkpoint fails closed and blocks. Blocked and escalated cases are logged as security events.
In["Text to check: input, tool result or draft answer"]
Rules["Rules: length, format, known patterns"]
Cls["Classifier model"]
D{"Verdict"}
Allow["Allow"]
Block["Block and tell the user"]
Esc["Escalate to human review"]
Fail["Classifier timeout or refusal"]
Log["Security event in the Trace store"]
In --> Rules --> Cls --> D
D -->|"clear"| Allow
D -->|"flagged"| Block
D -->|"uncertain"| Esc
Cls -->|"no verdict"| Fail --> Block
Block --> Log
Esc --> Log
How to read this: every path ends in allow, block or escalate, and the "no verdict" path ends in block, which is what fail closed means.
Text reaches a checkpoint: user input (E1), a tool result (E3) or the draft answer (E4).
Cheap rules run first, then a classifier returns a verdict against a taxonomy.
The verdict maps to allow, block or escalate. No verdict means block.
Blocked and escalated cases are logged.
Quantity
Value
How it is derived
Requests per day
100,000
Assumption
Hostile share
0.01%
Assumption
Hostile requests
10
100,000 x 0.0001
Detection rate
90%
Assumption
Caught and missed
9 and 1
10 x 0.9, then 10 - 9
False-positive rate
1%
Assumption
Legitimate requests blocked
about 1,000
99,990 x 0.01 = 999.9
Share of flagged requests that are real attacks
0.9%
9 / (9 + 999.9)
How to read this: with rare attacks, a 1% false-positive rate means about a thousand legitimate requests are blocked for each nine attacks caught.
Assumptions: all rates are illustrative. The point is the base-rate effect, not the values.
What this means for architects
What changes
What you do
A classifier is a layer, and the design-patterns paper calls such filters fundamentally heuristic
Never make it the only control on an action; keep the structural layers
Rare attacks make false positives the main cost
Set thresholds with the cost of a wrongly blocked user in mind; route "uncertain" to review
It sits on the request path
Decide per checkpoint: synchronous before a tool call, asynchronous for monitoring
Greshake et al. call it unclear whether filters survive obfuscation or encoding
Measure the classifier on your own traffic and attack set, define fail-closed behaviour and test it
Under the hood: Sandboxing and privilege separation
Intuition
Do not let the model that reads untrusted text hold the tools. Split the work so the part that can act never sees the part that can be steered.
Mechanism
"Design Patterns for Securing LLM Agents against Prompt Injections" (Beurer-Kellner et al., arXiv 2506.08837, v3, June 2025) states a guiding principle: once an agent has ingested untrusted input, it must be constrained so that input cannot trigger consequential actions. The dual LLM pattern, proposed by Simon Willison and described in the paper, puts that into practice.
Figure 9. Dual LLM with an orchestrator that is ordinary code
How to read this: the privileged LLM only handles variable names, and the values are filled in by ordinary code at call time.
The privileged LLM sees only trusted input and plans tool calls using variable names.
The orchestrator reads the untrusted data and asks the quarantined LLM to extract what is needed.
The result is constrained (a boolean, a category) and stored in a variable.
The orchestrator substitutes the value into the tool call, which the privileged LLM never reads.
Pattern (Beurer-Kellner et al.)
What it constrains
Limit the authors state
Action-selector
The model picks from predefined actions, with no feedback to the agent
Work moves into the predefined actions; flexibility is lost
Plan-then-execute
The action list is fixed before untrusted data is read
Arguments can still be altered
LLM map-reduce
One isolated model per document, constrained output
A document can still mislead its own map step
Dual LLM
A privileged model with tools, a quarantined one without
The quarantined model can still be injected
Code-then-execute
The model writes a program that calls tools and models
Injected data can still alter content produced
Context minimisation
The user prompt is removed from context after it informs the action
Targets injection in the user prompt only
How to read this: each pattern removes one capability from the model, and the last column is what the paper says remains.
CaMeL (Debenedetti et al., arXiv 2503.18813, v2, June 2025) goes further: a privileged model writes a program, a custom interpreter runs it, and every value carries capabilities that policies check at each tool call. On AgentDojo the authors report "solving 77% of tasks with provable security (compared to 84% with an undefended system)".
From these rounded figures that is 84 - 77 = 7 points, or 7 / 84 = 8.3% of what the undefended agent solved.
The drop depends on the model: the paper's Table 2 lists 3.1 to 32.0 points across six model settings.
Limits the CaMeL authors state:
No defence against text-to-text attacks with no data-flow consequence, such as a summary that differs from the email.
Users must write and maintain policies, confirmations can cause user fatigue, and side channels are possible.
Asked whether prompt injection is solved: "No, prompt injection attacks are not fully solved."
Check yourself: in the dual LLM design, what can a steered quarantined LLM still do?
It can return a wrong value or tampered text. It cannot call a tool or change the plan. The paper still lists tampered output, such as a changed recipient, as a remaining risk.
What this means for architects
What changes
What you do
The task needs a model to read untrusted text and act freely
The paper says reliable guarantees are unlikely: narrow the task or add approvals
The action sequence is known before the data is read
Plan-then-execute, with allow-listed arguments
Many documents are handled independently
Use map-reduce with constrained outputs such as a boolean or a category
Third-party tools lack capability support
CaMeL's authors note utility degrades
The agent runs code
Sandbox it with only the tools and network it needs
Under the hood: Redacting sensitive data from traces and logs
Intuition
A trace is a second copy of every conversation. Whatever you store unredacted lives as long as the retention period and is readable by everyone with access to the store.
Mechanism
The Presidio documentation describes predefined or custom recognisers that use named entity recognition, regular expressions, rule-based logic and checksums. That gives two families: pattern-based (regular expressions, checksums) and entity-based (named entity recognition). The same page warns that there is no guarantee Presidio will find all sensitive information and that additional systems and protections should be employed.
Figure 10. Redact before the span leaves the application
sequenceDiagram
accTitle: Redacting a span before it is stored
accDescr: The Orchestrator sends a span containing the prompt, answer and tool arguments to the redactor. If the redactor is available it replaces sensitive values with typed placeholders and the Orchestrator publishes the redacted span to the Event bus. A Worker consumes it and writes it to the Trace store with a retention tag. If the redactor is unavailable, only metadata such as ids, timings and token counts is published.
autonumber
participant O as Orchestrator
participant R as Redactor
participant E as Event bus
participant W as Worker
participant S as Trace store
O->>R: span with prompt, answer and tool arguments
Note over R: Pattern rules first, then entity detection
alt redactor available
R-->>O: span with typed placeholders
O-)E: event with the redacted span
else redactor unavailable
Note over O: Keep metadata only: ids, timings, token counts
O-)E: event with metadata only
end
E-)W: consume event
W->>S: write with a retention tag
How to read this: redaction happens before the bus, and when it fails the system stores less, not more.
Redact in the application, before the event reaches the Event bus or any store.
Run pattern rules, then entity detection, and replace values with typed placeholders so traces stay debuggable.
If the redactor fails, store metadata only.
Tag each record with a retention class, restrict reads and test with seeded synthetic data.
Redactor recall
Prompts with personal data per day
Values reaching the store per day
98%
2,000
40
99.9%
2,000
2
How to read this: a redactor that misses 2% of values still lets 40 prompts a day into the store, so retention and access control must assume some leakage.
Assumptions: 2,000 prompts a day contain personal data, one value per prompt, and the recall figures are illustrative. Leaked values per day = 2,000 x (1 - recall).
What this means for architects
What changes
What you do
No redactor is complete (Presidio's own documentation says so)
Treat the store as sensitive anyway: retention limits, restricted reads, logged access
Redaction after storage leaves a window of exposure
Redact before the Event bus; fail to metadata only
OWASP says tokens, passwords, keys and sensitive personal data should usually not be logged directly
Exclude headers and secrets by rule, not by detection
Debugging needs some content
Sample content with a short retention and explicit access, not by default
Audit duties pull the other way
The next page covers audit logging and its tension with privacy
Decision framework
Figure 11. Choosing the control for a new feature or tool
flowchart TD
accTitle: Choosing security controls for a new feature
accDescr: The first question is whether any context text comes from someone you do not control. If not, apply standard application security and scoped credentials. If yes, ask whether the output can trigger an action or leave the system. If not, apply output handling. If it can, ask whether the action is irreversible or external. If so, require approval, a scoped credential and an egress allow-list. If not, ask whether the action sequence can be fixed before the data is read. If so, use plan-then-execute with constrained arguments, and if not, narrow the task or use privilege separation.
Start(["New AI feature or tool"])
Q1{"Does any context text come from someone you do not control?"}
Q2{"Can the output trigger an action or leave the system?"}
Q3{"Is the action irreversible or external?"}
Q4{"Can the action sequence be fixed before the data is read?"}
A0["Standard application security and scoped credentials"]
A1["Output handling: encode, no remote fetch"]
A2["Approval, scoped credential, egress allow-list"]
A3["Plan-then-execute with constrained arguments"]
A4["Narrow the task or use privilege separation"]
Start --> Q1
Q1 -->|"no"| A0
Q1 -->|"yes"| Q2
Q2 -->|"no"| A1
Q2 -->|"yes"| Q3
Q3 -->|"yes"| A2
Q3 -->|"no"| Q4
Q4 -->|"yes"| A3
Q4 -->|"no"| A4
How to read this: start at the top, answer yes or no, and stop at the first rectangle. Real designs often end at several.
If
Then
Because
Revisit when
Any untrusted text reaches the context
Draw it as an entry point with an owner and a control
Each source is a way in
A source is added
The model's output can trigger a tool
Authority comes from the session and a scoped credential, never from a model-written value
Prompts cannot enforce the split
A tool schema changes
An action is irreversible or external
Require approval, show the exact content, restrict destinations
OWASP (LLM03:2026) recommends human approval for high-impact actions, and the design-patterns paper warns about fatigue
Approval load grows
A classifier is proposed as the fix
Keep it as one layer next to structural controls
Sources call such defences heuristic
A source shows a guarantee for your setting
Comparison of the main options
Option
What it constrains
Strength
Limit
Prompt rules, spotlighting
What the model weighs
Cheap; measured gains
Not a guarantee
Input and output classifiers
What passes a checkpoint
Catches known patterns
Heuristic; false positives
Scoped credentials, least privilege
What an action can reach
Holds even if the model is steered
Needs per-request issuing
Egress allow-list
Where data can go
Closes P2
Does not stop P1 or P3
Human approval
Irreversible actions
Independent check
Fatigue
Dual LLM, plan-then-execute
Which component can act
Strong against hijacked actions
Lower utility; arguments can change
Redaction, retention, access control
What the store holds
Shrinks the data at risk
Detectors miss values
Failure scenarios to design for
Failure
What the system does
Why
Injected text in a retrieved document asks for a send
The Policy engine denies it, the step has no send tool, a security event is logged
Structural control holds when steering works
The classifier is down or refuses
Tool calls are blocked, answer-only paths continue with remote images off
Fail closed where actions exist
The redactor misses a value
Retention and read limits cap exposure, a weekly seeded test flags the miss
No redactor is complete
A third-party MCP server changes a tool description
The approved description stays pinned until reviewed
Descriptions are text the model reads
Approvers approve everything
Approvals cover only external side effects, and a sample is reviewed
Both papers warn about tired approvers
The Policy engine is down
Calls are refused and the user is told
An unavailable control must not mean allow
Worked example
Assumptions:
An agent proposes 2,000 actions a day.
15% are external or irreversible and need approval.
A reviewer needs 20 seconds per approval.
The red-team suite has 300 adversarial prompts and all are blocked. The interval assumes the prompts are independent draws from one attack distribution.
Quantity
Value
Calculation
Approvals if only external or irreversible actions need one
300 a day
2,000 x 0.15
Reviewer time for those approvals
6,000 s, 1.7 h
300 x 20 s
Reviewer time if every action needs approval
40,000 s, 11.1 h
2,000 x 20 s
How to read this: approving everything costs 6.7 times as much reviewer time (40,000 s against 6,000 s), and that is where fatigue begins.
Set the widget to 297 passes: how far does the lower bound fall?
What a clean red-team run can claim (interactive when JavaScript is on)
Observed pass rate
100.0%
Interval (z = 1.645)
99.1% to 100.0%
Gate: lower bound at least 99.0%
pass
With 300 of 300 blocked, the one-sided 95% Wilson lower bound on the block rate is 99.1%. The claim is: on this suite, the block rate is at least 99.1% with that confidence. It says nothing about attacks outside the suite. Statistics for gates are in Evaluating AI systems.
A tool dispatcher that the model cannot talk past
The sketch runs a tool call the model requested. Tenant, user and credential come from the session, tool schemas may not accept identity fields, URLs must be on an allow-list and listed tools need a recorded human approval.
from dataclasses import dataclass
from urllib.parse import urlparse
ALLOWED_HOSTS = {"tickets.example.com", "mail.example.com"}
NEEDS_APPROVAL = {"send_email", "delete_record"}
IDENTITY_FIELDS = {"tenant", "user", "tenant_id", "user_id", "credential"}
@dataclass(frozen=True)classSession:
user_id: str
tenant_id: str
approved: frozenset = frozenset() # call ids that a human approveddefdispatch(session, call_id, tool, args, registry):
"""Run a tool call requested by the model. The model may name a tool and
its arguments, never the tenant, the user or the credential."""
spec = registry.get(tool)
if spec isNone:
return {"status": "denied", "reason": "unknown tool"}
if IDENTITY_FIELDS & set(spec["params"]):
raise ValueError(f"{tool}: schema must not accept identity fields")
ifset(args) - set(spec["params"]):
return {"status": "denied", "reason": "unexpected argument"}
if"url"in args and urlparse(args["url"]).hostname notin ALLOWED_HOSTS:
return {"status": "denied", "reason": "host not on allow-list"}
if tool in NEEDS_APPROVAL and call_id notin session.approved:
return {"status": "needs_approval", "call_id": call_id}
# identity comes from the session, so a tool argument cannot widen accessreturn spec["run"](tenant=session.tenant_id, user=session.user_id, **args)
It is a sketch: a real deployment adds credential issuing, audit events and a Policy engine call, as in Figure 5.
Common mistakes
Trusting the system prompt to hold the line. Put authority in code outside the model.
Letting the model choose identity. Take tenant and user from the session.
Making a classifier the only control. Keep scoped credentials and approvals behind it.
Approving everything. Reserve approvals for external or irreversible actions.
Redacting after storage. Redact before the event leaves the application.
Reading a clean test as proof. State what the suite covers and no more.
Try it
Time: 30 to 40 minutes. Take one AI feature you know, or invent a small one.
Copy Figure 1 and relabel it with your components, keeping E1 to E6.
For each entry point, write the control and mark it heuristic or structural.
For each tool, decide the credential scope, approval and reachable hosts.
Write ten illustrative, non-functional adversarial test descriptions.
Write one sentence on what a clean run would and would not prove.
Done when every entry point has an owner and a control, every action tool has a scope and an approval decision, and you can state in one sentence what your test result can claim.
Key takeaways
Number every entry point where text you did not write reaches something that acts.
The model cannot reliably separate instructions from data: treat its output as a request that code decides on.
Authority never lives in a model-written value. Use scoped, per-request credentials from the session.
Heuristic layers lower the odds; no cited source claims a full defence.
Structural controls hold when steering succeeds: no tool in the reading step, a Policy engine, an egress allow-list.
Privilege-separation patterns trade utility for guarantees, and their authors list what remains.
Redact before storage and assume the redactor misses something.
Self-check
1. Why can a system prompt alone not separate the developer's rules from retrieved data?
Both reach the model as one input. Roles are signals the model learned to weigh, not an enforced boundary, and NIST notes that models combine the data and instruction channels. Enforce the split in code outside the model.
2. What do the CaMeL authors report for AgentDojo, and what do they say about whether prompt injection is solved?
77% of tasks solved with provable security, against 84% undefended. They say prompt injection is not fully solved and list user-written policies, user fatigue, side channels and text-to-text attacks as limits.
3. Name the three root causes of Excessive Agency in OWASP LLM03:2026.
Excessive functionality, excessive permissions and excessive autonomy.
4. Why redact before the Event bus rather than after the Trace store?
After storage there is a window in which unredacted data exists and may be copied. Redacting before the bus means the value never reaches a store, and a failed redactor falls back to metadata.
5. A red-team suite of 300 attacks has zero successes. What can you claim?
The one-sided 95% Wilson lower bound on the block rate for that suite is 99.1%. Nothing follows about attacks outside the suite, and the risk is not solved.
Lesson 4 of 7 · AI Architect (Advanced): Operate, Secure and Govern
Threat-model an AI system: trust boundaries, the OWASP LLM Top 10 (2026), prompt injection chains, confused deputy, guardrails, sandboxing and log redaction.
In the rest of this lesson
Case files
Under the hood: Why prompt injection works
Under the hood: Direct and indirect injection attack chains
Under the hood: The confused deputy and tool abuse
Under the hood: Guardrail classifier pipelines
Under the hood: Sandboxing and privilege separation
Enrolling is free. It unlocks every lesson in AI Architect (Advanced): Operate, Secure and Govern, tracks your progress across devices and keeps your quiz scores. The newsletter is optional.
Real architecture decisions, LLMOps patterns that survive production, and engineering leadership advice — from 12+ years of building at enterprise scale. Free. No spam. Unsubscribe anytime.
Get new articles by email
Free. We email a confirmation link first, and you can unsubscribe from any email.