Security and Threat Modeling for AI Systems

Sandboxing, logs and supply chain

How to read this: each row is an area outside the model that still needs a control, with the source's recommendation.

On MCP, the specification (revision 2026-07-28, the one its "latest" tools page links to as of 2026-10-22) says there should always be a human in the loop able to deny tool invocations, and that clients must consider tool annotations untrusted unless they come from trusted servers. Servers must validate inputs, apply access controls, rate limit and sanitise outputs.

Case files

• EchoLeak, Indirect Injection in an AI Assistant. Problem: an arXiv case study describes CVE-2025-32711 as a zero-click prompt injection vulnerability in Microsoft 365 Copilot; per its abstract, one crafted email could enable remote data exfiltration. Pattern: defence across trust boundaries. Public case.
• Email Summariser Injection, Trust Boundaries. Problem: a hostile message tries to steer an assistant that summarises inbound email. Pattern: trust boundaries, no tools in the summarising step. Composite.
• Tool Abuse and Data Exfiltration, Privilege Split. Problem: a tool-using agent can be tricked into sending data out. Pattern: privilege separation with an egress allow-list. Composite.
• PII in Traces and Logs, Redaction Design. Problem: prompts and answers with personal data land in traces. Pattern: log redaction with retention and access control. Composite.

Under the hood: Why prompt injection works

Intuition

A model reads one input made of your rules, the user's message, retrieved text and tool results. Roles such as system, user and tool result exist in the API, but how much weight the model gives each is learned behaviour, not an enforced boundary. Text in any slot can influence what comes out.

Mechanism

Greshake et al. (arXiv 2302.12173) argue that LLM-integrated applications "blur the line between data and instructions". NIST AI 100-2 E2025 says the same from the defender's side: because models combine the data and instruction channels, attackers can use the data channel to affect system operations.

Figure 4. Four sources, one context window

How to read this: four authors feed one input, and the only decision point for an action is after the output, in your code.

1. The Orchestrator assembles the prompt from four kinds of source.
2. The provider presents the model one input, with roles as signals the model was trained to weigh.
3. The model generates output, which text anywhere in the input can influence.
4. The Orchestrator reads the output and may run a tool request.

How to read this: illustrative sizes for one request. The developer wrote 400 of 5,960 tokens (6.7%), so 93.3% of the input came from someone else.

Assumptions: the token counts are invented for illustration. The share column is tokens divided by 5,960.

What this means for architects

Under the hood: Direct and indirect injection attack chains

Intuition

In a direct attack the user is the attacker. In an indirect attack someone plants text where the system will read it later, and the victim is whoever runs the assistant. Both chains end the same way: an action, or an output that leaves.

Mechanism

Greshake et al. list four ways to deliver an injection.

How to read this: each row is a delivery method from the paper and the entry point it uses, which tells you where a control must sit.

Figure 5. Direct attempt, stopped at the Policy engine

How to read this: time runs downward, the note marks the decision, and the deny at step 6 ends the chain.

Figure 6. Indirect chain, with the defended path shown

How to read this: the alt block is the injected path, and the three notes are the controls that do not depend on detecting the email.

Step through the same chain with and without defences.

What this means for architects

Under the hood: The confused deputy and tool abuse

Intuition

Hardy's compiler had authority from two sources and could not say which one a request should use. An agent holds a credential and acts on text it cannot attribute. The mistake is the same: authority used for the wrong principal.

Mechanism

Norm Hardy's paper "The Confused Deputy (or why capabilities might have been invented)" describes a compiler at a timesharing company. The compiler could write files in its own directory, which also held the billing file. A user named the billing file as the compiler's debugging output file.

Figure 7. Hardy's compiler, step by step

How to read this: the compiler is correct in isolation, and the failure is that its own authority was used for the invoker's request.

1. The deputy holds authority of its own (the directory licence).
2. A caller supplies an input that names a resource (the output file).
3. The system checks the deputy's authority, not the caller's intent, and allows the action.
4. Hardy's fix is a capability: "The capability both identifies the file and authorizes the compiler to write there."

How to read this: the same tool, two designs, with 400 tenants as in the helpdesk composite case.

Assumptions: 400 tenants, as in the helpdesk case file. In the first design a steered model can name any of them. In the second it has no parameter to name one.

What this means for architects

Under the hood: Guardrail classifier pipelines

Intuition

A guardrail is a small check in front of or behind the model. It can be rules, or a classifier model, that returns allow, block or uncertain. It is cheap relative to the main model, and it is wrong some of the time.

Mechanism

Llama Guard, described in an arXiv paper (2312.06674), is one published example: an instruction-tuned model that uses a safety-risk taxonomy to classify both user prompts and model responses. Anthropic's documentation (as of 2026-10-22) describes a similar pattern for tool output: pass the raw output to a small classifier call, use structured output so the verdict is a parseable boolean, and treat a refusal from the screening model as a harmful verdict.

Figure 8. A checkpoint with a fail-closed path

How to read this: every path ends in allow, block or escalate, and the "no verdict" path ends in block, which is what fail closed means.

1. Text reaches a checkpoint: user input (E1), a tool result (E3) or the draft answer (E4).
2. Cheap rules run first, then a classifier returns a verdict against a taxonomy.
3. The verdict maps to allow, block or escalate. No verdict means block.
4. Blocked and escalated cases are logged.

How to read this: with rare attacks, a 1% false-positive rate means about a thousand legitimate requests are blocked for each nine attacks caught.

Assumptions: all rates are illustrative. The point is the base-rate effect, not the values.

What this means for architects

Under the hood: Sandboxing and privilege separation

Intuition

Do not let the model that reads untrusted text hold the tools. Split the work so the part that can act never sees the part that can be steered.

Mechanism

"Design Patterns for Securing LLM Agents against Prompt Injections" (Beurer-Kellner et al., arXiv 2506.08837, v3, June 2025) states a guiding principle: once an agent has ingested untrusted input, it must be constrained so that input cannot trigger consequential actions. The dual LLM pattern, proposed by Simon Willison and described in the paper, puts that into practice.

Figure 9. Dual LLM with an orchestrator that is ordinary code

How to read this: the privileged LLM only handles variable names, and the values are filled in by ordinary code at call time.

1. The privileged LLM sees only trusted input and plans tool calls using variable names.
2. The orchestrator reads the untrusted data and asks the quarantined LLM to extract what is needed.
3. The result is constrained (a boolean, a category) and stored in a variable.
4. The orchestrator substitutes the value into the tool call, which the privileged LLM never reads.

How to read this: each pattern removes one capability from the model, and the last column is what the paper says remains.

CaMeL (Debenedetti et al., arXiv 2503.18813, v2, June 2025) goes further: a privileged model writes a program, a custom interpreter runs it, and every value carries capabilities that policies check at each tool call. On AgentDojo the authors report "solving 77% of tasks with provable security (compared to 84% with an undefended system)".

• From these rounded figures that is 84 - 77 = 7 points, or 7 / 84 = 8.3% of what the undefended agent solved.
• The drop depends on the model: the paper's Table 2 lists 3.1 to 32.0 points across six model settings.

Limits the CaMeL authors state:

• No defence against text-to-text attacks with no data-flow consequence, such as a summary that differs from the email.
• Users must write and maintain policies, confirmations can cause user fatigue, and side channels are possible.
• Asked whether prompt injection is solved: "No, prompt injection attacks are not fully solved."

What this means for architects

Under the hood: Redacting sensitive data from traces and logs

Intuition

A trace is a second copy of every conversation. Whatever you store unredacted lives as long as the retention period and is readable by everyone with access to the store.

Mechanism

The Presidio documentation describes predefined or custom recognisers that use named entity recognition, regular expressions, rule-based logic and checksums. That gives two families: pattern-based (regular expressions, checksums) and entity-based (named entity recognition). The same page warns that there is no guarantee Presidio will find all sensitive information and that additional systems and protections should be employed.

Figure 10. Redact before the span leaves the application

How to read this: redaction happens before the bus, and when it fails the system stores less, not more.

1. Redact in the application, before the event reaches the Event bus or any store.
2. Run pattern rules, then entity detection, and replace values with typed placeholders so traces stay debuggable.
3. If the redactor fails, store metadata only.
4. Tag each record with a retention class, restrict reads and test with seeded synthetic data.

How to read this: a redactor that misses 2% of values still lets 40 prompts a day into the store, so retention and access control must assume some leakage.

Assumptions: 2,000 prompts a day contain personal data, one value per prompt, and the recall figures are illustrative. Leaked values per day = 2,000 x (1 - recall).

What this means for architects

Decision framework

Figure 11. Choosing the control for a new feature or tool

How to read this: start at the top, answer yes or no, and stop at the first rectangle. Real designs often end at several.

Comparison of the main options

Failure scenarios to design for

Worked example

Assumptions:

• An agent proposes 2,000 actions a day.
• 15% are external or irreversible and need approval.
• A reviewer needs 20 seconds per approval.
• The red-team suite has 300 adversarial prompts and all are blocked. The interval assumes the prompts are independent draws from one attack distribution.

How to read this: approving everything costs 6.7 times as much reviewer time (40,000 s against 6,000 s), and that is where fatigue begins.

Set the widget to 297 passes: how far does the lower bound fall?

With 300 of 300 blocked, the one-sided 95% Wilson lower bound on the block rate is 99.1%. The claim is: on this suite, the block rate is at least 99.1% with that confidence. It says nothing about attacks outside the suite. Statistics for gates are in Evaluating AI systems.

A tool dispatcher that the model cannot talk past

The sketch runs a tool call the model requested. Tenant, user and credential come from the session, tool schemas may not accept identity fields, URLs must be on an allow-list and listed tools need a recorded human approval.

from dataclasses import dataclass
from urllib.parse import urlparse

ALLOWED_HOSTS = {"tickets.example.com", "mail.example.com"}
NEEDS_APPROVAL = {"send_email", "delete_record"}
IDENTITY_FIELDS = {"tenant", "user", "tenant_id", "user_id", "credential"}


@dataclass(frozen=True)
class Session:
    user_id: str
    tenant_id: str
    approved: frozenset = frozenset()  # call ids that a human approved


def dispatch(session, call_id, tool, args, registry):
    """Run a tool call requested by the model. The model may name a tool and
    its arguments, never the tenant, the user or the credential."""
    spec = registry.get(tool)
    if spec is None:
        return {"status": "denied", "reason": "unknown tool"}
    if IDENTITY_FIELDS & set(spec["params"]):
        raise ValueError(f"{tool}: schema must not accept identity fields")
    if set(args) - set(spec["params"]):
        return {"status": "denied", "reason": "unexpected argument"}
    if "url" in args and urlparse(args["url"]).hostname not in ALLOWED_HOSTS:
        return {"status": "denied", "reason": "host not on allow-list"}
    if tool in NEEDS_APPROVAL and call_id not in session.approved:
        return {"status": "needs_approval", "call_id": call_id}
    # identity comes from the session, so a tool argument cannot widen access
    return spec["run"](tenant=session.tenant_id, user=session.user_id, **args)

It is a sketch: a real deployment adds credential issuing, audit events and a Policy engine call, as in Figure 5.

Common mistakes

• Trusting the system prompt to hold the line. Put authority in code outside the model.
• Letting the model choose identity. Take tenant and user from the session.
• Making a classifier the only control. Keep scoped credentials and approvals behind it.
• Approving everything. Reserve approvals for external or irreversible actions.
• Redacting after storage. Redact before the event leaves the application.
• Reading a clean test as proof. State what the suite covers and no more.

Try it

Time: 30 to 40 minutes. Take one AI feature you know, or invent a small one.

1. Copy Figure 1 and relabel it with your components, keeping E1 to E6.
2. For each entry point, write the control and mark it heuristic or structural.
3. For each tool, decide the credential scope, approval and reachable hosts.
4. Write ten illustrative, non-functional adversarial test descriptions.
5. Write one sentence on what a clean run would and would not prove.

Done when every entry point has an owner and a control, every action tool has a scope and an approval decision, and you can state in one sentence what your test result can claim.

Key takeaways

• Number every entry point where text you did not write reaches something that acts.
• The model cannot reliably separate instructions from data: treat its output as a request that code decides on.
• Authority never lives in a model-written value. Use scoped, per-request credentials from the session.
• Heuristic layers lower the odds; no cited source claims a full defence.
• Structural controls hold when steering succeeds: no tool in the reading step, a Policy engine, an egress allow-list.
• Privilege-separation patterns trade utility for guarantees, and their authors list what remains.
• Redact before storage and assume the redactor misses something.

Self-check

Further reading

• OWASP GenAI Security Project, OWASP Top 10 for LLM Applications 2025 and the 2026 edition.
• Greshake et al., Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection (arXiv 2302.12173).
• NIST, Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations (AI 100-2 E2025, March 2025).
• Beurer-Kellner et al., Design Patterns for Securing LLM Agents against Prompt Injections (arXiv 2506.08837).
• Debenedetti et al., Defeating Prompt Injections by Design (arXiv 2503.18813), and Norm Hardy, The Confused Deputy (ACM Operating Systems Review 22(4), 1988).

Last verified: 2026-10-22

Next

Page 13 covers governance and responsible AI: Governance and Responsible AI in Practice.