Composite scenario, not a real company. Assembled from common industry patterns; every number is illustrative and its assumptions are listed at the end.
This case belongs to Security and Threat Modeling for AI Systems. For a sourced real-world example of the same class of failure, read EchoLeak. Defensive framing only: the examples below are placeholders, not working payloads.
Context
- A financial-services operations team receives about 6,000 emails a day in shared mailboxes: customer queries, supplier mail and messages from unknown senders.
- An assistant summarises each email for the agent and suggests a next step, such as looking up an account or drafting a reply.
- Version 1 puts the rules, the email text and three tools (account lookup, draft reply, send reply) into one prompt and runs with one shared service credential.
Problem statement
| Question from the security review | Finding |
|---|---|
| Who wrote the text in the prompt? | The firm wrote the rules; the sender wrote the email. The model sees both as one block of text |
| What if the email contains a sentence addressed to the assistant? | Example, non-functional: [sentence asking the assistant to reply to a third party]. Nothing in version 1 stops the model acting on it |
| What can the summarising step do? | Call any of three tools with the shared credential |
Goals (illustrative targets): zero tool calls caused by email content, and every outbound message approved by a person.
Constraints
- Anyone can send anything: plain text, HTML, long forwarded threads.
- Agents expect a summary within 5 seconds.
- Account lookups are personal data, so access must be attributable to an agent.
- The model provider receives no credentials.
Why the traditional approach fell short
Keyword routing sorts mail by topic but cannot summarise it. Spam and malware filters judge volume, attachments and reputation; a polite paragraph aimed at an assistant looks like ordinary business mail to them. Manual reading is safe and slow: 6,000 emails at 90 seconds each is 150 hours a day. The model removes the reading cost and, in version 1, also removes the separation between "text to read" and "instructions to follow".
Options considered
| Option | Verdict |
|---|---|
| A. No model: keyword routing, templates and people reading the mail (the simpler, non-AI option) | Kept as the fallback for sensitive mailboxes, and as the right answer if the summary is not worth the control work |
| B. Keep the design, add an injection classifier in front | Rejected as the only control: a message that passes it still reaches a step that has tools |
| C. Split the work: a summariser with no tools and a fixed output shape, actions proposed by rules and approved by a person | Accepted: a steered model has nothing to act with |
Solution architecture
Figure 1. Zones for the redesigned mail assistant
How to read this: only Zone C can act, and it is reached from Zone B through a person.
Figure 2. Runtime path in the fixed design
How to read this: the model's output is never an instruction to a tool; a rule table and a person sit between them.
How AI solved it, step by step
The stepper runs one hostile email through both designs.
Under the hood
A model has no enforced boundary between instructions and data: both are tokens in one context. Marking the email as untrusted asks the model to behave; it is not a barrier. So the design assumes step 2 can be steered and limits what a steered step can do.
Ask three questions of every step that reads untrusted text.
| Question | Version 1 | Fixed design |
|---|---|---|
| What can it do? | Call three tools | Nothing: no tools |
| What can it emit? | Free text and tool arguments | A fixed schema, shown as plain text |
| What does its output trigger? | Tool calls directly | A rule table, then a person |
The mechanism and the attack steps are in Direct and indirect injection attack chains.
What went wrong / failure modes
| Failure | What the system does | Why |
|---|---|---|
| The model is steered into a misleading summary | Shows the summary labelled as AI-written, with the original one click away; actions still need approval | Limiting actions does not make wording trustworthy |
| A steered category proposes the wrong action | The approval screen shows the full action; the Policy engine limits what any agent can run | A person and a policy both sit after the model |
| The output check quarantines a legitimate email | Shows the original email with a warning | A false quarantine costs a minute; a missed attack costs more |
| Approval fatigue | Groups similar actions and samples approved ones for review | Approvals only work if people read them |
| Someone adds a tool to the summariser | A configuration test fails the build if that step lists any tool | Capability creep is the usual regression |
Results
These are illustrative targets, not measurements.
| Target | Value |
|---|---|
| Tool calls caused by email content, in a 200-email adversarial test set | 0 (structural: the step has no tools) |
| Non-allow-listed URLs rendered to agents | 0 |
| False quarantines | at most 2% of 6,000 = 120 a day |
| Actions needing approval | 15% of 6,000 = 900 a day, about 7.5 hours of review at 30 seconds each |
| Output check, added latency | 20 ms |
Tradeoffs and when NOT to use this approach
- Review time is a real cost: 900 approvals a day is nearly a full-time role.
- The fix limits what a steered model can do. It does not stop a wrong summary.
- It does not address malicious attachments.
- If the team will not run an approval queue, use option A.
Design rules extracted
- Give a step that reads untrusted text no tools.
- Fix the shape of its output and render it as plain text.
- Let code, not model output, choose which action is proposed.
- Put a person and the Policy engine between a proposal and an action.
- Test that the untrusted-text step stays tool-less.
Assumptions
- 6,000 emails a day; reading cost 6,000 x 90 s = 540,000 s = 150 hours.
- 15% need an action review: 6,000 x 0.15 = 900; at 30 s each, 900 x 30 = 27,000 s = 7.5 hours.
- 2% false quarantine: 6,000 x 0.02 = 120.
- The 200-email test set, the 5 s summary budget, the 20 ms check, the 60 s credential and all names are invented for illustration.
Last verified: 2026-10-22