Composite scenario, not a real company. Assembled from common industry patterns; every number is illustrative and its assumptions are listed at the end.
This case belongs to Security and Threat Modeling for AI Systems. Defensive framing only: the injected text below is a placeholder, not a working payload.
Context
- A regional clinic network gives 300 staff an assistant that answers questions over referral letters and care plans.
- It has three tools: search documents, read a record, and send an email.
- It runs under one service account (a shared, non-human identity) that can read all 200,000 patient records. A staff member normally needs about 150.
- Some documents arrive from outside organisations, so their text is not written by the clinic.
Problem statement
| Defect found in review | Consequence |
|---|---|
| Over-broad authority: the tools act as the service account, not as the staff member | Text in a letter, such as [sentence addressed to the assistant], can cause reads the user could not make |
| Unrestricted outbound channel: the send tool takes any recipient | What the assistant reads can leave the network |
Goals (illustrative targets): a session can reach only the records its user may see, and no email goes outside the organisation without a person approving it.
Constraints
- Staff do need to email summaries to colleagues, and occasionally to an outside GP.
- Answers must stay interactive.
- Every read and send must be attributable to a named person.
- The model provider holds no credentials.
Why the traditional approach fell short
Role-based access control (RBAC) checks the user, but the tools called with the service account, so the check never saw the user. Data-loss rules on outbound email match known formats, such as an identifier pattern, and miss a paraphrased summary. Doing it by hand is safe and slow, which is why the assistant exists.
Options considered
| Option | Verdict |
|---|---|
| A. Remove the send tool: the assistant drafts, and staff send from their own mail client (the simpler option) | Accepted as the default wherever sending is occasional |
| B. Tell the model in the prompt never to send data outside | Rejected: it is a request, and the same context that carries the request carries the letter |
| C. Least privilege and egress control: user-scoped credentials, a recipient allow-list, approval for external sends, a network allow-list | Accepted where sending is needed |
Solution architecture
Figure 1. Where authority and egress are controlled
How to read this: thick arrows cross a trust boundary; the egress proxy is the last control, and it works even if every control above it fails.
Figure 2. Runtime path with a steered request
How to read this: the model's request is evaluated by code that knows the user, the recipient and the destination.
How AI solved it, step by step
The stepper follows one letter that contains steering text.
Under the hood
A confused deputy is a program that holds authority from more than one source and cannot keep them apart. Norm Hardy's 1988 account of a compiler at Tymshare says so: "The fundamental problem is that the compiler runs with authority stemming from two sources." It adds that the compiler "serves two masters and carries some authority from each" and "has no way to keep them apart" (Hardy, The Confused Deputy).
This page's mapping, not Hardy's: the Orchestrator holds the service account's authority, acts on a user's request, and its model output is shaped by third-party text. Hardy's remedy is to carry authority with the request, so that a credential both identifies a resource and authorises use of it. Here that is a user-scoped credential per call.
What went wrong / failure modes
| Failure | What the system does | Why |
|---|---|---|
| Staff cover for a colleague and need a record outside their scope | Break-glass: a second person approves, and the access is audited | Denying everything pushes staff back to shared logins |
| An "internal" mailbox auto-forwards outside | The allow-list names individual groups, and the relay never auto-forwards | A domain check alone is too coarse |
| A new tool gets its own internet route | Tool servers have no direct outbound route, and a network test checks it | Egress control fails if one path skips the proxy |
| A tool result tries to widen privileges | Results are data; only the Policy engine grants credentials | Authority never comes from model-visible text |
| Approval fatigue on external sends | Show recipient and body, and sample approved sends | Approvals work only if they are read |
Results
These are illustrative targets, not measurements.
| Target | Value |
|---|---|
| Records reachable per session | from 200,000 to about 150, roughly 1,333 times fewer |
| Cross-user reads in a 150-prompt adversarial set | 0 |
| External sends without approval | 0 |
| External-send approvals | 4,000 sessions x 12% send x 5% external = 24 a day |
| Outbound destinations from the tool tier | 1 (the internal mail relay) |
Tradeoffs and when NOT to use this approach
- User-scoped credentials need integration work in every data source.
- Approval adds delay to external sends; 24 a day is manageable, 2,400 would not be.
- This limits what a steered model can reach and send. It does not stop it writing a misleading summary.
- If sending is rare, take option A and remove the tool.
Design rules extracted
- Run the agent with the requesting user's authority, not a service account's.
- If a person can do a step as easily as the tool, remove the tool.
- Require approval for any action that crosses the organisation boundary.
- Enforce egress below the application, with a network allow-list.
- Treat denied calls as signals: alert, and pause the session.
Assumptions
- 300 staff, 200,000 records, about 150 records per staff member: 200,000 / 150 = 1,333.
- 4,000 sessions a day; 12% use the send tool (480 sends); 5% of those are external: 480 x 0.05 = 24 approvals.
- The 150-prompt test set and the threshold of three denied reads are chosen for illustration.
- The clinic network and all names are invented.
Last verified: 2026-10-22