Composite scenario, not a real company. Assembled from common industry patterns; every number is illustrative and its assumptions are listed at the end.
This case belongs to Diagramming AI systems.
Context
A software vendor sells a multi-tenant B2B product. Each customer organisation is a tenant, and its administrators contact the vendor's support team about configuration, billing and open tickets. The vendor adds a support assistant inside the product. A tenant administrator asks a question in plain language, and the assistant can look up that tenant's tickets and account settings through a small set of tools. The data lives in one shared Document store, with a tenant identifier on every record. A single shared Tool server exposes the lookup tools to the Orchestrator.
The team brings its design to a review with a container diagram, no sequence diagrams and no trust boundaries drawn.
Problem statement
The review asks one question of the container diagram: where does the tenant identity come from at each step, and who checks it? The diagram cannot answer, so the reviewers ask for the runtime sequence of one tool call. Drawing it takes ten minutes and shows the problem.
Figure 1. Runtime sequence as first designed (flawed)
How to read this: time runs downward, and the two notes mark where tenant identity is decided and where it is not checked.
The tenant identifier is an argument the model writes. Anything in the model's context can influence it: the administrator's own message, the text of an earlier ticket, a pasted log. The Tool server trusts the argument and can read every tenant. Every component does what it was told; the missing item is a trust boundary between the Orchestrator and the Tool server, with a control on it.
The measurable goal that follows: a tool call must be able to read only the data of the tenant who is logged in, regardless of what the model writes, and the review must be able to point to the control that guarantees it.
Constraints
- One shared Tool server and one shared Document store, because separate stacks per tenant are out of budget.
- Support answers must stay interactive, so any added control has a small latency budget.
- Tenant administrators may paste arbitrary text, including logs and emails, into the assistant.
- Audit: every tool call must be attributable to a tenant and a user.
- The model provider must receive no credentials.
Why the traditional approach fell short
Before the assistant, support was a help centre search plus a ticket form. Search over public articles cannot answer "why was my invoice higher this month?", because the answer is in the tenant's own data. A rules engine can map fixed questions to fixed lookups, but administrators phrase questions in open-ended ways and the rule set never ends. Manual handling is correct and slow. The assistant turns an open question into a lookup, and the same flexibility means a tool argument is no longer written by a developer who knows the tenant model.
Options considered
| Option | Idea | Verdict |
|---|---|---|
| A. Validate in the Orchestrator | Keep the design, add a check that the tenant id in the tool request equals the session's tenant | Rejected as the only control: it is one check in one place, the Tool server stays able to read every tenant, and the next tool someone adds may skip it |
| B. Tenant-scoped credential, enforced below the model | The tool schema has no tenant parameter. The gateway sets the tenant, and the Policy engine returns a short-lived credential scoped to that tenant. The Tool server and Document store enforce it | Accepted: the model cannot widen access, and two layers must both fail for a leak |
| C. No tenant-data tools | The assistant answers only from public documentation and offers a ticket form | Kept as the fallback for tenants that decline tool access, and as the simplest option if the lookups are rarely used |
The default here is B, because authority must never live in a value the model writes. Option C is not a failure: for a small feature set it is the cheaper and safer design.
Solution architecture
Figure 2. Container view with trust boundaries
How to read this: zones are boxes, thick arrows cross a trust boundary and name the control applied there, thin arrows stay inside a zone, and the dotted arrow is asynchronous.
Figure 3. Runtime sequence of the corrected tool call
How to read this: the alt block has an allowed path and a denied path, and in both the Orchestrator ends by publishing an audit event.
How AI solved it, step by step
One request, end to end, in the corrected design.
- The administrator of tenant T asks, "Why is ticket 4821 still open?" The Client app sends it to the API gateway with a session token.
- The API gateway verifies the session and attaches the tenant and the user as fields of the request. These fields never appear in prompt text as instructions.
- The Orchestrator builds the prompt from the question and the tool descriptions. No tool takes a tenant argument.
- The Model gateway forwards the prompt to the LLM provider, which replies with a request for the ticket-history tool with the ticket reference.
- The Orchestrator treats the reply as a request, not a command, and asks the Policy engine whether this user of this tenant may call this tool.
- If the answer is yes, the Policy engine returns a 60-second credential scoped to tenant T and that tool. If the answer is no, the Orchestrator makes no call and tells the user.
- The Tool server validates the credential, then queries the Document store with a tenant filter taken from the credential.
- The Document store returns the record, or nothing if the reference belongs to another tenant. The Orchestrator gives the result to the model for the final answer and publishes an audit event.
Every decision is made by code the model cannot influence: the gateway (who is this), the Policy engine (may they), the Tool server and Document store (enforce).
Under the hood
The mechanism behind the original flaw is that a model's output is shaped by every token in its context, and the context contains text the team did not write: the user's message, retrieved documents, tool results. A tool argument is part of that output. So any value that grants authority, such as a tenant id, a user id or a file path root, is attacker-influenced if it can be set by the model, whether or not anyone attacks. The remedy is structural: remove the parameter, supply the value from a verified channel, and enforce it again where the data is read. The trust-boundary section of Diagramming AI systems shows how to mark these crossings; the course's security lesson will cover the attack chains in detail.
What went wrong / failure modes
- A new tool is added later with a tenant parameter. Mitigation: the tool schema review is part of the design checklist, and the Tool server rejects calls whose credential is missing no matter what the arguments say.
- A cache is keyed by ticket reference only. One tenant's cached record could be served to another. Mitigation: every cache key includes the tenant, and the review asks of each store, "what is its key?"
- The credential is longer-lived than the request. Mitigation: a short lifetime and a scope of one tenant and one tool.
- Audit gaps. If the Event bus is down, events could be lost. Mitigation: the Orchestrator buffers events locally and retries, and a tool call whose audit event cannot be buffered is refused.
- Policy engine unavailable. Mitigation: fail closed and tell the user the lookup is unavailable.
Results
These are illustrative targets, not measurements. The target for cross-tenant reads in a pre-release adversarial test set of 300 prompts (pasted text that names another tenant, requests to "ignore the account", ticket text containing an instruction) is zero. The target for added latency from the Policy engine call is 30 ms at the median. The target for audit coverage is that 100% of tool calls produce an audit event carrying tenant, user and tool. As a what-if for sizing the risk, a 0.1% wrong-tenant error rate would mean 10 wrong-tenant reads a day across 10,000 daily tool calls (25,000 requests x 40%). The review outcome is qualitative: the missing boundary was found before any tenant data was exposed, because a sequence diagram was drawn.
Tradeoffs and when NOT to use this approach
- Option B adds a Policy engine call and credential handling to every tool call. For a read-only assistant over public data it is not worth it.
- A shared Tool server remains a concentration of access. If tenants have contractual isolation requirements, separate stacks (the option the budget ruled out) may be required, and no amount of in-process checking replaces that.
- Do not treat this as a fix for prompt injection. It limits what an injected instruction can reach; it does not stop the model from being steered.
Design rules extracted
- Authority never lives in a value the model writes: tenant, user and permission come from a verified channel.
- Draw the runtime sequence of one tool call before approving a design; the container diagram alone hides where identity is decided.
- Enforce at two layers, the tool tier and the data tier, so that one missed check is not a leak.
- Keys of caches, indexes and logs include the tenant.
- When the policy check or the audit path is down, fail closed.
Assumptions
- 400 tenants, 25,000 assistant requests per day, and 40% of requests call a tool, which gives 25,000 x 0.40 = 10,000 tool calls per day.
- A what-if used in the review: an error rate of 0.1% on 10,000 daily tool calls would be 10 wrong-tenant reads per day. The 0.1% is a what-if, not an observed rate.
- The adversarial test set of 300 prompts, the 60-second credential lifetime and the 30 ms latency target are chosen for illustration.
- The vendor, product and tenant ids (such as t-17) are invented.
Last verified: 2026-10-21