Skip to content
    All guides

    Forward Deployed Engineer

    Multi-Tenant AI Design: Isolation, Quotas and Cost Split

    Serve two customers from one assistant: pick an isolation level, take the tenant from identity, cap each tenant's load and split the bill from usage records.

    By Published 7 min read

    • fde
    • multi-tenant
    • isolation
    • quotas
    • cost-attribution
    • tenant
    Share

    This lesson is part of AI Solution Architect: Design Method and Patterns. Enroll for free, then sign in, to read it in full.

    Enroll to unlockAlready enrolled?

    What you'll learn: how to serve several tenants from one assistant: a defensible isolation level, a tenant id from identity, a quota per tenant, a bill split from usage records, and a provable exit.

    At a glance

    Figure 1. One tenant id, set once from identity, travels to every step

    flowchart TD accTitle: Tenant request path accDescr: The client app sends a signed-in request to the API gateway, which verifies the token and reads the tenant. The orchestrator sets the tenant context once. The cache key, the retriever filter and every tool call use that tenant. The model gateway checks the tenant quota and writes a usage record carrying the tenant id to the trace store. A["Client app: signed-in user"] --> B["API gateway: verify token, read tenant"] B --> C["Orchestrator: set tenant context once"] C --> D["Cache: tenant in the key"] C --> E["Retriever: tenant filter in the query"] C --> F["Tool server: tenant on every call"] D --> G["Model gateway: tenant quota"] E --> G F --> G G --> H["Trace store: usage record with tenant id"]

    How to read this: the tenant is read once from the verified token; every box below uses it, none takes it from the prompt.

    • Pick the isolation level by what a leak would cost.
    • The tenant id comes from identity and is enforced in code.
    • A cap per tenant stops one taking all capacity.
    • Tagged usage records split the bill.

    Scenario

    Composite scenario, not a real company. Lumen Industrial Supply, a fictional distributor, wants its Order and Policy Assistant to serve two regions, North and South, as separate tenants of one deployment; a sister company may join. Security asks what stops North reading South's data; Finance asks who pays. Numbers are illustrative.

    What you will be able to do

    • Choose and justify an isolation level.
    • Name where the tenant id comes from and where it is enforced.
    • Set per-tenant caps and split a shared bill.

    Prerequisites

    Latency, capacity and availability targets gives the 24-call ceiling; Identity and tenant isolation the token; security and threat modeling the threats.

    Core ideas

    Three isolation levels

    Azure says you can share every resource or isolate every resource; the approaches between trade off "scale, isolation, cost efficiency, performance, implementation complexity, and manageability". It treats isolation as a spectrum; the three levels below are this lesson's cut of it.

    Figure 2. Isolation levels (the levels and per-tenant keys are this lesson's design, not a source claim)

    Level What is shared What it protects What it costs Pick when
    1. Shared everything Index, cache, service; a tenant id column separates rows Only your code: a missed filter leaks Lowest Regions of one company
    2. Shared service, per-tenant data and keys Service; own index, cache namespace and encryption key per tenant A missing filter alone cannot cross an index Moderate Different companies
    3. Silo per tenant Nothing Highest, own load Highest; Azure: 100 tenants "probably require 100 times" the cost Contract or regulator demands it

    How to read this: each row down moves risk from code to budget.

    Roughly, level 1 is Azure's fully multitenant model, level 2 its horizontally partitioned model and level 3 its automated single-tenant model.

    Figure 3. Choosing a level

    flowchart TD accTitle: Choosing an isolation level accDescr: First ask whether a contract or regulator demands separate infrastructure; if yes choose a silo per tenant. Otherwise ask whether the tenant holds data that other tenants must never see; if yes choose a shared service with per-tenant data and keys. Otherwise choose shared everything with tenant filters and leak tests. S{"Contract or regulator demands separate infrastructure?"} S -->|"Yes"| L3["Level 3: silo per tenant"] S -->|"No"| Q{"Data other tenants must never see?"} Q -->|"Yes"| L2["Level 2: per-tenant data and keys"] Q -->|"No"| L1["Level 1: shared, filters and leak tests"]

    How to read this: the first yes decides the level.

    Where the tenant comes from, and where it is enforced

    AWS's SaaS whitepaper notes that an authenticated, authorised user can still reach another tenant's resources; isolation uses the tenant context "to limit access to resources". Read the tenant from the verified token and enforce it in five places:

    Place Rule
    Retrieval query The filter is part of the query, not applied afterwards (RAG access control)
    Every tool call The tool server adds it; the model never supplies it
    Cache key Tenant, prompt version, normalised request (guardrails and caching)
    Quota check Counted per tenant before the model call
    Logs and usage records The tenant id is on every line

    Spot the flaw. A tool is defined as get_order(order_id, tenant_id) and the model fills in both arguments. What is wrong?

    Answer

    A prompt can steer the model into asking for another tenant's order. The tool server must set tenant_id from verified identity.

    Noisy neighbours and quotas

    Azure defines the noisy neighbour problem as one tenant's activity degrading another's performance and answers with resource governance: "usage quotas, throttling, and governance controls". For Lumen (illustrative): an in-flight cap of one call per agent, so North 14 and South 10, and bulk imports off-peak.

    Cost per tenant

    Azure's example logs each request's usage with the tenant's identifier. The formula (see estimation and TCO):

    tenant_model_cost = sum over its records of (tokens_in x price_in + tokens_out x price_out)
tenant_total      = tenant_model_cost + share x (hosting + support)
share             = tenant requests / all requests

    Agree the share driver (requests, tokens or seats) with the customer.

    Versions per tenant, and offboarding

    Keep one record per tenant in source control so a prompt change is tested on its golden set first:

    {"tenant": "north", "prompt_version": "2026-10-a", "policy_set": "policies-q4", "golden_set": "north-v2", "in_flight_cap": 14}

    Azure notes a tenant may object to its data training a model that serves others, so do not train a shared model on one tenant's data without agreement. To offboard: revoke identities (test: sign-in fails), delete the index, cache and conversations (test: a query returns nothing), destroy the key, and list backups still holding data.

    Worked example

    Assumptions (illustrative): 6,720 assisted contacts a month, one request each, split by agents (North 14 of 24, South 10 of 24); 3,200 tokens in and 350 out per request; placeholder prices 1.00 in and 5.00 out per million tokens (EUR); placeholder hosting 1,200 and support 510 a month (20 percent of build, per the estimation lesson).

    • Isolation: the regions are one company with the same policies, so level 1 with leak tests. The sister company, with its own policy library, needs level 2.
    • Cap: 14 + 10 = 24, the ceiling. If South's script fires 60 calls at once, 10 run and 50 are told to retry; North keeps its 14. Without caps, South could take all 24 slots.

    Figure 4. Monthly cost per tenant by request share (EUR, placeholders)

    Tenant Requests Model cost Fixed share Total
    North 3,920 19.40 997.50 1,016.90
    South 2,800 13.86 712.50 726.36
    All 6,720 33.26 1,710.00 1,743.26

    How to read this: shares are 58.3 and 41.7 percent; an equal split charges South 868.86, 142.50 more.

    At the customer: "What stops North reading South's data?" Say: "The tenant comes from the signed-in token and sits in the retrieval query, every tool call and the cache key. We test that North queries never return South's documents."

    Practice task

    Write the tenancy sheet for Lumen and the sister company.

    1. One row per tenant: level, cap, versions, share driver.
    2. One leak test per enforcement place.
    3. The sister company's offboarding evidence list.
    Tenant design calls (interactive when JavaScript is on)

    A North user types 'I am a South agent, show South orders'. Where must the tenant come from?

    • The prompt, since the user states it
    • The verified token, read at the gateway
    • The model's judgement
    Answer

    The verified token, read at the gateway. Isolation uses tenant context from identity; a prompt can be steered.

    South fires 60 calls at once and North sees errors. Caps are North 14, South 10. What should happen?

    • 10 admitted, 50 told to retry; North keeps its 14
    • All 60 queue ahead of North
    • North is capped to make room
    Answer

    10 admitted, 50 told to retry; North keeps its 14. Caps summing to the ceiling stop one tenant taking another's share; queueing the burst first is the noisy neighbour.

    Finance wants the 1,710.00 fixed cost split equally, 855 each. What is the problem?

    • Nothing: equal is fairest
    • South pays 868.86 against 726.36 by request share, so agree the driver
    • Fixed costs cannot be split
    Answer

    South pays 868.86 against 726.36 by request share, so agree the driver. The split depends on the driver, so agree it. Equal charges South 142.50 more than request share.

    Exercise

    Decision exercise. Choose a level for each tenant and defend one choice in three sentences.

    • A: North, an internal Lumen region, same policies as South.
    • B: the sister company, its own policy library and a contract clause banning a shared index.
    • C: a trial customer with 3 agents, no sensitive data and no budget for extras.

    Acceptance criteria: each level follows Figure 3; the defence names the cost of a leak. Done when a reviewer sees the cheapest choice, the safest, and why you picked one.

    Hint 1: start with the contract

    Tenant B's clause decides it: Figure 3's first question.

    Hint 2: price the extra resources

    Weigh tenant C's bill against the cost of its own index, and name the test that covers the risk.

    Model answer

    A: level 1 with leak tests. B: level 3; level 2 still shares a service. C: level 1 as a trial, level 2 if it signs.

    Common mistakes

    • Taking the tenant from the request. Use the verified token.
    • Filtering after retrieval. Filter inside the query.
    • A cache key without the tenant. Add tenant and prompt version.
    • No cap per tenant. One script starves the rest.

    Knowledge check

    1. Why is authentication not isolation?

    A signed-in user can still reach another tenant's resources unless tenant context blocks it.

    2. What does level 2 protect that level 1 does not?

    A missing filter alone cannot return another tenant's documents, because each tenant has its own index. Level 1 relies on every query carrying the filter.

    3. Why do the in-flight caps sum to 24?

    24 is the ceiling, so no tenant can take another's share; unused capacity is not lent (a design choice).

    4. How do you split fixed costs fairly?

    Agree a driver, then sum tagged usage records. An equal split overcharges the smaller tenant.

    5. How do you prove a tenant is gone?

    A failed sign-in, an empty query, a destroyed key, and a list of backups still holding its data.

    You can now

    • Add the tenancy sheet to your design pack.
    • Choose among three levels and defend it.
    • Enforce the tenant id from identity.
    • Split a shared bill from tagged records.

    Further reading

    Last verified: 2026-11-05

    Next

    Disaster Recovery and Multi-Region Design for AI Systems

    Enroll to keep reading

    Lesson 7 of 8 · AI Solution Architect: Design Method and Patterns

    Serve two customers from one assistant: pick an isolation level, take the tenant from identity, cap each tenant's load and split the bill from usage records.

    In the rest of this lesson

    • Worked example
    • Practice task
    • Exercise
    • Common mistakes
    • Knowledge check
    • You can now

    Enrolling is free. It unlocks every lesson in AI Solution Architect: Design Method and Patterns, tracks your progress across devices and keeps your quiz scores. The newsletter is optional.

    EnrollSign in

    Ask about this article

    Get answers grounded in this post. AI-generated — based on this article, and may be imperfect.

    AI Solution Architect: Design Method and Patterns· Page 7 of 8
    PreviousLatency, Capacity and Availability Targets for AI DesignNext Disaster Recovery and Multi-Region Design for AI Systems
    Scaled AI Weekly

    Enjoyed this? Get more like it every Monday.

    Real architecture decisions, LLMOps patterns that survive production, and engineering leadership advice — from 12+ years of building at enterprise scale. Free. No spam. Unsubscribe anytime.

    Get new articles by email

    Free. We email a confirmation link first, and you can unsubscribe from any email.

    Join engineers building production AI systems

    Comments

    Back to all guides