Put Apigee in front of LLM providers with token quotas, prompt limits, semantic caching, Model Armor, failover and streaming, then review the whole course.
What you'll learn: Which AI and LLM features Apigee documents today (token quotas, prompt token limits, semantic caching, Model Armor screening, SSE streaming, routing and fallback), how to assemble them into a working LLM proxy with valid policy XML, and how to use the course-wide best-practices and anti-patterns checklist as a final review of any Apigee design.
Every team that adds a language model feature ends up needing the same things: a way to know who is calling, a budget that cannot be blown by one runaway loop, a place to keep the provider key, a fallback when the provider is down, and a record of what happened. Those are gateway concerns, and Apigee is a gateway. Since Apigee added policies written for LLM traffic, the job is less about clever workarounds and more about composing documented building blocks in the right order.
The site already has a short post, Building an AI Gateway with Apigee (LLM Proxy), that builds a gateway from general-purpose policies such as a request-count Quota. This lesson goes further and differs in three ways: it uses the AI-specific policies Apigee now documents, it counts tokens rather than requests, and it ends with the review checklist for the whole course.
The core idea
Picture a company travel desk. Nobody books flights with the company card directly. Everyone goes through the desk, which knows who you are, checks your team's budget, picks an approved airline, rebooks you on another if the first cancels, and keeps a record for finance. Employees get convenience, the company gets control, and the airlines see one well-behaved customer.
An AI gateway is the travel desk for model calls. Apps and agents call one stable API on Apigee. Apigee identifies the caller, limits how many tokens they may send and consume, screens the prompt and the answer, serves repeat questions from a cache, forwards to a model provider with a key the caller never sees, falls back to a second provider on failure, and logs usage without logging secrets. The unit of cost is no longer the request: it is the token, so the controls must count tokens.
What Apigee documents for AI traffic today
Everything below comes from Apigee's own documentation (links at the end). Where something is not documented, the lesson says so.
Capability
Policy or feature
What the documentation says
Short-term prompt throttling
PromptTokenLimit
Throttles the number of tokens in user prompts, like SpikeArrest but counting tokens; rate in tokens per second (ps) or minute (pm). For short-term protection of the backend.
Longer-term token budgets
LLMTokenQuota
Counts tokens over a time window and rejects calls once the quota is exhausted (HTTP 429). Two modes: count the tokens in the response, enforce the limit on the request.
Semantic caching
SemanticCacheLookup and SemanticCachePopulate
Embeds the prompt with the Vertex AI text embeddings API and searches Vertex AI Vector Search for a similar earlier prompt; a hit returns the cached answer without calling the model.
Prompt and response screening
SanitizeUserPrompt and SanitizeModelResponse
Evaluate prompts and responses with Model Armor, using a Model Armor template you create.
Streaming
SSE support and EventFlow
Lets clients receive a model's server-sent events as they arrive; an EventFlow can run a limited set of policies on each event.
Routing and fallback
Policy-driven rules, shown in the "AI gateway example"
Routes requests to different models by condition and falls back to another provider on errors.
Agents and tools
MCP support, Extension processor policies
Exposes APIs as MCP tools and applies policies to an agent's model and tool calls that do not pass through a proxy. Covered briefly below.
Limits and conditions that matter in design, all from the same documentation:
LLMTokenQuota and PromptTokenLimit handle text-based tokens only, need you to supply the exact JSON path where the token count lives (it is not auto-detected), and PromptTokenLimit uses a standard tokenizer that may differ from a given provider's count. The last request allowed by LLMTokenQuota can overshoot the remaining quota.
These are Extensible policies, so their use may have cost or utilisation implications depending on your Apigee licence.
Edition and environment requirements differ by feature and page. The token policies are listed for Apigee and for hybrid from 1.16.1; Model Armor and semantic caching policies for hybrid from 1.15.1. The semantic caching tutorial says those policies deploy only in Intermediate or Comprehensive environments, while the AI gateway example states Comprehensive. Check the page for your edition before you design around a feature.
The documentation does not describe built-in adapters between provider API formats. If you expose one API in front of several providers, translating request and response shapes and model names is your work (the AI gateway example shows one way to do it).
How a request flows through the gateway
The first diagram is the architecture: one front door, several controls, several providers.
Reading the picture: the gateway is the only component holding provider credentials, and the only path to the providers. The consumers hold an Apigee API key that identifies their app and, through the API product, their budget.
The second diagram is the order of checks for one request. Cheap, local checks come first; paid or slow calls come last; the cache sits before the model because a hit saves the model call entirely.
sequenceDiagram
participant C as Client app
participant G as Apigee proxy
participant MA as Model Armor
participant VS as Vector Search
participant LLM as Model provider
C->>G: POST /ai/v1/chat/completions with x-api-key
G->>G: VerifyAPIKey, PromptTokenLimit, LLMTokenQuota enforce
G->>MA: SanitizeUserPrompt
MA-->>G: no match, continue
G->>VS: SemanticCacheLookup
alt Cache hit
VS-->>G: similar prompt found
G-->>C: Cached response, model never called
else Cache miss
G->>LLM: Forward with the provider key from the KVM
LLM-->>G: Answer plus token usage
G->>MA: SanitizeModelResponse
G->>G: LLMTokenQuota count, SemanticCachePopulate
G-->>C: Answer
end
G->>G: MessageLogging in PostClientFlow
This ordering is a design choice, not a documented requirement: it follows the principle of rejecting cheaply before spending. One question to settle by testing: the documentation says a hit returns the cached answer without calling the model, but it does not spell out which response-side steps still run, so check in a debug session whether the response-side LLMTokenQuota counter moves on a hit. Whether hits should consume a team's budget is a policy decision; see the architect's notes.
Implement it: a proxy for an OpenAI-compatible provider
The proxy below accepts chat-completions style requests at /ai/v1/chat/completions, authenticates the app, enforces prompt and token limits, injects the provider key from an encrypted key value map, counts tokens on the way back and logs usage. Each snippet is a separate file in the bundle (policies/*.xml, proxies/default.xml, targets/openai.xml). It uses an OpenAI-compatible request shape; for other providers, change the JSON paths.
The FaultRule follows the pattern in the PromptTokenLimit reference: match on fault.name and run a step that builds the client-facing error. The documentation advises trapping the errorcode rather than the faultstring, because the text may change.
Identify the caller. The consumer's key travels in a header; the policy records which app and which API products it belongs to.
Throttle bursts of large prompts. The prompt text is read from the last message. The Rate is tokens per minute per consumer key (2,000 here is an assumed value). The reference says UseEffectiveCount should be true for Apigee organisations.
Enforce the team's token budget on the request, count tokens on the response. The two policies share one counter through SharedName, and both use the same per-consumer Identifier. The LLMTokenQuota reference says Identifier creates a separate counter per value, but its shared-counter examples do not use Identifier, and it does not state that the two combine; confirm in a debug session that each app gets its own counter (the ratelimit.<policy>.used.count variable shows it). The limit, interval and unit come from custom attributes on the API product, read through the flow variables VerifyAPIKey exposes. The attribute names (llmQuota.limit and so on) are your choice; the Apigee token tutorial uses the same technique with its own names. The literal count, 1 and day are fallbacks.
Points taken from the reference: exactly one of CountOnly or EnforceOnly must be true, and either requires SharedName; type can be calendar (resets on a schedule from a StartTime), rollingwindow (each request is checked against tokens used in the preceding interval) or flexi (starts at the first request and resets after the interval); a violation returns 429 with the fault policies.llmtokenquota.LLMTokenQuotaViolation; the counters are scoped to the proxy that contains the policy. total_tokens is the field an OpenAI-compatible response uses; confirm the field names in your provider's API reference.
Translate the budget fault into a clean client error.
<AssignMessagename="AM-TokenBudgetExhausted"><Set><PayloadcontentType="application/json">{"error":"token_budget_exhausted","message":"This app has used its token budget for the current window."}</Payload><StatusCode>429</StatusCode><ReasonPhrase>Too Many Requests</ReasonPhrase></Set><AssignTocreateNew="false"transport="http"type="response"/></AssignMessage>
Keep the provider key out of sight. The key sits in an encrypted key value map at environment scope. The private. prefix on the variable name keeps the value out of debug sessions, as the KeyValueMapOperations reference explains; without it the value appears in cleartext in a debug session even though the map is encrypted.
Log usage, never prompts. The MessageLogging reference recommends the PostClientFlow, which runs after the response is sent and also after an error, and logs asynchronously. Log identity and numbers, not content.
The policy needs the Cloud Logging API enabled, and the proxy must be deployed with a service account that can write logs (the reference recommends roles/logging.logWriter). The reference also warns that response variables are not available in the PostClientFlow after an error flow, so use the message variable for anything you must log on error responses.
Target endpoint. Provider errors become faults (so fallback is possible), and the I/O timeout is raised because model calls are slow: the endpoint-properties reference gives 55,000 ms as the default and returns 504 on a timeout, so 120,000 ms here is an assumed value to tune.
Call it. The caller sees only the Apigee URL and its own key.
curl -sS -i https://api.example.com/ai/v1/chat/completions \
-H "x-api-key: ${TEAM_API_KEY}" \
-H "Content-Type: application/json" \
-d '{"model":"MODEL_ID","messages":[{"role":"user","content":"Summarise this support ticket in one sentence."}]}'
A normal answer returns the provider's JSON with status 200. When the app has used its budget the proxy returns the 429 body from AM-TokenBudgetExhausted; in a debug session the underlying fault shows policies.llmtokenquota.LLMTokenQuotaViolation. Replace MODEL_ID with a model your provider offers.
Patterns
Authentication and per-team quotas
One API product per tier (for example "assistant-standard" and "assistant-premium"), each with its own token limit attributes, and one app per team. The team's key identifies the app; the product decides the budget. Because LLMTokenQuota can read its limit from product attributes, raising a team's budget is a product change, not a proxy redeploy. Use PromptTokenLimit for the per-minute guard and LLMTokenQuota for the daily or monthly budget; the documentation positions them exactly this way, short-term spike control versus longer-term consumption.
Token budgets and cost attribution
Count tokens where they are reported, in the response. For attribution, log the app, the product, the model and the token count (as above), then multiply by your provider's price in your analytics tool: Apigee does not hold your pricing. If you want the numbers inside Apigee Analytics, the DataCapture policy can capture a flow variable (for example llm.total_tokens) into a data collector you create first; it supports up to 100 variables of up to 400 bytes each.
Routing and failover across providers
Routing by condition uses ordinary RouteRule elements: the first matching rule wins, and an unconditional rule goes last. The AI gateway example routes by a model-name prefix (names starting google/ go to one target, everything else to the default), and builds fallback this way: the target treats non-2xx responses as failures (success.codes of 2XX), a fault rule re-sends the prompt to the other provider with a ServiceCallout, and the client receives a normal 200. The example also runs a RaiseFault policy in each target's response flow when response.status.code is 400 or above, so that the DefaultFaultRule fires for every error response. Two caveats from that example: fallback triggers on 4xx and 5xx responses only, and the client may get a different model's output for the same prompt.
An outline of that shape for this proxy (valid XML, but the secondary host and its key are placeholders, and request and model-name translation between providers is not shown):
Attach both as steps of a DefaultFaultRule in the primary target endpoint. A DefaultFaultRule handles errors no other fault rule covers, so any fault in that target's flow triggers the failover: keep the target flow small so only provider failures reach it.
flowchart TD
A["Request arrives with a logical model name"] --> B{"Does the app's product restrict where data may go?"}
B -->|"Yes: approved provider only"| P["Route to the approved provider, no cross-provider fallback"]
B -->|No| C{"Is the model name prefixed for a specific provider?"}
C -->|Yes| R["RouteRule condition sends it to that target"]
C -->|No| D["Default target: primary provider"]
D --> E{"Provider returned 4xx or 5xx?"}
R --> E
E -->|No| F["Return the answer"]
E -->|Yes| G{"Is the request safe to replay on another provider?"}
G -->|"Yes: nothing has streamed yet"| H["Fault rule calls the secondary provider and returns its answer"]
G -->|"No: partial stream or side effects"| I["Return a clean error and let the client retry"]
The "safe to replay" question is a design judgement, not a documented rule: once tokens have started streaming to a client, switching providers mid-answer is not possible, so streaming endpoints usually fail fast instead of failing over.
Caching: exact and semantic
For repeated identical questions, ResponseCache is the general caching policy. Its reference limits a cached object to 256 KB and a key to 2 KB, and builds the key from KeyFragment elements; it does not say how request bodies are handled for POST requests, so test your cache key before relying on it. For similar-but-not-identical questions, the semantic cache is the documented answer.
The semantic cache needs Vertex AI infrastructure: an embeddings model endpoint, a Vector Search index and endpoint, and a service account with the AI Platform User role. SemanticCacheLookup sits in the request flow with a Threshold (default 0.9; the tutorial suggests 0.95) and a DistanceMeasureType; SemanticCachePopulate sits in the response flow with a TTLInSeconds (default 60). A hit adds a Cached-Content: true header in the tutorial. The tutorial also notes that Apigee ignores cache-control headers from the model, so the TTL is yours to set, and that deploying a new index can take 20 to 30 minutes.
The braces in capitals are placeholders for your own project values. Two threshold details from the reference: with DOT_PRODUCT_DISTANCE a hit needs the value to be at or above the threshold, while with the other distance measures it must be at or below; and the reference strongly encourages normalised vectors.
A privacy point that is reasoning, not documentation: the reference describes no per-consumer partition of the cache. Treat it as shared across everything that passes through the proxy, and do not use it for personalised answers or for conversations that include one user's data.
Redacting sensitive data
Think in layers, because no single layer catches everything.
Screen with Model Armor.SanitizeUserPrompt sends the prompt to a Model Armor template you define and exposes the verdict in flow variables (filterMatchState, matchesFound, per-filter results including prompt injection and sensitive data protection). A match on the template fails the call with a 400 (steps.sanitize.user.prompt.response.FilterMatched). The policy reference does not say whether a template rewrites sensitive text or only flags it, so read the Model Armor documentation before promising redaction. SanitizeModelResponse does the same for the answer.
Redact what you can name yourself. For patterns you know (e-mail addresses, long digit runs), a small JavaScript callout can rewrite the request before it leaves your perimeter. This sketch is deliberately simple and illustrative: regular expressions miss cases, so treat it as a second layer, not the only one.
// resources/jsc/redact-prompt.js (runs in the request flow, before the provider call)var body = JSON.parse(context.getVariable("request.content"));
varEMAIL = /[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}/g;
varDIGITS = /\b(?:\d[ -]?){13,16}\b/g;
for (var i = 0; i < body.messages.length; i++) {
var m = body.messages[i];
if (typeof m.content === "string") {
m.content = m.content.replace(EMAIL, "[EMAIL]").replace(DIGITS, "[NUMBER]");
}
}
context.setVariable("request.content", JSON.stringify(body));
Keep secrets out of debug output. Use private. variables for keys. Apigee's data masks are JSON configurations that apply to debug sessions: they replace matching values in the debug output with asterisks and do not change what is sent to targets or clients. They protect what a human sees while debugging; they do not sanitise logs or provider calls.
Do not log content at all unless there is a specific, approved need, and then redact first. The usage log above has no prompt field on purpose.
Streaming
Model providers stream tokens as server-sent events. Apigee documents SSE support: point the proxy at an SSE-enabled target and events reach the client as they arrive. For control over individual events there is a special flow, EventFlow, with content-type="text/event-stream". Facts from the reference worth knowing:
Only one EventFlow runs; if both endpoints define one, the target's is used.
It may contain up to four policies, and only these types: DataCapture, JavaScript, LLMTokenQuota, MessageLogging, PublishMessage, RaiseFault and SanitizeModelResponse.
Inside it, response.event.current.data holds the event's data and response.event.current.content the whole event string; setting the latter to an empty string stops that event being sent. Each event is limited to 10 MB.
LLMTokenQuota skips events that carry no token usage metadata, so a stream is counted only where the provider reports usage. For SSE the quota reference says to read the count from response.event.current.data with the jsonPath array argument set to false.
Use separate target endpoint definitions for SSE and non-SSE targets; mixing them can cause inconsistent behaviour such as empty response.content.
A proxy deployed with an EventFlow is billed as Extensible.
A streaming target for the same proxy, selected by a route rule on a request field you extract (confirm the extracted value in a debug session before relying on the condition):
Many OpenAI-compatible APIs include usage in a stream only when the request asks for it; check your provider's documentation, and if the stream carries no usage the counter will not move. The llm.stream variable is not set by Apigee: create it with a small ExtractVariables policy (prefix llm, variable stream, JSONPath $.stream) in the proxy request flow, before routing, and put this route rule before the default one.
Agents and tools
For agents, the AI capabilities page lists two features. MCP support lets you expose APIs as MCP tools, with an OAuth V2 policy protecting an MCP proxy and JSON-RPC policies for finer enforcement. Extension processor policies apply Apigee policies to traffic that does not pass through a proxy, including an agent's model and tool calls. These are outside this lesson's hands-on scope; see the linked pages if agents are your use case.
Use cases
1. A shared internal LLM endpoint with team quotas
A platform team publishes /ai/v1/chat/completions once. Each team registers an app and subscribes to a product that carries its token limit. The proxy above enforces per-minute prompt limits and per-day budgets per app, holds the one provider key, and writes usage records tagged with app and product. A team that hits its budget gets a 429 with a clear message and nobody else notices. Finance gets per-team token totals from the usage log.
2. Provider failover
An assistant feature must keep answering when the primary provider has a bad hour. The target treats non-2xx as failure, the fault rule replays the prompt to the secondary provider, and the client gets a 200. Test it by pointing the primary target at a URL that returns 500. Use it only for requests that are safe to replay, accept that the answer may come from a different model, and log which provider answered so quality complaints can be traced.
3. A customer-facing assistant with data redaction
Customers type free text, which will contain things nobody planned for. The proxy runs SanitizeUserPrompt with a Model Armor template, applies a redaction callout for known patterns, never logs prompt text, screens the answer with SanitizeModelResponse, and does not use the semantic cache for personalised content. The result is layered protection, and a documented place to tighten each layer without touching the client.
In practice: consolidating three teams' LLM keys behind one gateway
NOTE
A realistic scenario, not a case history. This is a composite example built from common production patterns. It does not describe any specific company, and every number in it is an illustrative assumption.
The situation. Three product teams each call a model provider directly with their own key in an environment variable. Nobody can say which team caused last month's usage spike. A leaked key would need three teams to rotate it, and one team's retry loop once exhausted the provider's rate limit for everyone sharing the account.
What changes.
Create the proxy from this lesson, one environment key value map for the provider key, and an encrypted-map entry per provider.
Create three API products carrying token limit attributes (the numbers are assumed starting points and are changed in the product, not the proxy), and an app per team.
Switch each team's base URL to the Apigee proxy and give each its own key; rotate the provider key once, in the KVM.
Turn on PromptTokenLimit and the LLMTokenQuota pair, and add the usage log.
Add the semantic cache for the search-summaries team only, whose questions repeat, with a high threshold.
Add the secondary provider and failover for the support assistant only.
Run the integration tests from the CI/CD lesson against the dev proxy, including a test that exhausts a small budget and expects a 429.
Result. The provider sees one client. Each team has its own budget and a clear error when it exceeds it. A runaway retry loop hits that team's budget, not everyone's. Usage reports show tokens per app and model, so the spike has an owner. Rotating the provider key is one change in one place. Latency for the repeated search-summary questions drops on cache hits, at the cost of the Vertex AI components and their running cost (qualitative: measure your own hit rate before expanding the cache).
The consolidated best-practices checklist
This is the final review for the whole course. Use it as a design review list, then check each item against the lesson named beside it.
flowchart LR
R["Apigee design review"] --> D["Design"]
R --> S["Security"]
R --> P["Performance"]
R --> O["Operations"]
R --> L["Delivery"]
D --> D1["Thin proxy, business logic in backends"]
D --> D2["Products and apps model who may call what"]
S --> S1["Verify identity at the edge, least privilege"]
S --> S2["Secrets in encrypted KVM, private variables"]
P --> P1["Spike arrest, quota, token limits"]
P --> P2["Cache where safe, set timeouts"]
O --> O1["Fault rules, debug sessions, analytics"]
O --> O2["Async logging without sensitive data"]
L --> L1["Proxies in git, lint, tests"]
L --> L2["Build once, promote the same revision"]
Area
Do
Avoid
Design
Keep the proxy thin: cross-cutting policy at the gateway, domain logic in backends. Model consumers as apps subscribed to API products. Use shared flows for logic reused across proxies. Version the contract (base path major version) and run old and new side by side.
Business rules in JavaScript policies. One giant proxy for everything. Breaking a published contract in place.
Security
Verify identity at the edge (API key, OAuth, JWT, mTLS as the risk demands). Add threat protection to request bodies. Keep secrets in encrypted key value maps and read them into private. variables. Give deploy and runtime identities least privilege. For AI, screen prompts and responses and never expose provider keys.
Secrets in policy XML or the bundle. Trusting a header the client can set. Logging prompts and tokens in clear text.
Performance
Protect backends with SpikeArrest plus Quota; for AI add PromptTokenLimit and LLMTokenQuota. Cache what is safe to cache. Set explicit target timeouts. Prefer native policies to custom code. Use the ServiceCallout policy for extra HTTP calls so you get timeouts and error handling.
Unprotected proxies. Hand-written HTTP calls inside JavaScript. Caching personalised or authorised responses under a shared key.
Operations
Handle errors with fault rules that return one consistent error shape. Use debug sessions and Analytics to find problems. Log asynchronously in the PostClientFlow. Use flow variables and labels to attribute usage to apps and products. Alert on 4xx, 5xx and quota violations.
Synchronous logging to a slow sink in the request path. Leaking backend stack traces to clients. Debugging in production with unmasked sensitive data.
Delivery
Proxies and shared flows in git. Lint, unit and integration tests in the pipeline. Build once and promote the same revision with an approval before production. Federated identity from CI, per-environment service accounts. Rehearse rollback by redeploying the previous revision.
Editing proxies in the production console. A bundle per environment. Long-lived service account keys. Shipping without a smoke test.
Architect's notes
Choose the gateway for governance, not for model logic. Apigee is the right place for identity, budgets, screening, routing and audit. Prompt engineering, tool orchestration and retrieval belong in the application or an agent framework behind the gateway.
Decide whether cache hits cost budget. A hit saves the model call but did serve an answer. Many teams choose not to charge hits against token quotas, because the quota is a proxy for provider spend. Test what your proxy does on a hit, make it match the decision, and write the decision down, because teams ask.
Fallback is a quality decision as much as an availability one. A different model may answer differently, follow a different safety policy, or cost more. Decide per use case whether degraded availability or a different answer is the lesser evil.
Token counts are approximate in three places.PromptTokenLimit uses its own tokenizer, the last request in a window can overshoot, and unreported streaming usage is not counted. Use the numbers for control and attribution, and reconcile with the provider's invoice for money.
The AI policies are licence-sensitive. Extensible policies, the extra Vertex AI components for semantic caching, and Model Armor all have cost. Price them against the problem they solve before turning everything on.
Dedicated LLM gateways exist. If you do not already run Apigee, standing it up only for LLM traffic is a large step. If you do, reuse it: identity, products, analytics and delivery pipelines are already there.
Common mistakes
Counting requests instead of tokens. A request-count Quota treats a one-line prompt and a 100,000-token prompt alike. Use PromptTokenLimit and LLMTokenQuota for model traffic.
Reusing one LLMTokenQuota policy in several flows. The reference warns that one policy in several flows shares a single counter and can exhaust the quota faster than expected. Use the CountOnly and EnforceOnly pair with a shared name instead.
Forgetting the JSON paths. The policies cannot detect where usage appears in a provider's response. A wrong path means nothing is counted, so test with a debug session and a deliberately tiny budget.
Provider key in a plain variable. Without the private. prefix the value appears in debug sessions. Use it on every secret.
Logging prompts and completions "for debugging". They contain personal and confidential data. Log identity, model, status and token counts; capture content only with an explicit reason and redaction.
Failing over streaming responses. Once data has reached the client you cannot switch providers mid-answer. Fail fast on streams and let the client retry.
Mixing SSE and non-SSE in one target. The documentation warns of inconsistent behaviour. Keep separate target endpoints.
Setting a semantic cache threshold too low. Different questions then share answers. Start high (0.95 in the tutorial) and measure.
Skipping the integration test for the budget. A quota that has never been exhausted in a test has never been shown to work.
Key takeaways
Apigee documents AI-specific policies: PromptTokenLimit, LLMTokenQuota, SemanticCacheLookup and SemanticCachePopulate, SanitizeUserPrompt and SanitizeModelResponse, plus SSE streaming through EventFlow.
Count tokens, not requests: use PromptTokenLimit for short bursts and the CountOnly/EnforceOnly pair of LLMTokenQuota for budgets, with limits held on API products.
Keep provider keys in an encrypted key value map, read them into private. variables and never forward the consumer's credential.
Order checks cheap to expensive: identity, limits, screening, cache, then the model; log usage after the response in the PostClientFlow, without prompt content.
Fail over only when a replay is safe, and treat streaming, redaction and cache sharing as deliberate design decisions.
Several AI features depend on Vertex AI, Model Armor, licensing and environment type; confirm them for your edition before committing to a design.
Use the five-area checklist (design, security, performance, operations, delivery) as the closing review of every Apigee proxy.
Practice exercises
Starter (about 30 minutes). Read the LLMTokenQuota and PromptTokenLimit reference pages and write, in your own words, when you would use each. Then build the PromptTokenLimit and the CountOnly/EnforceOnly pair on a copy of an existing proxy pointed at any mock endpoint that returns a usage.total_tokens field, and confirm in a debug session that the counter moves.
Intermediate (about 2 hours). Build the full proxy from this lesson against a real or mock provider. Create two API products with different token limit attributes and two apps. Show with curl that app A is rejected with a 429 after its small budget is used while app B is unaffected. Add the usage log and confirm no prompt text appears in it. Write the integration test for the exhausted budget.
Advanced (half a day). Add a second provider and the DefaultFaultRule failover, and prove it by pointing the primary at a failing URL. Add SanitizeUserPrompt with a Model Armor template and the redaction callout, then write tests with an e-mail address and a long digit run in the prompt. Add a streaming target with an EventFlow and verify tokens are counted. Finally, run the review checklist against your own proxy and list five findings with the lesson that addresses each.
Lesson 10 of 11 · Apigee API Management: Beginner to Advanced
Put Apigee in front of LLM providers with token quotas, prompt limits, semantic caching, Model Armor, failover and streaming, then review the whole course.
In the rest of this lesson
Patterns
Use cases
In practice: consolidating three teams' LLM keys behind one gateway
The consolidated best-practices checklist
Architect's notes
Common mistakes
Enrolling is free. It unlocks every lesson in Apigee API Management: Beginner to Advanced, tracks your progress across devices and keeps your quiz scores. The newsletter is optional.
Real architecture decisions, LLMOps patterns that survive production, and engineering leadership advice — from 12+ years of building at enterprise scale. Free. No spam. Unsubscribe anytime.
Get new articles by email
Free. We email a confirmation link first, and you can unsubscribe from any email.