The first LLM feature at most companies ships with a provider API key in an environment variable. A year later there are a dozen teams doing the same thing — no cost visibility, keys sprawled across services, no shared rate limits, and no way to enforce a policy. This is the exact problem API gateways solved for REST a decade ago, and the fix for AI traffic is the same: put a gateway between your apps and the model providers. If you already run Apigee, you don't need a new product — you can build an AI gateway with policies you already know.
(New to the concept? Start with AI gateways: managing LLM traffic in the enterprise for the why; this post is the how, in Apigee.)
What the AI gateway does
A single control point in front of OpenAI / Anthropic / Vertex that centralises the cross-cutting concerns that don't belong in every app:
- Authentication of internal consumers (which team/app is calling?)
- Cost control via quotas and token metering per consumer
- Rate limiting to protect budgets and provider limits
- Logging of prompts/responses for audit and debugging
- Guardrails — payload masking, blocklists, request validation
- Provider routing — swap or fail over between models centrally
Build the proxy
Create an Apigee API proxy whose target endpoint is the LLM provider (e.g. https://api.anthropic.com). Your apps call the Apigee proxy URL instead of the provider directly — so you can attach policies to every request. Keep the real provider key in the target/backend config (or a Key Value Map), never in the client apps.
The policy stack
Attach these policies to the proxy request/response flow:
1. Authenticate the consumer — VerifyAPIKey (or OAuthV2 for OAuth). Each team gets its own app credential, so you can attribute usage.
<VerifyAPIKey name="VA-VerifyKey">
<APIKey ref="request.header.x-api-key"/>
</VerifyAPIKey>2. Rate limit — SpikeArrest smooths bursts; Quota enforces a per-consumer budget.