Composite scenario, not a real company. Assembled from common industry patterns; every number is illustrative and its assumptions are listed at the end.
This case belongs to Cloud platforms for AI.
The leaders of the business ask for a provider change. This page is about the engineering that makes the change a configuration and an adapter, not a rewrite. Provider facts are as of 2026-10-23.
Context
- A retailer's assistant answers shopper questions about products from the catalogue. It handles 12,000 requests a day.
- Calls go through a Model gateway to one provider on one cloud. Prompts, tool definitions and retrieval settings are versioned in a prompt registry.
- A golden set of 400 labelled questions exists, built for the evaluation gate.
- The company decides to consolidate on a second cloud's model service.
Problem statement
| Requirement | Measure |
|---|---|
| R1: no change to Orchestrator code | The diff is one new adapter and configuration |
| R2: quality does not drop | One-sided 95% Wilson lower bound on the golden set at least 90% |
| R3: a safe release | 5% of 12,000 = 600 requests a day first, then a ramp |
| R4: fast rollback | One configuration value returns traffic to the old provider |
Constraints
- Shoppers must not see an interruption or a visible change in answer style.
- No credential may be copied from the old provider's configuration to the new one.
- The second cloud's region must satisfy the same residency rule as the first (data residency case).
Why the traditional approach fell short
A keyword FAQ cannot answer open questions about product fit, so the assistant exists. Rewriting every call site against a new provider's SDK mixes the migration with feature work and leaves no baseline to compare against. A prompt edited in place for the new model has no version to roll back to.
Options considered
| Option | Why accepted or rejected |
|---|---|
| A. Rewrite the feature against the new provider's SDK | Rejected: it touches every call site and cannot be rolled back with a setting |
| B. Stay on the current provider | The non-AI-change option: it remains the fallback if the business case fails |
| C. Gateway adapter, versioned configuration, evaluation gate and canary | Accepted |
| D. A weekend switch of all traffic | Rejected: it has no staged exposure and quality problems show up as complaints |
Solution architecture
Figure 1. The gateway absorbs the difference, the registry versions the rest
How to read this: the Orchestrator and the registry stay as they are, the release router splits traffic, and the evaluation service scores a sample of the new path.
Figure 2. One canary request
How to read this: steps 1 to 3 are the routing, steps 4 and 5 are the identity exchange the new adapter needs, and the last step is the quality check.
What the adapter must absorb, as documented on 2026-10-23:
| Difference | Examples from the provider pages |
|---|---|
| Authorization | AWS: IAM credentials or a Bedrock API key bearer token. Azure: a Microsoft Entra token for a stated audience, or a key. Google: Google credentials or a federated token |
| Address | AWS: bedrock-runtime.{region}.amazonaws.com. Azure: the resource's own name, a custom subdomain for Entra tokens. Google: a regional or global endpoint |
| What names the model | AWS: a model or inference profile ID. Azure: "your deployment name, which can differ from the model name" |
| Operation | AWS: Converse and InvokeModel. Google: generateContent for Gemini, rawPredict for Anthropic Claude |
How AI solved it, step by step
Under the hood
Adapter B is the one new piece that touches security, so it gets its own identity.
- AWS: on AWS compute the platform delivers the temporary credentials of an IAM role.
- Azure: a managed identity is assigned a role such as "Cognitive Services OpenAI User". Tokens for management and for inference have different audiences, so the adapter's identity does not need management rights.
- Google: a service account is attached to the workload, or an outside identity is federated through the Security Token Service.
Two Azure facts matter during a canary. Role assignments "can take up to five minutes to take effect", so a 403 right after setup is not yet a defect, and the resource needs a custom subdomain to accept Entra tokens. Where the provider's call log records the principal, as AWS's CloudTrail and invocation logs do, you can confirm that only that identity calls provider B. Mechanism: Under the hood: Workload identity and short-lived credentials.
What went wrong / failure modes
| Failure | Designed mitigation |
|---|---|
| A prompt tuned for provider A scores lower on B | The gate fails, a new prompt version is written and the golden set is run again |
| Tool schemas behave differently | Contract tests in the adapter, and the gate. The pages read do not document this, so it is treated as untested |
| Throttling differs | The gateway owns retries (reliability) and the adapter maps the provider's error |
| Content logging defaults differ | Record traces at the gateway, not in the provider's log |
| The new region breaks the residency rule | Region is a release gate item, see the data residency case |
| Adapter B's role was created minutes ago | Retry briefly before alerting, because assignments take time |
Results
These are illustrative targets, not measurements.
| Measure | Target |
|---|---|
| Orchestrator files changed | 0 |
| Canary traffic at the start | 600 of 12,000 requests a day |
| Provider A baseline on the golden set | 376 of 400 = 94.0%, lower bound 91.7% |
| Provider B passes at | 372 of 400 = 93.0%, lower bound 90.6% |
| Provider B fails at | 366 of 400 = 91.5%, lower bound 88.9% |
| Time to roll back | One configuration change |
Tradeoffs and when NOT to use this approach
- A common contract can hide a feature only one provider has, and using it means bypassing the adapter.
- Running two providers during the canary costs more for the overlap.
- The gate measures the golden set, not every question.
- If the feature depends on a provider-specific fine-tuned model or reserved capacity, the migration is not minimal.
Design rules extracted
- Put provider details in an adapter and in configuration, never in the Orchestrator.
- Version prompts and settings so a rollback is a value, not a revert.
- The evaluation set is the safety net: gate on a lower bound, not an average.
- Give each adapter its own identity, and copy no credential.
- Release by canary, and keep the old path deployable for an agreed time.
Assumptions
- 12,000 requests a day; 12,000 x 0.05 = 600. The 400 items, the 90% threshold and the one-sided z of 1.645 are invented.
- Wilson lower bounds (z = 1.645): 376/400 gives 91.7%, 372/400 gives 90.6%, 366/400 gives 88.9%. Statistics for gates: Evaluating AI systems.
- The 5-minute figure is Azure's; the others in the table are as the provider pages state them.
- Sources as of 2026-10-23: https://docs.aws.amazon.com/bedrock/latest/userguide/api-keys.html, https://docs.aws.amazon.com/bedrock/latest/userguide/vpc-interface-endpoints.html, https://learn.microsoft.com/en-us/azure/foundry-classic/openai/how-to/managed-identity, https://docs.cloud.google.com/iam/docs/workload-identity-federation, https://docs.cloud.google.com/gemini-enterprise-agent-platform/models/capabilities/request-response-logging.
Last verified: 2026-10-23