Capstone: Design, Build and Ship an Orders API on Apigee

Threat protection. Writes get structural limits. The numbers are assumed; derive real ones from the largest legitimate order. The policy runs only for application/json, which is why RF-UnsupportedMediaType rejects any other content type first.

<JSONThreatProtection name="JTP-LimitBody" continueOnError="false" enabled="true">
  <Source>request</Source>
  <ArrayElementCount>100</ArrayElementCount>
  <ContainerDepth>4</ContainerDepth>
  <ObjectEntryCount>20</ObjectEntryCount>
  <ObjectEntryNameLength>64</ObjectEntryNameLength>
  <StringValueLength>200</StringValueLength>
</JSONThreatProtection>

CORS is deliberately omitted: callers are servers, and a client secret cannot live in a browser. OAuth scopes are also skipped, since the product already restricts operations by path and method.

Backend credentials and secrets. The backend token lives in an environment-scoped KVM and is read into a private. variable so Debug never shows it. The consumer's token is replaced before the call, and identity headers are removed then set, so a caller cannot forge them:

<KeyValueMapOperations name="KVM-GetBackendToken" mapIdentifier="orders-config">
  <Scope>environment</Scope>
  <Get assignTo="private.backend.token"><Key><Parameter>backend-token</Parameter></Key></Get>
</KeyValueMapOperations>

<AssignMessage name="AM-BackendAuth">
  <Remove><Headers>
    <Header name="Authorization"/><Header name="x-client-id"/><Header name="x-client-app"/>
  </Headers></Remove>
  <Set><Headers>
    <Header name="Authorization">Bearer {private.backend.token}</Header>
    <Header name="x-client-id">{client_id}</Header>
    <Header name="x-client-app">{developer.app.name}</Header>
  </Headers></Set>
  <IgnoreUnresolvedVariables>false</IgnoreUnresolvedVariables>
  <AssignTo>request</AssignTo>
</AssignMessage>

IgnoreUnresolvedVariables is false on purpose: a missing identity fails the call instead of reaching the backend anonymous. Southbound TLS is enforced on the target servers (step 7). The backend must still decide whether this app may see this order. A KVM change can take up to the cache expiry (default 300 seconds) to reach traffic.

Step 4: traffic and performance

Three different jobs, three policies, plus timeouts and balancing already in the target.

<SpikeArrest name="SA-ProtectBackend" continueOnError="false" enabled="true">
  <Rate>50ps</Rate>
  <UseEffectiveCount>true</UseEffectiveCount>
</SpikeArrest>

SA-PerClient is the same policy with <Identifier ref="client_id"/> and <Rate>10ps</Rate> (assumed). The plan allowance reads from the product the token belongs to:

<Quota name="Q-PlanFromProduct" continueOnError="false" enabled="true">
  <Identifier ref="client_id"/>
  <Distributed>true</Distributed>
  <Synchronous>false</Synchronous>
  <UseQuotaConfigInAPIProduct stepName="OAuth-VerifyAccessToken">
    <DefaultConfig><Allow>100</Allow><Interval>1</Interval><TimeUnit>hour</TimeUnit></DefaultConfig>
  </UseQuotaConfigInAPIProduct>
</Quota>

The reference confirms stepName may name an OAuthV2 policy; the default config is a small fallback (assumed). Counters are per product operation. With asynchronous sync a few calls over the limit can slip through; if billing needs exactness, set Synchronous to true and accept lower throughput.

Caching only where it is safe. GET /orders/{orderId} is cached; the paged list is not, because cursors and fresh orders make hits rare and stale pages confusing. The response depends on which partner asks, so the caller's identity is part of the key, the opposite of the "no API key in the key" rule for shared public data:

<ResponseCache name="RC-GetOrder" continueOnError="false" enabled="true">
  <CacheKey>
    <Prefix>orders-get</Prefix>
    <KeyFragment ref="request.uri"/>
    <KeyFragment ref="client_id"/>
  </CacheKey>
  <ExpirySettings><TimeoutInSeconds>30</TimeoutInSeconds></ExpirySettings>
  <SkipCachePopulation>response.status.code != 200</SkipCachePopulation>
</ResponseCache>

A 30-second TTL (assumed) means a partner can see a status up to 30 seconds old; agree that with the business. It sits in the conditional flow, so it runs after authentication.

Timeouts, balancing and retries are in the target above: a failing server leaves rotation and returns when healthy. A retry after the listed 503 is safe for POST only because the backend honours Idempotency-Key. Keep api.timeout (10 seconds, assumed) below the partner's client timeout, or partners give up before Apigee can answer cleanly.

Step 5: faults and observability

One error shape for every failure. The policy pattern is identical for 401, 500, 503 and 504; here is the 429 one, which also sets Retry-After:

<AssignMessage name="AM-Error-429">
  <Set>
    <Headers><Header name="Retry-After">1</Header></Headers>
    <Payload contentType="application/json" variablePrefix="@" variableSuffix="#">{"error":{"code":"RATE_LIMITED","message":"Too many requests. Slow down and retry.","correlationId":"@messageid#"}}</Payload>
    <StatusCode>429</StatusCode>
    <ReasonPhrase>Too Many Requests</ReasonPhrase>
  </Set>
  <IgnoreUnresolvedVariables>true</IgnoreUnresolvedVariables>
</AssignMessage>

The RF-* RaiseFault policies and AM-Correlation-Header follow the templates in the fault-handling lesson. A constant one second suits the spike limit; for a quota rejection, derive the wait from ratelimit.Q-PlanFromProduct.expiry.time. This body uses the same contract as every other error the API returns, so partners see one shape. Codes: UNAUTHENTICATED 401, MISSING_IDEMPOTENCY_KEY and UNSUPPORTED_MEDIA_TYPE 400, RATE_LIMITED 429, BACKEND_UNAVAILABLE 503, BACKEND_TIMEOUT 504, INTERNAL_ERROR 500.

Whichever branch fires, the partner gets the same JSON shape with the id that is in the log.

Logging. One MessageLogging policy in the PostClientFlow covers success and failure without adding latency:

<MessageLogging name="ML-Access-Log">
  <CloudLogging>
    <LogName>projects/{organization.name}/logs/orders-api-access</LogName>
    <Message contentType="text/plain">correlation_id={messageid} client_id={client_id} verb={request.verb} path={request.path} status={message.status.code} fault={fault.name}</Message>
  </CloudLogging>
</MessageLogging>

Response-side variables can be unavailable in the PostClientFlow after an error, so check in a trace which fields are populated. Never log tokens or bodies.

Dashboards, alerts and SLO checks. The built-in Analytics dashboards (API Proxy Performance, Error Code Analysis, Latency Analysis, Cache Performance, Target Performance) cover trends. Cloud Monitoring alerts cover the SLOs (assumed 99.9 percent, p95 under 600 ms):

• An error-ratio alert on proxy/response_count filtered by response_code, as a share of all responses. A 99.9 percent target leaves about 43 minutes of outage budget per 30 days.
• A p95 alert on the proxy/latencies distribution.
• A watch on 429 counts per app and the cache hit ratio, since a silent limit looks like an outage. Filter by fault_source to split proxy from target failures. Treat Apigee's sample thresholds as starting points, and expect alerts to lag by minutes.

Step 6: products, developers and apps

Create the two tiers. Quotas sit on each operation (assumed values), and the Plus tier adds POST. The proxy and target must already be deployed, since product creation can fail otherwise. Setup assumes ORG, a gcloud token and jq.

export TOKEN=$(gcloud auth print-access-token)
export API=https://apigee.googleapis.com/v1/organizations/$ORG

product-basic.json:

{
  "name": "partner-orders-basic",
  "displayName": "Partner Orders Basic (read only)",
  "approvalType": "auto",
  "environments": ["test", "prod"],
  "operationGroup": {
    "operationConfigs": [
      { "apiSource": "orders-v1", "operations": [{ "resource": "/orders", "methods": ["GET"] }],
        "quota": { "limit": "600", "interval": "1", "timeUnit": "hour" } },
      { "apiSource": "orders-v1", "operations": [{ "resource": "/orders/*", "methods": ["GET"] }],
        "quota": { "limit": "1200", "interval": "1", "timeUnit": "hour" } }
    ]
  }
}

Create partner-orders-plus from the same shape with the Plus column and "approvalType": "manual", so a person vets every write-capable partner. Then onboard:

curl -s -X POST "$API/apiproducts" -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" -d @product-basic.json
curl -s -X POST "$API/apiproducts" -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" -d @product-plus.json

curl -s -X POST "$API/developers" -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
  -d '{"email":"dev@partner.example","firstName":"Dana","lastName":"Partner","userName":"dana"}'

curl -s -X POST "$API/developers/dev@partner.example/apps" -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
  -d '{"name":"partner-erp-app","apiProducts":["partner-orders-basic"]}' | tee app.json

Exchange the credentials for a token and call the API (hostname illustrative):

HOST=partner-test.example.com
KEY=$(jq -r '.credentials[0].consumerKey' app.json); SECRET=$(jq -r '.credentials[0].consumerSecret' app.json)
ACCESS=$(curl -s -X POST "https://$HOST/oauth/token" -u "$KEY:$SECRET" -d grant_type=client_credentials | jq -r .access_token)
curl -i -H "Authorization: Bearer $ACCESS" "https://$HOST/orders/v1/orders?limit=5"

How to choose the tier for a new partner:

Step 7: environments and deployment

The same bundle runs everywhere. What differs is outside it: target servers orders-a and orders-b and the orders-config KVM, created in each environment.

Hostnames belong to the environment groups, the revision never changes on the way up, and only the target servers differ. An evaluation organization allows two environments at most, so run the lab with test and prod.

Create the per-environment configuration (repeat per environment with its own hosts; BACKEND_TOKEN comes from your secret store, never from git):

create_target() {   # $1 environment, $2 name, $3 host
  curl -sS -X POST "$API/environments/$1/targetservers" -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
    -d "{\"name\":\"$2\",\"host\":\"$3\",\"protocol\":\"http\",\"port\":443,\"isEnabled\":true,\"sSLInfo\":{\"enabled\":true,\"enforce\":true}}"
}
create_target test orders-a orders-a.test.internal.example.com
create_target test orders-b orders-b.test.internal.example.com

curl -sS -X POST "$API/environments/test/keyvaluemaps" -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
  -d '{"name":"orders-config","encrypted":true}'
curl -sS -X POST "$API/environments/test/keyvaluemaps/orders-config/entries" -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
  -d "{\"name\":\"backend-token\",\"value\":\"$BACKEND_TOKEN\"}"

Build once, deploy to the lower environment, then promote the same revision (the token proxy ships the same way):

(cd proxies/orders-v1 && zip -r -X ../../dist/orders-v1.zip apiproxy)
apigeecli apis create bundle -n orders-v1 -p dist/orders-v1.zip -e test -o "$ORG" -t "$TOKEN" --ovr --wait
apigeecli apis listdeploy -n orders-v1 -o "$ORG" -t "$TOKEN"                      # note the revision, say 7
apigeecli apis deploy -n orders-v1 -v 7 -e prod -o "$ORG" -t "$TOKEN" --ovr --wait

Verify with curl through the environment's own hostname. Each negative test must show the contract body, not an Apigee default:

Roll back by redeploying the previous revision (undeploying alone does not restore it):

gcloud apigee apis deploy 6 --api=orders-v1 --environment=prod --organization="$ORG" --override

Step 8: the CI/CD pipeline

The shape is the CI/CD lesson's. This API has no JavaScript callouts, so there is no unit-test stage; lint and integration tests carry the weight.

# Outline only: the CI/CD lesson has the full workflow, federated auth and rollback steps.
on: { pull_request: {}, push: { branches: [main] } }
jobs:
  verify:       { runs-on: ubuntu-latest }              # npx apigeelint -s proxies/orders-v1/apiproxy --profile apigeex --maxWarnings 0
  build:        { needs: verify }                       # zip once, upload the artifact
  deploy-dev:   { needs: build, environment: dev }      # import, apigeecli apis deploy --ovr --wait, newman tests
  promote-prod: { needs: deploy-dev, environment: production }   # required reviewers; same revision, smoke test, rollback

Step 9: production readiness review

Walk this list before launch; each line needs evidence (a policy, a test, a dashboard), not an opinion.

• Design: one spec, proxy and product set per version; no business rules in the proxy; a sunset plan exists.
• Security: every path passes verification (try an unknown path); products list proxies, operations and environments; secrets only in encrypted KVMs behind private. variables; the backend authorises per object; southbound TLS enforced; token endpoint rate limited.
• Performance: limits tested above their values; timeouts from measured latency; two or more target servers with MaxFailures above zero; cache keys reviewed; load test done outside production.
• Operations: one error shape proven by negative tests; correlation id returned and logged; alerts with an owner and a runbook; Debug masks configured.
• Delivery: all in git; the pipeline is the only path to production; same revision promoted; rollback rehearsed; no long-lived CI keys.

How to practise this yourself

Use an evaluation organization (free for 60 days, learning only, no real data). The hosted mock target does not implement /orders, so the lab makes flagged, lab-only edits: point both target servers at mocktarget.apigee.net, add <Path>/echo</Path> in the HTTPTargetConnection (Path applies only with a load balancer) and set the health monitor path to /json. Confirm in Debug where /orders/v1/orders/1001 lands on the mock; if it returns a 404, set target.copy.pathsuffix to false as in the proxies lesson.

How to extend it (next steps)

• Next step: an AI endpoint. Add a summaries flow that calls an LLM with token quotas and the key in a KVM: Apigee as an AI Gateway.
• Next step: multi-region. Create an instance in a second region and attach the existing environments to it (environments lesson).

Interview-style questions

1. Why OAuth client credentials instead of an API key here? Callers are servers handling customer data; tokens are short lived and revocable per app. A key is a static secret.
2. Where does a partner's quota live, and what makes it apply? On the product operations; it applies only because a Quota policy with UseQuotaConfigInAPIProduct runs after the verification policy it names.
3. Why spike arrest first and quota after authentication? Spike arrest is cheap and needs no identity; quota needs the caller's identity.
4. What if a product lists no proxies? Every app holding it can reach every proxy in the organization.
5. Version versus revision? A version is the consumer contract (the base path); a revision is an immutable snapshot, the unit of deployment and rollback.
6. How do you keep one bundle across environments? Name target servers and KVMs in the bundle, define values per environment, promote unchanged.
7. Why is client_id in the cache key? The response depends on who asks; without it one partner could receive another's cached order.
8. How does a backend 503 become your contract body? A target FaultRule matches fault.name, AssignMessage sets the JSON and status, and a DefaultFaultRule catches the rest. Never match message text.
9. Why log in the PostClientFlow? It runs after the response on success and error, so it adds no latency.
10. A revoked token still works. Why? Apigee caches tokens, so revocation can take about three minutes; use short lifetimes.
11. Is retrying a failed POST safe? Only if the backend deduplicates by Idempotency-Key; the gateway does not.
12. What does the gateway not do? Object-level authorisation. It authenticates and shapes traffic; the service checks that this app may read this order.

In practice: a launch review of the Orders API

The situation

A team built orders-v1 as a demo: a shared API key and a hard-coded backend URL. Two weeks before three partners go live, a launch review asks for evidence on security, limits, errors and rollback.

Starting architecture

What changes

1. Write the contract; split the shared key into per-partner apps and two products.
2. Add OAuth, spike arrest, per-client limits and the product-driven quota.
3. Add the error layer, the access log and the alerts.
4. Move the backend into target servers and a KVM; add the pipeline with approval and rollback.
5. Run the negative-test table, then the readiness list.

Result

The review finds evidence for each line: a rejected call per failure type, one rehearsed rollback, alerts with an owner. Open items are recorded, such as the cache TTL awaiting business sign-off. A misbehaving partner hits its own limits, and an incident starts from a correlation id.

Architect's notes

• Choose the simplest control you can defend. For low-risk data a key plus quota is simpler; the token proxy is one more component to operate.
• Limits come from measurement. Replace every assumed number after a load test that includes a backend failure.
• Own the retry story. Client, gateway and backend retries multiply load; decide who retries.
• Cost. Several policies here are Extensible and may carry licensing implications; check your licence.

Common mistakes: the ten findings that fail audits and incidents

1. A product that protects nothing. No verification policy, or no proxies listed. Verify in the PreFlow and scope products.
2. Quota that never runs. Values on the product without a Quota policy, or Quota before authentication. Test with a tiny limit.
3. Wrong variable names. verifyapikey.* after OAuth verification. Use the unprefixed names and confirm in Debug.
4. Raw errors reaching partners. No DefaultFaultRule, message-text matching, stack traces. Test every failure type.
5. Secrets in the bundle or in Debug. Use encrypted KVMs and private. variables.
6. Caching the wrong thing. Per-partner data under a shared key, or errors and writes.
7. Defaults left in place. MaxFailures 0, a 55-second I/O timeout, non-distributed quota.
8. One bundle per environment. Tests stop predicting production.
9. No rollback rehearsal. Undeploy is not rollback; redeploy the previous revision.
10. No alert owner. Assign an owner and a runbook.

Key takeaways

• Design the contract, limits and SLOs first; label every assumed number.
• One proxy per major version; the product is the allow-list and the offer.
• Order the PreFlow: spike arrest, authenticate, per-client limit, quota, then per-operation flows.
• One error shape for every failure, one correlation id in the response and the log.
• Environment differences live in target servers and KVMs; build once, promote the same revision, rehearse rollback.
• Confirm every hedged detail (fault names, variable names, cache hits) in Debug.
• The backend still owns object-level authorisation.

Practice exercises

Starter (about 30 min): Write the requirements and assumptions table and the OpenAPI paths for your own three-operation API, with the Error schema. Map each decision to the lesson that implements it.

Intermediate (about 2 h): Complete lab steps 1 to 7. Run the negative-test table and record the fault.name Debug shows for each OAuth failure, then adjust the unauthenticated rule.

Advanced (half a day): Finish lab steps 8 to 10, then write a design note for one next-step extension (AI endpoint or second region) before building it.

Related reading

• Course overview — the roadmap and glossary this lesson closes.
• Apigee Fundamentals: Architecture and Request Flow — planes, groups and request flow.
• API Design, Products and Developer Apps — contract, versioning, products.
• Building Apigee API Proxies — bundle anatomy, flows, route rules.
• Securing APIs in Apigee — OAuth, threat protection, secrets.
• Traffic Management in Apigee — spike arrest, quota, caching.
• Fault Handling, Debugging and Monitoring — errors, logging, alerts.
• Environments, Revisions and Deployment — config and rollback.
• Apigee CI/CD — the full pipeline outlined here.
• Apigee as an AI Gateway and Best Practices — the AI extension.
• Observability and SLOs — SLIs, SLOs and burn alerts.

Official documentation

• OAuthV2 policy
• Quota policy
• Handling faults and Managing API products
• REST method: environments.keyvaluemaps.create
• apigeecli: apis create bundle and apis deploy
• Deployment overview and Provisioning an eval org