Fault Handling, Debugging and Monitoring in Apigee

<AssignMessage name="AM-Error-503">
  <DisplayName>AM-Error-503</DisplayName>
  <Set>
    <Payload contentType="application/json" variablePrefix="@" variableSuffix="#">{"error":{"code":"BACKEND_UNAVAILABLE","message":"The service is temporarily unavailable. Retry later.","correlationId":"@messageid#"}}</Payload>
    <StatusCode>503</StatusCode>
  </Set>
  <IgnoreUnresolvedVariables>true</IgnoreUnresolvedVariables>
</AssignMessage>

Create AM-Error-504 (code BACKEND_TIMEOUT, status 504) and AM-Error-500 (code INTERNAL_ERROR, status 500) the same way. Then one shared policy that stamps the correlation id as a header on every error:

<AssignMessage name="AM-Correlation-Header">
  <DisplayName>AM-Correlation-Header</DisplayName>
  <Add>
    <Headers>
      <Header name="X-Correlation-Id">{messageid}</Header>
    </Headers>
  </Add>
  <IgnoreUnresolvedVariables>true</IgnoreUnresolvedVariables>
</AssignMessage>

Step 2: add FaultRules to the TargetEndpoint

Backend failures occur at the target, so the rules live in the TargetEndpoint (the docs say to put rules where the policy runs). The conditions below are mutually exclusive, so evaluation order does not matter.

<TargetEndpoint name="default">
  <FaultRules>
    <FaultRule name="backend-timeout">
      <Step><Name>AM-Error-504</Name></Step>
      <Step><Name>AM-Correlation-Header</Name></Step>
      <Condition>(fault.name = "GatewayTimeout")</Condition>
    </FaultRule>
    <FaultRule name="backend-unavailable">
      <Step><Name>AM-Error-503</Name></Step>
      <Step><Name>AM-Correlation-Header</Name></Step>
      <Condition>(fault.name = "ServiceUnavailable") or (fault.name = "NoActiveTargets") or (fault.name = "ErrorResponseCode")</Condition>
    </FaultRule>
  </FaultRules>
  <DefaultFaultRule name="generic-error">
    <Step><Name>AM-Error-500</Name></Step>
    <Step><Name>AM-Correlation-Header</Name></Step>
  </DefaultFaultRule>
  <HTTPTargetConnection>
    <LoadBalancer>
      <Server name="orders-backend"/>
    </LoadBalancer>
  </HTTPTargetConnection>
</TargetEndpoint>

The fault.name values in these conditions come from Apigee's runtime error catalog. The distinction matters: ServiceUnavailable (503) and GatewayTimeout (504) are raised when Apigee cannot connect to the backend or the backend does not answer in time, and NoActiveTargets when every load-balanced server is out. When the backend itself answers with an error status, the catalog lists ErrorResponseCode (status 500, 503 or 504, matching what the backend returned), which is why the rule above includes it. That rule therefore answers 503 for any such backend error; if you want a backend 504 treated differently, add a status condition and confirm the variable in a Debug session. This is a normal workflow: reproduce, read fault.name from the trace, write the rule.

The sequence below shows what the client experiences when the backend fails, with the rules above in place.

The client never sees the backend's page; the log line carries the same correlation id the client received.

Step 3: raise your own errors for bad requests

For a client mistake that no policy catches, use RaiseFault behind a condition in the request flow. It runs in the ProxyEndpoint, so the FaultRule that handles it goes there too.

<RaiseFault name="RF-Missing-Customer-Id">
  <DisplayName>RF-Missing-Customer-Id</DisplayName>
  <FaultResponse>
    <Set>
      <Payload contentType="application/json" variablePrefix="@" variableSuffix="#">{"error":{"code":"MISSING_CUSTOMER_ID","message":"The x-customer-id header is required.","correlationId":"@messageid#"}}</Payload>
      <StatusCode>400</StatusCode>
    </Set>
  </FaultResponse>
  <IgnoreUnresolvedVariables>true</IgnoreUnresolvedVariables>
</RaiseFault>

Attach it in the ProxyEndpoint request flow with a step condition such as request.header.x-customer-id = null. Then add a ProxyEndpoint FaultRule so the shared header policy still runs. The docs say a FaultRule or DefaultFaultRule executes after the RaiseFault policy and its response data wins, and that the DefaultFaultRule runs only when no FaultRule ran (unless AlwaysEnforce is true). Matching RaiseFault explicitly keeps a ProxyEndpoint DefaultFaultRule, if you add one, from replacing your 400:

<FaultRules>
  <FaultRule name="client-raised">
    <Step><Name>AM-Correlation-Header</Name></Step>
    <Condition>(fault.name = "RaiseFault")</Condition>
  </FaultRule>
</FaultRules>

The docs warn that a RaiseFault and a FaultRule resetting the same status code fight each other. Dividing the work (RaiseFault sets the body and status, the FaultRule only adds a header) avoids it. Always check the final response in a trace; this is the place where interplay surprises appear.

Step 4: log every call, success or failure

The MessageLogging policy writes to Cloud Logging or to a syslog server (not both in one policy). It writes into an in-memory buffer and the response does not wait, so it adds no latency, and a logging failure does not fail the API call. The docs recommend attaching it in the PostClientFlow, which runs after the response is sent and accepts only MessageLogging policies. Because it runs on success and on error, one policy covers both.

<MessageLogging name="ML-Access-Log">
  <DisplayName>ML-Access-Log</DisplayName>
  <CloudLogging>
    <LogName>projects/{organization.name}/logs/orders-api-access</LogName>
    <Message contentType="text/plain">correlation_id={messageid} proxy={apiproxy.name} verb={request.verb} path={request.path} status={message.status.code} fault={fault.name}</Message>
  </CloudLogging>
</MessageLogging>

<PostClientFlow>
  <Response>
    <Step><Name>ML-Access-Log</Name></Step>
  </Response>
</PostClientFlow>

Notes from the docs: the proxy must use Google authentication, and the deployment service account needs the logging.logEntries.create permission (the roles/logging.logWriter role is the suggested way to grant it). Response variables can be unavailable in the PostClientFlow after an error, so the policy page says to use the message variable. The flow-variables reference describes message.status.code as the target response status, and fault.name as scoped to the error flow, so either may be empty in the log line; confirm in a trace which fields are populated. Keep log names to a small fixed set rather than building them from variables. Never log tokens or full bodies.

Debugging with Debug sessions

A Debug session records what happens to real requests inside the proxy: each flow, each policy, the variables read and written, and the response.

To start one in the Google Cloud console: open Apigee > Proxy development > API Proxies, select the proxy, open the Debug tab, click Start debug session, pick the environment, optionally set a filter, and click Start. Facts to plan around, all from the docs:

• There is a delay of about 15 seconds before the session activates, so requests sent immediately may be missed.
• In the UI a session records for 10 minutes or until 15 requests have been captured. Through the API the timeout defaults to 300 seconds with a maximum of 600, and can only be set when the session is created.
• A filter, using the same syntax as Apigee conditions, limits which transactions are captured. It can only be set when the session is created, and filtered-out requests do not count toward the limit.
• Session data is deleted automatically within 24 hours. You can share a session with people who have access to the organization, and the shared link is valid for 24 hours.

Debug captures message content, so it can show sensitive data. Apigee's answer is data masking: a JSON mask configuration, one per environment and empty by default, that replaces matching values with asterisks in debug output. It can target XML with XPath, JSON with JSONPath, and flow variables (naming request.content, response.content or message.content masks the whole body). It affects only debug data, never what reaches the target or client. Separately, a custom variable whose name starts with private. is hidden from Debug entirely rather than masked; use it for values read from a key value map.

# Mask a card number in request bodies and an auth header in debug output.
# Replace ORG and ENV; the mask applies to new debug sessions only.
curl -X PATCH \
  -H "Authorization: Bearer $(gcloud auth print-access-token)" \
  -H "Content-Type: application/json" \
  -d '{"requestJSONPaths":["$.card.number"],"variables":["request.header.authorization"]}' \
  "https://apigee.googleapis.com/v1/organizations/ORG/environments/ENV/debugmask"

Observability: what to use for which question

Apigee has several observation tools, and picking the right one is half the skill. From Apigee's own overview:

Analytics and custom data

Built-in dashboards (API Proxy Performance, Error Code Analysis, Latency Analysis, Cache Performance, Target Performance) are in the console under Apigee > Analytics. After a call, data can take up to around ten minutes to show. For business dimensions, such as "orders by customer tier", capture your own data with the DataCapture policy. (Apigee Edge's policy for this was called StatisticsCollector; the Apigee X documentation describes DataCapture, which needs a data collector resource created first, and Pay-as-you-go organizations must enable the API Analytics add-on.)

curl -X POST \
  -H "Authorization: Bearer $(gcloud auth print-access-token)" \
  -H "Content-Type: application/json" \
  -d '{"name":"dc_customer_tier","description":"Customer tier of the caller","type":"STRING"}' \
  "https://apigee.googleapis.com/v1/organizations/$PROJECT_ID/datacollectors"

<DataCapture name="DC-Customer-Tier" continueOnError="true">
  <Capture>
    <DataCollector>dc_customer_tier</DataCollector>
    <Collect ref="request.header.x-customer-tier"/>
  </Capture>
</DataCapture>

Collector names start with dc_, and continueOnError="true" keeps an analytics hiccup from failing the API call. The policy captures at most 100 variables and 400 bytes per variable. Then add a custom report using the collector as a dimension, ideally with a small, bounded set of values.

API Monitoring, metrics and alerts

API Monitoring needs the roles/apigee.readOnlyAdmin and roles/monitoring.viewer roles. In the console it lives under Apigee > Proxy development > API monitoring. Alerts are built on Cloud Monitoring metrics: choose a metric, a threshold and a duration, and attach a notification channel (email, PagerDuty, Slack and webhooks are supported). The metrics you will use most are proxy/request_count, proxy/response_count (filterable by response_code, fault_code and fault_source) and the proxy/latencies distribution. Apigee's guidance for an error alert is the ratio of error responses to total responses, with a threshold that follows your SLO. Alerts can lag the event by a few minutes, which is why they sit beside, not instead of, your own SLO reasoning.

Audit logs

Cloud Audit Logs record who changed Apigee configuration. The category follows the permission type of the method: methods that need an admin-write permission, such as updating an API proxy or creating a developer app, produce Admin Activity entries. Audit logs say nothing about runtime traffic. Check the audit-logging page for your product edition before relying on a specific filter, because Apigee hybrid and the API Management service use different documentation.

From signals to SLOs

An SLI is a measurement of what users experience; an SLO is the target you promise. For an API, two SLIs cover most needs:

• Availability: the share of requests that do not end in a server error, for example proxy/response_count with response_code not in the 5xx range, divided by all responses.
• Latency: a percentile of proxy/latencies, such as the 95th percentile.

Targets are yours to choose; an illustrative pair (assumed): 99.9 percent availability and p95 latency under 600 ms over 30 days. A 99.9 percent target leaves an error budget of 0.1 percent: across a 30-day month of 43,200 minutes that is about 43 minutes of full outage-equivalent. Alert on how fast you burn the budget, not on single blips. The general method is covered in observability and SLOs.

Apigee's own documentation gives sample alert shapes, which you should treat as starting points to tune rather than standards: an error ratio such as 5 percent of 500 responses sustained for 5 minutes, and a p95 latency threshold evaluated over a 10-minute window.

Use cases

1. Normalising backend errors. Three backends each fail differently: one returns HTML, one returns a plain-text stack trace, one returns its own JSON. The FaultRules above map them all to the same three-field contract, so client SDKs have one error parser. The rule of thumb: the gateway owns the client-facing contract; backends own their internals.

2. Alerting on 5xx and latency. Build two alert policies in Cloud Monitoring: one on the ratio of 5xx responses to all responses from proxy/response_count, one on p95 proxy/latencies. Route both to the on-call channel. Use the fault_source filter to see whether failures come from the proxy or the target when you investigate.

3. Tracing a customer's failing call. A customer reports an error and quotes the correlationId from the response body. You search Cloud Logging for that id and read the one line written by MessageLogging: proxy, path, status, fault name. If you need more, start a filtered Debug session for the next attempt.

4. Reporting by customer tier. The business wants to know whether paid tiers get worse latency. DataCapture records the tier per call; a custom report groups latency by that dimension.

In practice: cleaning up the errors of an orders API

The situation. A mobile app calls an Orders API through Apigee. When the orders backend restarts, the app shows a blank error screen, because the proxy returns whatever the backend sent: sometimes an HTML page, sometimes a Java stack trace that includes an internal hostname. Support receives screenshots but cannot link them to log lines. Nobody is paged until customers complain.

Starting architecture.

What changes.

1. Add the three AssignMessage error policies and the shared correlation-header policy.
2. Add the TargetEndpoint FaultRules and the DefaultFaultRule shown in Step 2, and the ProxyEndpoint client-raised rule.
3. Attach ML-Access-Log in the PostClientFlow.
4. Start a Debug session with a filter, stop the backend, and confirm the fault.name the trace shows, adding it to the condition if it differs.
5. Create two Cloud Monitoring alerts (error ratio, p95 latency) with a notification channel.

Result. The client now receives the same JSON shape for every failure, with a 503 status for an outage, a correlationId it can show on the error screen, and no internal detail. Support pastes the id into Cloud Logging and finds the matching line. The on-call engineer is notified when the error ratio stays above the agreed threshold for the agreed window, before the first complaint arrives.

Architect's notes

• One contract, owned centrally. Put the error format in a small set of AssignMessage policies reused by every endpoint. Copying JSON bodies into every proxy guarantees drift. If many proxies need the same behaviour, consider a shared flow, and check the shared-flow documentation for how it is invoked from your rule.
• Be careful with continueOnError. It hides failures. Use it only for optional work, and make sure something still records the failure (a log line or the .failed variable).
• Alert on symptoms, investigate with details. Alerts should reflect what users feel (error ratio, latency), not every internal fault code.
• Mind cardinality and cost. High-cardinality log names and custom dimensions are expensive and hard to query. MessageLogging and DataCapture are Extensible policies, so their use may have cost implications under your licence.
• Privacy by default. Logs and debug data are copies of customer data. Decide what is logged, mask the rest, and remember that API Monitoring data can contain query parameters and user agents.
• Know the limits of each tool. Analytics is for trends, API Monitoring for near-real-time, Debug for short sampled sessions. None replaces your own logs for a single call.

Common mistakes

• No DefaultFaultRule. Unmatched errors fall through to Apigee's default message. Add a generic rule with a safe body.
• Conditions on steps instead of rules. A step condition does not choose the FaultRule. Put the fault.name test on the rule.
• Using RaiseFault as a step inside a FaultRule. Use AssignMessage to shape the response there.
• Letting the DefaultFaultRule overwrite your RaiseFault response. Add a FaultRule that matches fault.name = "RaiseFault" so the DefaultFaultRule is not the one that runs, then verify in a trace.
• Leaking internals. Return a generic message and log the detail instead. Check trace output of the final response, not only the policy list.
• Matching on message text. Match on fault.name or the error code.
• Debugging with sensitive data on. Configure the debug mask and private. variables before opening a session on production traffic.
• Relying on a Debug session for history. It is short-lived and limited to a number of requests. Use logs for history.
• Expecting instant data. Analytics can lag, and alerts have a delay. Do not conclude "no data" too early.

Key takeaways

• A failing proxy leaves the normal flow permanently; FaultRules, the DefaultFaultRule or Apigee's default message decide the response.
• Put the fault.name condition on the FaultRule; step conditions only decide whether a step runs.
• Backend 4xx and 5xx responses are errors by default; success.codes changes that.
• Design one JSON error contract with a stable code, a correlation id and no internal detail.
• Use AssignMessage in FaultRules and RaiseFault for your own checks, and keep them from overwriting each other.
• Log in the PostClientFlow with MessageLogging so success and failure are both recorded.
• Debug sessions are short, sampled and must be masked; logs and analytics tell the long story.
• Define SLIs first, alert on error ratio and latency, and pick the tool that matches the question.

Practice exercises

Starter (about 30 minutes). In an evaluation or sandbox organization, create a proxy in front of a public echo or sample service. Add a RaiseFault behind a condition that fires when a header is missing, and call it with and without the header. Open a Debug session first and find the step that raised the fault.

Intermediate (about 2 hours). Build the full error layer from this lesson on a proxy with a target server you can switch off. Steps: write AM-Error-503, AM-Error-500 and AM-Correlation-Header; add the FaultRules and DefaultFaultRule; start a filtered Debug session; stop the backend; read the fault.name in the trace; adjust the condition until the client receives your JSON. Then add ML-Access-Log in the PostClientFlow and find the log line by correlation id.

Advanced (half a day). Add observability end to end. Create a DataCapture collector and report for a business dimension, set a debug mask for a sensitive field and prove it hides the value in a new session, create Cloud Monitoring alerts for 5xx ratio and p95 latency, and write a one-page SLO with an error budget calculation. Trigger a failure and walk the decision flowchart from the alert to the root cause.

Related reading

• Apigee Fundamentals: Architecture and Request Flow — where the flows and endpoints you attach rules to sit
• Building Apigee API Proxies: Flows, Endpoints, Policies — the proxy structure and policy basics used here
• Securing APIs in Apigee: OAuth, JWT, mTLS, Threat Protection — the security policies whose faults you will map
• Traffic Management in Apigee: Quotas, Spike Arrest, Caching — the 429 faults and the metrics behind them
• Apigee Environments, Revisions and Deployment — how the same proxy and logging reach each environment
• Capstone: Design, Build and Ship an Orders API on Apigee — uses this error contract in a full build
• Observability and SLOs — the general method behind SLIs, SLOs and burn alerts
• API gateway and service mesh — where a gateway sits in the larger design
• Apigee API management for microservices — an applied tour of Apigee in a service landscape

Official documentation

• Handling faults
• RaiseFault policy
• AssignMessage policy
• MessageLogging policy
• DataCapture policy and Collecting custom data
• Using Debug and Masking and hiding data
• API Monitoring overview, Setting up alerts and notifications and Fault codes
• Understand Apigee observability and Apigee API Analytics overview
• Flow variables reference and Endpoint properties reference
• Apigee audit logging