Public case (sourced). Based on Anthropic Engineering's published post "A postmortem of three recent issues" (17 September 2025); every fact below is attributed to it.
This case belongs to LLMOps and observability for AI systems. Where the post is silent, this page says so. Sections that interpret the post are marked as this page's reading, not Anthropic's claims.
Context
- Anthropic's post says that between August and early September 2025, three infrastructure bugs intermittently degraded Claude's response quality.
- Users began reporting degraded responses in early August. By late August the growing frequency and persistence of reports prompted an investigation.
- The post states: "We never reduce model quality due to demand, time of day, or server load." It says the problems were due to infrastructure bugs alone.
- The post does not say how many users reported problems. It describes no monitoring beyond its validation steps (see Options considered).
Problem statement
The post describes three bugs, and says their overlap made diagnosis particularly challenging.
| Bug | What the post says | Dates in the post |
|---|---|---|
| 1. Context window routing error | Some Sonnet 4 requests were misrouted to servers configured for the upcoming 1M token context window. Initially 0.8% of requests. A load balancing change on August 29 increased the traffic. 16% of Sonnet 4 requests at the worst impacted hour, August 31 | Started August 5. Fix deployed September 4 |
| 2. Output corruption | A misconfiguration on Claude API TPU servers caused errors during token generation, such as Thai or Chinese characters in response to English prompts. Opus 4.1 and Opus 4 on August 25-28, Sonnet 4 on August 25 to September 2. Third-party platforms were not affected | Rolled back September 2 |
| 3. Approximate top-k miscompilation | Code deployed to improve token selection triggered a latent bug in the XLA:TPU compiler. Confirmed on Haiku 3.5. The post believes a subset of Sonnet 4 and Opus 3 on the Claude API could also have been affected | Deployed August 25. Haiku 3.5 rolled back September 4, Opus 3 on September 12 |
Constraints
As the post states them:
- Each bug produced different symptoms on different platforms at different rates.
- Internal privacy and security controls limit how and when engineers can access user interactions with Claude.
- The post could not reproduce bug 3 on Sonnet 4, and rolled that model back as a precaution.
Why the traditional approach fell short
The post says: "The evaluations we ran simply didn't capture the degradation users were reporting." It adds that this was in part because Claude often recovers well from isolated mistakes, and that the team "relied too heavily on noisy evaluations".
It also says the team did not immediately connect the August 29 rise in reports to an otherwise standard load balancing change.
Options considered
This page lists the responses the post describes.
| Option | What the post says |
|---|---|
| A. Pre-release validation plus user reports (simplest, no new system) | The status before the incident. The post says validation relied on benchmarks, safety evaluations, performance metrics, spot checks and small "canary" groups, and that its evaluations missed the degradation. It asks users to keep sending feedback by /bug or the thumbs-down button |
| B. More sensitive evaluations | The post says it developed evaluations that can more reliably differentiate between working and broken implementations |
| C. Quality evaluations on production | The post says: "we will run them continuously on true production systems" |
| D. Debugging tools for community feedback | The post says it will build tooling to debug such feedback without sacrificing user privacy |
Solution architecture
The post publishes no architecture diagram. The figures below are this page's reading aid, drawn from the dates and changes the post lists.
Figure 1. The detection changes the post lists (this page's reading)
How to read this: three signals (production evaluations, feedback, deploy-time tests) feed the same investigation; the post names each as a change, not as a diagram.
Figure 2. The timeline in the post
How to read this: the bugs began days or weeks before the investigation, and reports were the main signal until late August.
How AI solved it, step by step
Step through the incident as the post tells it, then through what the post says changes.
Under the hood
The post describes two mechanisms.
- Bug 1, sticky routing. Once a request was served by an incorrect server, follow-ups were likely to be served by the same one. The post says this is why some users were affected more severely.
- Bug 3, precision. The post says the models compute next-token probabilities in bf16 and the XLA compiler can convert some operations to fp32. This caused the highest probability token to sometimes disappear from consideration, only for certain batch sizes and model configurations. The fix was exact top-k with enhanced precision.
This page's reading: a failure that depends on route, batch size or platform shows up only when results are cut by those dimensions, and a per-request record of route and version makes that cut possible. The post does not mention traces or logs. See Under the hood: The trace and span data model and Under the hood: Shadow, canary and A/B releases.
What went wrong / failure modes
| Failure (per the post) | What the post says was done or planned |
|---|---|
| Sticky routing sent follow-ups to the same wrong server | Routing logic fixed so short and long context requests go to the correct pools |
| A misconfiguration produced rare tokens such as unexpected characters | Rolled back, and detection tests added to deployment |
| A latent compiler bug exposed by a token-selection change | Rolled back, a compiler fix pursued with the XLA:TPU team, exact top-k with enhanced precision |
| Evaluations missed the degradation | More sensitive evaluations, run continuously on production |
| Privacy controls limited access to problem interactions | Tooling to debug feedback without sacrificing privacy |
| Different symptoms per platform made diagnosis harder | No step aimed only at this; the post says its changes apply "across all platforms where we serve Claude" |
Results
Only what the post states.
| Item | Per the post |
|---|---|
| Bug 1 peak | 16% of Sonnet 4 requests at the worst hour, August 31 |
| Claude Code users | Approximately 30% of those who made requests had at least one message misrouted |
| Other platforms, bug 1 | Amazon Bedrock peak 0.18% of Sonnet 4 requests from August 12. Google Cloud Vertex AI less than 0.0004% between August 27 and September 16 |
| Routing fix | Deployed September 4. Complete on first-party and Vertex AI by September 16, on Bedrock by September 18 |
| Not stated | Number of users who reported, how the percentages were measured, how long each bug took to resolve beyond the rollout dates |
Tradeoffs and when NOT to use this approach
- One organisation's account. The post describes one provider's infrastructure. A team calling a hosted API cannot instrument that layer and can only detect from its own side.
- Do not copy the remedies. Your failure signatures differ. Start from the failures you have seen.
- Privacy and debugging pull apart. The post names the tension but not the final design.
Design rules extracted
This page's reading of the post, as general rules.
- Evaluate quality on production traffic, per platform, not only before release.
- Do not rely on a single noisy evaluation.
- Turn each failure signature you have seen, such as unexpected characters, into a deploy-time detection test.
- Report impact by user and route, not only by share of requests: the post gives 0.8% and 16% of requests but roughly 30% of Claude Code users (different measures and windows).
- Build privacy-preserving ways to inspect bad interactions before you need them.
Sources
- Anthropic Engineering, "A postmortem of three recent issues" (17 September 2025): https://www.anthropic.com/engineering/a-postmortem-of-three-recent-issues. Accessed 2026-10-22 with several focused reads.
Last verified: 2026-10-22