Composite scenario, not a real company. Assembled from common industry patterns; every number is illustrative and its assumptions are listed at the end.
This case belongs to Multi-agent and event-driven AI architecture.
Context
A mid-sized property insurer receives claims through a portal and a call centre. Each submission becomes a "claim submitted" event on an event bus. A group of 12 workers reads the events; for each claim a model reads the narrative and form fields and returns a recommended route: fast track, standard adjuster or specialist. The worker stores the result and a downstream service creates the adjuster task and sends the customer an acknowledgement. A triage call takes about 3 seconds. After a storm the portal sees about 2 claims per second. The model only recommends a route; coverage decisions stay with adjusters.
Problem statement
After a routine deployment restarted all 12 workers, adjusters found duplicate tasks and some customers received two acknowledgement emails. A claim withdrawn within minutes of submission still got an adjuster task. Both have a design cause. Workers committed their position in the event stream after every 100 claims but created tasks and emails after every claim, so a restart replayed up to 99 claims per partition: up to 12 x 99 = 1,188 duplicate runs and side effects. And events had no key, so "submitted" and "withdrawn" for one claim could land on different partitions and be handled in either order.
The goal: no duplicate side effects from a worker restart, at most 12 repeated model calls (one in flight per partition), and 95% of claims triaged within 60 seconds of submission at peak.
Constraints
- A claim must never be lost, even if the model is down or returns garbage.
- The case-management system creates a task for every request and does not deduplicate.
- Customers get one acknowledgement.
- Injury claims reach a human adjuster whatever the model's confidence.
Why the traditional approach fell short
A rules router on form fields misroutes the long tail: a customer who selects "other" and writes "water through the ceiling after the storm, tenant has moved out" looks like a trivial claim. Manual triage by clerks is accurate but the queue grows exactly when claims arrive in bursts. A classic classifier needs labelled history and fails on event types it has not seen. The model reads the narrative, which is why it helps; the delivery machinery (queues, restarts, retries) is ordinary distributed-systems work that the model does nothing to fix.
Options considered
| Option | Idea | Verdict |
|---|---|---|
| A. Rules router (non-AI) | Route by claim type, amount and postcode | Kept as the fallback when the model is unavailable: claims go to the standard queue by these rules |
| B. Synchronous call from intake | The portal calls the triage service and waits | Rejected: submission time becomes 3 s plus the model's tail latency, and a model outage blocks claim submission |
| C. Event bus with idempotent workers | Keyed events, results and a deduplication record written in one transaction, offsets committed after the work, a dead-letter topic | Accepted |
| D. Rely on the broker's exactly-once transactions alone | Kafka transactions write records and offsets atomically | Rejected as the only defence: the transaction covers what is written to Kafka, not the model call, the email or the task creation |
The default is C with A behind it. For a few hundred claims a day, a job table polled by one worker is a fair, simpler choice.
Solution architecture
Figure 1. Claim triage components with keyed events, deduplication and a dead-letter path
How to read this: solid arrows are the normal path left to right, the bottom branch is where a claim goes when triage keeps failing, and each box that writes names its deduplication rule.
Figure 2. Runtime path of one claim event, including a replay
How to read this: the outer alt separates a replay from new work, the inner alt separates a valid answer from a dead-lettered one, and the last three lines are the separate delivery path from the outbox.
How AI solved it, step by step
One claim, end to end.
- The portal publishes "claim submitted" with the claim id as the event key, so all events for that claim go to one partition, in order.
- A worker reads the event and looks up its event id in the Document store. If found, it is a replay: skip to step 7.
- The worker sends the narrative and fields to the Model gateway, which returns JSON: a route, an injury flag and a confidence.
- The worker validates the JSON against its schema. Invalid output is retried once; a second failure goes to the dead-letter topic with the reason (step 8).
- The worker applies rules the model does not own: an injury flag or a confidence below 0.8 sends the claim to the adjuster queue, never to fast track.
- In one database transaction the worker writes the deduplication record, the result and an outbox row.
- The worker commits its offset.
- A dead-lettered claim lands in the manual triage queue and raises an alert, so a person sees it.
- The outbox relay publishes "claim triaged" with its own event id. The Tool server skips an id it has applied, otherwise creates the task and sends one email.
The decision that stops duplicates is in step 2 and is made by code.
Under the hood
Kafka's documentation describes the pieces this design depends on. Events with the same key are written to the same partition, and a consumer reads a partition's events in the order they were written. Each partition is consumed by exactly one consumer in a consumer group at a time, and a consumer's position is a single integer, the offset. If a consumer processes messages and then saves its position, a crash in between means some messages are seen again (at-least-once); if it saves first, a crash can lose messages (at-most-once). Kafka guarantees at-least-once delivery by default. This case chooses at-least-once on purpose, because losing a claim is worse than seeing one twice, and makes the consumer idempotent: the deduplication record shares a transaction with the result, so a replayed event finds it and does nothing. The arithmetic: committing every 100 claims leaves up to 99 uncommitted per partition, 12 x 99 = 1,188; committing after each event leaves one in flight per partition, 12. See Under the hood: Partitions, consumer groups and offsets.
What went wrong / failure modes
- Crash between the model call and the transaction. The event replays and the model is called again: at most 12 repeated calls per restart, accepted.
- Crash inside the Tool server. It can fail after creating the task or sending the email and before recording the event id. The case-management system does not deduplicate, so one repeated task or email per such crash remains possible; the design removes the replay of up to 1,188 claims, not this last window.
- A poison event. A malformed narrative makes the model return invalid JSON twice. Mitigation: the dead-letter topic, an alert when it is non-empty for 15 minutes, and a manual queue. A dead-letter topic nobody reads loses claims.
- A parked event lets later ones overtake it. Events carry a per-claim sequence number; a worker that sees a gap routes the claim to the manual queue.
- The deduplication record expires. Records are kept 14 days, longer than the topic's 7-day retention, so a replay older than the records cannot occur.
Results
These are illustrative targets, not measurements. Duplicate side effects from worker replays per restart fall from a worst case of 12 x 99 = 1,188 to 0, with at most 12 repeated model calls; the Tool server crash window above is separate. Capacity is 12 workers x (1 claim / 3 s) = 4 claims per second against a peak of 2, which is 50% utilisation, leaving room in the 60-second deadline for a restart or a slow model response.
Tradeoffs and when NOT to use this approach
- The deduplication write adds a database dependency to every claim, and committing per event adds broker traffic; at 2 claims per second both are small.
- Ordering holds only within one partition, never across all claims.
- At low volume, a job table and one worker may beat a bus.
- Never let the model route injury or coverage-sensitive claims without a human.
Design rules extracted
- Key events by the entity whose order matters, so its events share a partition.
- Commit the offset only after the work and its deduplication record are durable, and expect replays.
- Give every side effect a key derived from the event, and make each consumer skip keys it has applied.
- Send failing events to a dead-letter topic that a person owns, with an alert.
- Let the model recommend; let rules decide anything that cannot be undone.
Assumptions
- Peak load of 2 claims per second, 3 s per call and 12 partitions with one worker each are chosen for illustration; utilisation is 2 x 3 / 12 = 0.5.
- Offsets are committed every 100 claims in the first design: 99 uncommitted per partition in the worst case, 12 x 99 = 1,188.
- The confidence threshold of 0.8, the 2 model attempts, the 15-minute alert, the 14-day deduplication retention and the 7-day topic retention are invented settings.
- The Kafka statements are from the Apache Kafka documentation, read on 2026-10-21; the insurer, its systems and all figures are invented.
Last verified: 2026-10-22