Multi-Agent and Event-Driven AI Architecture

How to read this: time runs downward, each partition is a separate lifeline with its own offsets, and "Group offsets" is the stored committed position that survives a Worker restart.

Figure 6. Six partitions shared by four Workers

How to read this: each arrow is one partition owned by one Worker; no partition has two owners in the same group.

Worked numbers. Assignment depends on the configured strategy; the range assignor, one of the strategies Kafka lists, divides partitions by consumers and, in its words, "the first few consumers will have one extra partition" when the division is uneven. For 6 partitions and 4 Workers, 6 = 4 x 1 + 2, so Workers 0 and 1 own two partitions each and Workers 2 and 3 own one, as in Figure 6. With 8 Workers, Workers 6 and 7 own none. Lag: the Kafka tooling shows CURRENT-OFFSET, LOG-END-OFFSET and LAG per partition, where the operations documentation describes lag as "how far behind the end of the log" a group is; reading it as log end minus committed offset is this page's interpretation of those columns. Assume three partitions with log-end offsets 1200, 1180 and 1260 and committed offsets 1150, 1180 and 1100. Lags are 50, 0 and 160, total 210. If events arrive at 30 per second and the group consumes 45 per second, the backlog drains at 45 - 30 = 15 per second, so 210 / 15 = 14 seconds.

What this means for architects

• Parallelism is capped by the partition count. Extra Workers sit idle. Because model calls are slow, plan partitions as concurrency, not storage (see the worked example).
• The key is the ordering promise. Choose the smallest entity whose events must stay in order (an order id, not a customer id). A skewed key piles work onto one partition.
• Slowness shows up as lag, not errors. Alert on lag per partition and its trend; total lag can look small while one partition is far behind.
• A missing offset has a default. auto.offset.reset defaults to latest and applies when the stored offset no longer exists, so a group down longer than the retention can silently skip to the end. Prefer none (it throws) for work you cannot lose.
• Rebalances pause work. A Worker that does not call poll within max.poll.interval.ms (default 300000 ms, Kafka 4.3 documentation as of 2026-10-21) is treated as failed, which matters when one batch holds many LLM calls.

Under the hood: At-least-once and effectively-once processing

Intuition

Delivery guarantees are statements about a crash at the worst moment. A consumer reads a record, does the work and saves its place. Save first and a crash loses work; work first and a crash repeats work. "Once" needs the work and the saved place to change together, or repeating the work to be harmless.

Mechanism

The Kafka design documentation describes both orders. Reading, saving the position, then processing is "at-most-once" because after a consumer failure messages "may not be processed". Reading, processing, then saving the position gives at-least-once: a crash after processing but before saving makes the replacement's first messages ones that were already processed. The same section says Kafka "guarantees at-least-once delivery by default", and that idempotent updates (overwriting a record by primary key) make the duplicates harmless.

On the producer side, the Kafka documentation says the idempotent producer means retries no longer introduce duplicates into the log; the producer documentation limits that to "a single session" and says enable.idempotence defaults to true from Kafka 3.0 and requires acks=all, retries above 0 and at most five in-flight requests per connection. Transactions let a producer write to several partitions atomically, and a consumer with isolation.level=read_committed sees only committed transactional messages. For consume-transform-produce between Kafka topics, the producer's sendOffsetsToTransaction makes the consumed offsets count as committed only if the transaction commits. The design documentation says exactly-once delivery is supported inside Kafka Streams and through the transactional producer with read-committed consumers, and that for an external system it "generally requires cooperation" from that system. A refund posted over HTTP is an external system.

Figure 7. A Worker restart under each offset-commit order

How to read this: the alt block shows the same crash twice; the top branch repeats work that dedupe can absorb, the bottom branch loses work that nothing can recover.

Worked numbers. A batch is 5 records (offsets 100 to 104) and the crash comes after 3 are processed. Commit after processing: the replacement re-reads all 5; with a dedupe check 3 repeats are skipped and 2 effects applied. Commit before processing: it starts at 105, so 105 - 103 = 2 records are lost. Duplicates after a crash are bounded by the records handled since the last commit: at most one batch of max.poll.records if you commit after every batch.

"Effectively-once" here means at-least-once delivery plus an idempotent effect, so the effect lands once though the message may arrive twice. The design documentation discusses two ways to build it: store the consumer's position in the same transaction as the output, or make the write idempotent.

What this means for architects

• Default to commit-after-processing plus dedupe. Losing a claim or a refund is worse than repeating one, and dedupe is a table with a unique key.
• Dedupe before the model call. A duplicate that reaches the Model gateway pays for tokens and may return different text; store the result against the event id and reuse it. Price the repeats with the cost estimator.
• Name the scope of "exactly-once". For a Kafka-to-Kafka stage it is real; for the payments call it is your idempotency design.
• Auto-commit is risky with slow work. enable.auto.commit defaults to true with a 5000 ms interval, and the consumer documentation warns that the committed offset can run ahead of the work if a poll's records are not all consumed first. Turn it off for agent Workers.

Under the hood: Idempotency keys

Intuition

An idempotency key is a receipt number the caller attaches to a request. Show the same receipt twice and the server returns the first outcome instead of acting again. That makes the retry a timeout forces on you safe, because a timeout does not say whether the first attempt happened.

Mechanism

Stripe documents one concrete design for its v1 API. The client generates the key (V4 UUIDs are suggested; keys are up to 255 characters) and sends it in the Idempotency-Key header. Stripe saves "the status code and body of the first request made with a given idempotency key after endpoint execution begins", and retries with the same key return the saved response, "including 500 errors". The layer compares incoming parameters with the original request and errors if they differ. Results are saved only once execution begins, so a request that fails validation, or conflicts with a concurrent one, is not saved and can be retried. Keys may be removed once they are at least 24 hours old, and a key reused after it has been removed starts a new request. Only POST needs a key; GET and DELETE are idempotent by definition. The page states that API v2 behaves differently: retries do not repeat successful operations, and Stripe completes failed or partial work where it can.

There is also an IETF specification effort. The datatracker page shows draft-ietf-httpapi-idempotency-key-header as an Expired Internet-Draft of the httpapi working group, state "Expired & archived", revision 07 dated 2025-10-15, with the notice that it is no longer active. It is therefore not a standard, and the draft's own boilerplate says Internet-Drafts are not appropriate as reference material except as work in progress. Its text is still useful as a design: the header is a String that must be unique and must not be reused with a different payload, the draft specifies HTTP 400 when a required key is missing, 422 when the same key arrives with a different payload, and 409 when the original request is still being processed, and a resource that expires keys should publish its policy.

Figure 8. Life of one idempotency key record

How to read this: each box is the state of the stored record for one key, and each arrow says what a request or the clock does to it; the 409 and 422 codes are the ones the expired draft specifies.

Figure 9. A retried tool call with the same key

How to read this: steps 1 to 6 are the first attempt, steps 7 to 10 the retry, and the last note marks the crash window that a key store alone cannot close.

Worked numbers. Suppose the key store keeps keys for 24 hours and the topic keeps events for 7 days. A Worker paused for 2 days and then replaying reaches events whose keys have expired, so an effect can run again. Key retention must cover the longest period over which one event can be redelivered or replayed; 24 hours is less than the 48-hour pause, so extend it or store the key with the business record.

What this means for architects

• Derive the key from the event, not the attempt. A key made fresh on each retry defeats the purpose; use the event id or a hash of the business identity.
• Honour the key where the effect happens. Reserve it atomically (a unique constraint) and write it in the same transaction as the effect, or pass it to the system that performs the effect, as Figure 9's last note says.
• Return the saved response. It is also the cleanest answer to LLM non-determinism: a second delivery receives the first outcome and does not ask the model again.
• A key has an expiry. Publish the retention, decide what a late duplicate does, and treat 409 and 422 as expected outcomes.

Under the hood: The outbox pattern

Intuition

A service that changes its database and then publishes an event makes two writes to two systems, and a crash can fall between them. Publish first and the event can announce something that never committed; commit first and the event can be lost. The outbox makes the second write part of the first.

Mechanism

The microservices.io pattern states the problem as "How to atomically update the database and send messages to a message broker?" The solution: the service stores each message in an outbox table, in the same local transaction that updates its business entities, and a separate message relay reads the outbox and publishes to the broker. The page names two ways to build the relay, a polling publisher and transaction log tailing, and notes that two-phase commit is not an option when the database or broker does not support it. Its stated consequence is that messages are sent "if and only if the database transaction commits", in the order the application sent them. It also states the price: the relay may publish a message more than once (for example after a crash between publishing and recording that it did), so consumers must be idempotent.

Figure 10. Outbox with a polling relay and a duplicate publish

How to read this: steps 1 and 2 are the single local transaction, steps 3 to 5 are the first publish, steps 6 to 9 the retry after the crash, and the final note is the consumer's protection.

Worked numbers. A polling relay with interval T adds delay between commit and publish. If commits fall uniformly within the cycle, the mean added delay is T / 2 and the worst case is T plus query and publish time. With T = 500 ms the mean is 250 ms: fine beside a model call of seconds, not within a budget of a few hundred milliseconds, where the page's other option, log tailing, fits better.

What this means for architects

• Use an outbox when state and event must agree. A refund case and its refund.proposed event must not disagree. If losing an event is acceptable (a metrics ping), skip the cost.
• The outbox row id is the event id. It travels in the header and is the dedupe key for consumers and the idempotency key for the tool call.
• Order comes from the key. Publish rows per aggregate in order and key events by aggregate id, so Kafka keeps them ordered within a partition.
• Monitor the relay. Alert on the age of the oldest unsent row and purge sent rows on a schedule.

Under the hood: Sagas and compensating actions

Intuition

A refund that touches payments, inventory and a carrier cannot be one database transaction, because different services own them. A saga replaces the single rollback with a sequence of local transactions and, for each, a business action that undoes it. If step 3 fails, undo step 2, then step 1.

Mechanism

The idea is from Hector Garcia-Molina and Kenneth Salem, "Sagas" (SIGMOD 1987; also Princeton technical report TR-070-87, January 1987). The report's abstract says a long-lived transaction is a saga if it can be written as a sequence of transactions that can be interleaved with other transactions, that the database system guarantees either all of a saga's transactions complete or compensating transactions undo the partial work, and that the motivation is that long-lived transactions hold database resources for long periods and delay shorter ones. This page relies on the abstract.

The microservices.io saga page states the modern form: "a reliably coordinated sequence of local transactions", coordinated either by choreography (each service publishes events that trigger the next) or by orchestration (a coordinator tells participants what to execute). It lists the costs: no automatic rollback, so a developer "must design compensating transactions that explicitly undo changes"; no isolation, so concurrent sagas and transactions can cause data anomalies; the need to update a database and publish a message atomically (which the outbox addresses); and a client that needs a way to learn the outcome.

Figure 11. States of an orchestrated saga

How to read this: forward steps are T1 to T3, compensations are C1 and C2 run in reverse order, and the two terminal states are Completed and Aborted.

Figure 12. Saga failure at step 3 and its compensations

How to read this: steps 1 to 6 are forward steps, the note marks the decision, steps 7 to 10 undo the work in reverse order, and each call carries its own idempotency key.

Worked numbers. With three steps, each succeeding independently with probability p = 0.98, the saga completes with probability 0.98^3 = 0.941192, so 1 - 0.941192 = 0.058808 of sagas fail somewhere. A failure at step k runs k - 1 compensations, and the probability of first failing at step k is p^(k-1) x (1 - p): 0.02, 0.0196 and 0.019208 for k = 1, 2, 3. Expected compensations per saga: 0 x 0.02 + 1 x 0.0196 + 2 x 0.019208 = 0.058016. About 6 sagas in 100 fail somewhere, and 0.0196 + 0.019208 = 0.038808, about 4 in 100, fail after step 1 and so need at least one compensation. At volume the compensation path runs every day.

What this means for architects

• Order steps by reversibility. Easy-to-undo steps first (holds, reservations), irreversible ones last (a notification cannot be unsent). This is a design rule, not something the sources prescribe.
• Compensations must be idempotent and retried to completion. There is no further rollback if an undo fails, so give it an escalation path to a person.
• Design for the intermediate state. With no isolation, another request can see a held amount; mark such records pending so readers and agents treat them as provisional.
• Prefer orchestration for agent flows. One Orchestrator keeps the saga state in one place and makes the human decision (Figure 4) a state in that record; choreography spreads the logic across services.

Under the hood: Dead-letter queues and ordering

Intuition

A poison message fails every time. Retry it forever and the line behind it stops; skip it and the work is lost or, worse, later messages for the same entity are applied before it. A dead-letter queue is where the message goes so the line moves, with a person or tool responsible for what lands there.

Mechanism

Amazon SQS documents the standard form. Dead-letter queues are targets that source queues use for messages "that are not processed successfully", useful for debugging. A redrive policy sets maxReceiveCount, the number of times a consumer can receive a message before it moves to the dead-letter queue, and the page advises setting it high enough to allow sufficient retries. For standard queues the enqueue timestamp is unchanged when a message moves, so the best practice is a dead-letter retention period longer than the source queue's. Ordering is where the guidance turns cautious: "Don't use a dead-letter queue with a FIFO queue if you don't want to break the exact order of messages or operations". FIFO ordering holds within a message group ID, and no further messages from that group are returned until the received message is deleted or becomes visible again, so one failing message blocks its group while other groups continue.

Kafka's design differs. Ordering is per partition, and a consumer either commits past a failing record or does not. The consumer documentation read for this page describes no dead-letter feature, while Kafka Connect has one: its user guide says connectors fail fast by default (errors.tolerance=none) and routes failed records to a topic when errors.deadletterqueue.topic.name is set. For your own consumers, the usual design is to publish the failing record, with its key and error context, to a dead-letter topic and then commit.

Figure 13. Life of one message with retries and a dead-letter queue

How to read this: each box is a state of one message, and every path ends in Committed, so the line always moves.

Figure 14. A poison message on one partition, with per-key quarantine

How to read this: the loop is the retry policy, the note is the decision, and the two parked records keep the order of events for claim-5 while claim-9 goes on.

Worked numbers. Five retries that wait 1, 2, 4, 8 and 16 seconds stall a partition for 1 + 2 + 4 + 8 + 16 = 31 seconds, not counting the attempts themselves. At 20 events per second over 48 partitions, a partition receives 20 / 48 = 0.4167 per second, so about 31 x 0.4167 = 12.9, roughly 13 events queue behind the poison message. If you park it and carry on, any of those sharing key claim-5 would be applied ahead of the parked record unless you quarantine the key, as Figure 14 does.

What this means for architects

• Classify failures first. A rate limit or timeout is transient, so retry with backoff; a schema or validation failure never succeeds, so park it at once.
• Choose the ordering promise per flow. For a saga or a ledger, block or quarantine by key; for independent items such as documents, park and move on.
• A dead-letter queue needs an owner. Alarm on depth and age, keep a runbook and a replay tool; replay is safe only because consumers are idempotent.
• It is sensitive data. Parked events hold prompts, customer text and model output; give the queue the access control and retention of the system it came from.

Decision framework

1. If the task cannot be split into independent parallel subtasks, or its parts share context, then use one agent or a fixed workflow, because the research post names shared context and many dependencies as a poor fit and reports about 15 times a chat's tokens; revisit when you can measure that the extra cost buys more value.
2. If work can wait seconds and arrives in bursts, then put an Event bus between Orchestrator and Workers, because it absorbs bursts and lets each side scale and fail independently; revisit when the user must wait inline.
3. If delivery can repeat (it can, with at-least-once), then commit offsets after processing and make the effect idempotent with a key derived from the event id, because a lost refund is worse than a repeated one; revisit only for Kafka-to-Kafka stages that use transactions.
4. If a state change and an event must agree, then write both in one local transaction through an outbox, because a separate publish can lose or invent events; revisit if your broker and database support an atomic write.
5. If an action spans services and some steps cannot be undone, then use a saga with idempotent compensations and irreversible steps last, because there is no distributed rollback; revisit when a step needs a human decision first.
6. If one message can block a partition, then define retry limits, a dead-letter queue and a per-key ordering rule, because unbounded retry stalls every key on that partition; revisit when dead-letter depth stays above zero.

Comparison of the main options

Failure scenarios to design for

1. A Worker crashes mid-batch. Defined behaviour: the offset is committed only after the batch, the dedupe table skips applied events, and a recorded result is reused instead of calling the model again.
2. Slow processing triggers a rebalance. A batch of 50 events handled one after another at 2 s each takes 100 s, inside the 300 s default max.poll.interval.ms; at 7 s each it takes 350 s and the consumer is treated as failed. Defined behaviour: keep batch size x worst-case time well below the interval, lower max.poll.records, and expect duplicates anyway.
3. A poison message blocks a partition. Defined behaviour: a retry limit with backoff, a dead-letter queue with error context, per-key quarantine when order matters, and an alarm on dead-letter depth.
4. The outbox relay publishes twice. Defined behaviour: consumers dedupe on event id, and an alert watches the age of the oldest unsent row.
5. A saga compensation fails. Defined behaviour: compensations are idempotent and retried; after a set number of attempts the saga moves to a human queue with its full state.
6. An approval never arrives. Defined behaviour: the request expires at its deadline, emits an expiry event and compensates anything already held.
7. A consumer returns after its offset expired. With auto.offset.reset=latest it would skip silently. Defined behaviour: none or earliest for work you cannot lose, an alert on the exception, and dedupe for the replay.
8. An orchestrator fans out too far. The research post reports early agents spawning 50 subagents for simple queries. Defined behaviour: effort rules by task type, a cap on subagents and a token budget per job.

Worked example

Size the Worker pool for the returns pipeline and see what a restart costs.

Assumptions: illustrative figures. Peak load is 20 events per second. One agent step (model call plus tool call) takes 2.0 s. Each partition is processed one event at a time to keep per-key order. The topic has 48 partitions on 4 Worker instances (12 each). max.poll.records is 50 and the offset is committed after each batch. Events spread evenly. All Workers are down for 180 s.

1. Concurrency needed. By Little's law (in-flight work = arrival rate x time in the system), 20 x 2.0 = 40 events are in flight, so at least 40 partitions must be busy.
2. Capacity. 48 partitions at 2.0 s per event process 48 / 2.0 = 24 events per second; utilisation is 20 / 24 = 83%.
3. Backlog after the outage. 20 x 180 = 3,600 events, 3,600 / 48 = 75 per partition. It drains at 24 - 20 = 4 per second, so 3,600 / 4 = 900 s, 15 minutes. With 40 partitions, capacity is 40 / 2.0 = 20 per second, equal to arrivals, and the backlog never drains.
4. Restart duplicates. Each instance holds at most one uncommitted batch, so at most 4 x 50 = 200 events are redelivered. Without dedupe that is 200 x 2.0 = 400 worker-seconds and 200 extra model calls, about 400 / 48 = 8.3 s of whole-fleet time. With the check before the model call, repeats cost lookups.

The design rule: capacity is partitions divided by step time, and 4 events per second of headroom turns a 3-minute outage into a 15-minute recovery.

Code sample: an idempotent consumer that commits after processing

This Java sketch uses the Kafka consumer API and a SQL database that supports ON CONFLICT DO NOTHING (PostgreSQL and SQLite do). It compiles against kafka-clients 3.9.0; the consumer is configured with enable.auto.commit=false.

import java.nio.charset.StandardCharsets;
import java.sql.*;
import java.time.Duration;
import java.util.List;
import org.apache.kafka.clients.consumer.*;

public class ClaimWorker {
    // consumer config: group.id=claim-workers, enable.auto.commit=false, String deserializers
    static void run(KafkaConsumer<String, String> consumer, Connection db) throws Exception {
        db.setAutoCommit(false);
        consumer.subscribe(List.of("claim-events"));
        while (true) {
            ConsumerRecords<String, String> batch = consumer.poll(Duration.ofSeconds(1));
            for (ConsumerRecord<String, String> r : batch) {
                String eventId = new String(
                        r.headers().lastHeader("event-id").value(), StandardCharsets.UTF_8);
                try (PreparedStatement seen = db.prepareStatement(
                        "INSERT INTO processed_events(event_id) VALUES (?) ON CONFLICT DO NOTHING")) {
                    seen.setString(1, eventId);
                    if (seen.executeUpdate() == 1) {   // first time this event id is seen
                        applyEffect(db, r);            // same transaction as the dedupe row
                    }
                    db.commit();                       // effect and dedupe row become durable together
                } catch (Exception e) {
                    db.rollback();
                    throw e;                           // offset not committed: a restarted consumer gets the batch again
                }
            }
            if (!batch.isEmpty()) consumer.commitSync(); // offsets move only after the whole batch
        }
    }

    static void applyEffect(Connection db, ConsumerRecord<String, String> r) throws SQLException {
        // write the triage result for r.key() using db
    }
}

Why it is correct. event_id must be the table's primary key or carry a unique constraint, or ON CONFLICT has nothing to conflict with and nothing is deduplicated. The dedupe row and the business write share one database transaction, so both exist or neither does. A crash before db.commit() leaves neither, and the redelivered event is applied. A crash after db.commit() but before commitSync() leaves both, and the redelivered event inserts nothing (executeUpdate() returns 0), so the effect is skipped. After an exception the redelivery happens when a new consumer starts from the committed offset; if the loop caught the exception and kept polling, the consumer's in-memory position would already be past the batch, so it would have to seek back first. commitSync() with no arguments commits the offsets returned by the last poll, correct here because every record of the batch was handled first. The sample assumes each record has an event-id header (a missing one would throw and redeliver the batch, so a real consumer should park it) and that applyEffect calls no external system inside the transaction. It leaves out retry limits, the dead-letter step and rebalance handling (a commitSync after the consumer has left the group can fail), and it must finish each batch within max.poll.interval.ms.

Common mistakes

• Treating "exactly-once" as a property of the whole system. It has a defined scope in the Kafka documentation; the side effect outside Kafka is yours to make idempotent.
• Committing the offset before the work. It is the simplest way to lose events silently, and nothing alerts you.
• Generating a new idempotency key on every retry. The server cannot recognise the repeat, so every attempt runs.
• Publishing the event in a separate step after the database commit. A crash between them loses the event, and nobody notices until a customer asks.
• Writing compensations that are not idempotent, or that cannot fail safely. The compensation path runs about 4 times in 100 in the worked example; it needs the same care as the forward path.
• Adding a dead-letter queue and not watching it. It becomes a silent store of lost work, and of customer data.

Try it

Time: 30 to 40 minutes. Pick an asynchronous flow you know or invent a small one (a ticket-routing agent that posts to a chat tool).

1. Draw its sequence diagram with an Event bus, one Worker restart and one human approval, using Figure 3 as the template.
2. Write down the partition key and the ordering promise it gives.
3. State your commit order and, for a batch size you choose, compute the duplicates one crash can cause and the events a commit-before-work design could lose.
4. Define the event id, where the dedupe record lives and how long it is kept compared with the topic's retention.
5. Pick two steps that can fail halfway and write their compensations, in reverse order, with idempotency keys.
6. Define what happens to a message that fails five times, who is alerted and what order guarantee you keep.

Done when every diagram has a figure title, an accessible title and description and a "how to read this" line, and you can point to the crash case, the duplicate case and the poison-message case in your design and name the defined behaviour for each.

Key takeaways

• More than one agent pays off only for valuable, parallelisable tasks that exceed one context; Anthropic reports about 15 times a chat's tokens (about 4 times for a single agent) and names shared-context and dependent tasks as a poor fit.
• Kafka orders events per partition and gives each partition to one consumer in a group; the key is your ordering promise and the partition count your parallelism ceiling.
• Delivery is at-least-once by default: commit after processing and make the effect idempotent. Exactly-once has a defined scope that stops at Kafka's edge.
• An idempotency key comes from the event, is honoured where the effect happens and outlives the longest replay.
• The outbox makes a state change and its event atomic, at the price of duplicate publishes that consumers absorb.
• A saga replaces rollback with idempotent compensations in reverse order, and that path runs more often than intuition suggests.
• A dead-letter queue works only if someone owns it, and parking a message breaks per-key order unless the key is quarantined.

Self-check

Further reading

• Apache Kafka documentation, design: message delivery semantics and the KafkaConsumer and KafkaProducer javadoc.
• Anthropic Engineering, Multi-agent research system post and Building effective agents.
• Garcia-Molina and Salem, "Sagas" (Princeton TR-070-87, abstract), with the Saga and Transactional outbox patterns on microservices.io.
• Stripe, Idempotent requests, and the IETF draft The Idempotency-Key HTTP Header Field (an expired Internet-Draft, not a standard).
• AWS, Using dead-letter queues in Amazon SQS and FIFO queue delivery logic.

Last verified: 2026-10-22

Next

Continue to Evaluating AI systems: metrics, judges and gates, which shows how to test the agents and pipelines built here.