This is a reference architecture, not a client case study. It shows how I would design an event-driven payment-events platform on AWS. It contains no client data and no measured results, and every number below is an example to adjust from your own load tests.
The problem it solves
A monolith that handles payments, orders and notifications in one deployable fails as one unit. A spike in one area slows everything, one team's release blocks every other team, and a bug in a low-value feature can take down a high-value one. For a payment system that runs around the clock, the design goals are:
- Isolation. A failure or a traffic spike in one service must not cascade into the others.
- Independent delivery. Teams deploy their own services without coordinating a release.
- No lost events. Every payment event is processed at least once, in order per payment, and failures are never silently dropped.
- Zero-downtime releases. Deployments must not interrupt a system that never closes.
Architecture
Producers publish a fact ("payment completed") to a topic and move on. Each consumer reads the topic at its own pace, so adding a new consumer (say, a fraud model) needs no change to the producer.
Key design decisions
Partition by the business key
Kafka guarantees order only within a partition. Using the payment id as the message key sends every event for one payment to the same partition, so a consumer sees "authorised" before "captured" before "refunded". Choose the partition count from your target throughput and the number of consumers you want running in parallel, and remember that raising it later changes which partition each key maps to.
Idempotent consumers
Delivery is at-least-once: a consumer can see the same event twice after a retry or a rebalance. Every consumer must make repeats harmless. Store the event id (or a business idempotency key) in the same database transaction as the effect, and skip events already seen. For the ledger, that means a unique constraint on the event id.
Retries and dead-letter queues
A message that fails should not block the partition behind it, and it should not vanish. Retry a few times with backoff, then publish it to a dead-letter topic together with the error, the original topic, partition and offset. An alert tells the on-call engineer, and a replay tool sends fixed messages back through the normal path, without touching the main flow.
Stable consumer groups on Kubernetes
Rolling deployments restart pods, and each restart can trigger a consumer-group rebalance that pauses processing. Static group membership (group.instance.id) with a sensible session timeout lets a restarting pod reclaim its partitions instead of reshuffling the whole group. Also give pods a graceful shutdown so they commit offsets before they stop.
The database is a shared limit
Many small services each holding a connection pool can exhaust PostgreSQL's connection limit long before CPU runs out. Put a pooler such as PgBouncer in front of the database (transaction pooling), cap each service's pool and size the total deliberately.
Deployment pipeline
A canary sends a small share of traffic to the new version first and watches error rate and latency against the old version. Example gates: roll back automatically if the error rate rises above a set threshold or p95 latency exceeds a multiple of the baseline. Pick the numbers from your own service-level objectives, and make the rollback path as well tested as the rollout. Infrastructure (the cluster, the topics, the database) is defined as code with Terraform, so environments match.
Where this design is stressed
| Pressure | What breaks | The usual fix |
|---|---|---|
| Burst traffic | One consumer group falls behind | Enough partitions, consumers matched to partitions, lag alerts |
| Many services | Database connections run out | A pooler, per-service caps |
| Frequent deploys | Rebalances pause consumers | Static membership, graceful shutdown |
| A poison message | One bad event blocks a partition | Bounded retries, then the dead-letter topic |
| A slow dependency | Consumers back up | Timeouts, circuit breakers, bulkheads |
Observability
Track consumer lag per group and partition, processing time, dead-letter volume, error rate, and the age of the oldest unprocessed event. Trace a payment across services with a correlation id carried in the message headers. Alert on symptoms users feel (events not processed within the target time), not only on CPU.
Trade-offs to expect
- Eventual consistency. A payment is "completed" before every consumer has reacted. Screens and APIs that read derived data must tolerate a short delay.
- Operational weight. Kafka, topics, schemas and dead-letter handling need an owner, tooling and runbooks.
- Ordering is per key, not global. Design so nothing needs a global order.
- Schema evolution. Events are contracts. Use a schema registry and compatible changes only.