System Design Case Study: Chat System

• Messages: partition key conv_id, clustered by seq descending. One conversation lives on one partition, so history reads are sequential and fast. Row: (conv_id, seq, msg_id, sender_id, body, created_at). De-duplication by clientMsgId needs its own lookup, for example a small table or cache keyed by (conv_id, client_msg_id) with a TTL, since the main table is ordered by seq.
• Conversations and membership: conversation(conv_id, type, ...) and members(conv_id, user_id) with a reverse index user_conversations(user_id, conv_id, last_seq, last_read_seq) to render the inbox.
• Sessions: which gateway holds which device connection: session(user_id -> [device_id, gateway_id]). This is small, hot and ephemeral, so it fits an in-memory store with TTLs. It is refreshed by heartbeats.
• Offline queue / sync cursor: each device remembers the last seq it has per conversation. Sync is "give me everything after seq N", so no separate per-user queue is strictly required.

How to read it: MESSAGES is keyed by conv_id and seq, which is exactly the history read path. SESSIONS is the only ephemeral table and can be rebuilt from heartbeats.

High-level design

Gateways are mostly dumb: terminate TLS, authenticate, keep the socket, and forward frames to the message service. The message service validates, assigns the sequence number, stores the message, then asks the router to deliver. The router looks up recipients' sessions and hands the message to the right gateways. If a recipient has no live session, it goes to the push service.

The sender's ack means "stored", not "delivered". Delivery and read are separate, later events. That distinction is what turns "sent, delivered, read" ticks into honest status.

How to read it: only the server ack moves a message out of Sending, and later states are driven by cursors, not by individual receipts.

Reference deployment topology

The deployed system has a stateful gateway tier in front of the services (the message service is partitioned by conversation, the router and push service are stateless), with the log and the session store in the data tier.

How to read it: gateways are the only tier that holds connections, so they are the only tier whose deploys need draining. The message service is partitioned by conversation so one owner assigns each seq.

Deep dive 1: connection gateways and presence

Why a stateful tier. WebSockets are long-lived, so the server holds per-connection state. That breaks the usual "any server can handle any request" rule for this tier. Behind the gateways, durable state lives in the stores, and the only per-process state is partition ownership in the message service.

Load balancing connections. An L4 or L7 load balancer spreads new connections, and because a connection lasts hours, balancing happens at connect time. Deploys and scale-down must drain gracefully: tell clients to reconnect elsewhere, with random jitter and a capped reconnect rate. A deploy that drops 100,000 connections per node, all reconnecting at once, creates a reconnect storm that can take down the next tier.

Session lookup. On connect, the gateway writes (user, device) -> gateway into the session store with a TTL, renewed by heartbeat. The router queries it to find where to send. If the entry is stale (the gateway died), delivery fails fast and the message falls back to storage and push. Clients reconnect and sync by sequence number, so nothing is lost.

Presence. The client sends a heartbeat every N seconds, the gateway refreshes a key with a TTL slightly above N, and expiry means offline. Do not broadcast every change to every contact: let clients subscribe only to conversations on screen, and debounce flapping with a short delay before announcing "offline". Presence is the classic place to accept eventual consistency and approximate values.

Deep dive 2: ordering, acknowledgements and delivery guarantees

Ordering. Clocks across devices disagree, so timestamps cannot define order (see IDs, clocks and ordering). Instead, route all writes for a conversation through a single logical owner that assigns an increasing seq. In practice, partition the message service by conv_id (hash the conversation ID to pick a partition) and process each partition's messages serially. Clients then sort by seq. Ordering is guaranteed per conversation, not globally, and that is all users need.

If the owner fails, another node takes over the partition. Its leadership must be established safely (see consensus and leader election) so two nodes never assign the same seq. A message log such as Kafka gives per-partition ordering for free if you key by conv_id (see event streaming), though you still have to design the consumer side.

Delivery semantics. The honest guarantee is at-least-once delivery with idempotent receipt. The server retries until the client acknowledges, and the client discards duplicates using msgId or seq. The effect of "exactly once" comes from that combination, not from the network itself. Clients also detect gaps: if they have seq 480 and receive 482, they request 481 from history.

The reliable-send loop from the sender:

1. Generate clientMsgId, show the message locally as "sending".
2. Send over the socket. If no ack within a timeout, retry with the same clientMsgId.
3. The server de-duplicates by (conv_id, clientMsgId), returns the original seq.
4. Show "sent" on ack.

Delivered and read receipts. Treat them as cumulative cursors: "delivered up to seq 481" and "read up to seq 479" per user per conversation. A cursor is smaller and cheaper than one receipt per message and survives lost packets, because a later cursor covers earlier ones.

Multi-device. Each device holds its own socket and its own sync cursor. Delivery fans out to all of a user's devices. Read cursors are synchronised so reading on the phone clears the badge on the laptop.

Deep dive 3: offline delivery and group chat

Offline users. The message is already in the store, so nothing needs a special queue. The router sees no live session and calls the push service, which sends a notification through the platform push providers (APNs for Apple devices, FCM for Android). Push is best-effort: it can be delayed, dropped or throttled, and it should carry minimal content (or just a "new message" hint) because it passes through third parties. When the app opens, it connects and syncs from its last seq. Push is a nudge, the log is the truth.

Group chat fan-out. A group message is stored once (one row in the conversation's partition) and delivered to each member. For small groups, the router fans out to each member's sessions. The scale lever is the same one as in feeds: push to online members now, let offline members catch up by sync. For larger groups, route per gateway instead of per member: send one copy to each gateway that has any member, and let the gateway fan out locally. That cuts internal traffic from "members" to "gateways involved".

The choice of group delivery path follows from room size and how clustered the members are.

How to read it: the thresholds are tuning knobs, not rules. The last step is the same for every branch because the stored log is the guarantee.

Membership changes are events in the same log, so every device agrees on who was in the group at which seq. That matters for what a new member can read (history policy) and for encryption keys.

Encryption options (high level)

• In transit: TLS on every connection. This is the baseline.
• At rest: encrypt storage volumes or fields with keys held in a key management service.
• End-to-end encryption: messages are encrypted on the sender's device and only decrypted on recipients' devices, so the server stores ciphertext it cannot read. Well-known designs use per-device keys and a ratcheting protocol to limit how much a stolen key exposes. The consequences are real: the server cannot search or moderate message content, server-side history backfill to a new device needs a client-assisted key transfer, groups need key distribution on every membership change, and multi-device multiplies keys. Do not design your own cryptography: use an established, audited protocol and library.

Bottlenecks and scaling

• Gateway memory and file descriptors. Tune per-connection buffers, use event-driven networking, and scale out horizontally. Keep gateway logic thin so a deploy is cheap.
• Session store. About 20 million entries are small but hot. Shard by user_id and replicate.
• Message store write rate. About 140,000 writes per second at peak is comfortable for a sharded, write-optimised store with append-heavy access. Partition by conversation, and watch for hot conversations (a huge, busy group).
• Router fan-out. Batch per gateway. Keep the router stateless.
• Reconnect storms. Jittered exponential backoff on the client, connection rate limits at the edge, and gradual drains.

Failure modes

Trade-offs worth stating

Interview cheat sheet

In practice: operating the reference design

Assume a team of twelve runs this on Kubernetes across three zones. The gateways use a plain WebSocket server behind an L4 load balancer, the message store is a Cassandra-style wide-column database, sessions and presence live in Redis, and Prometheus and Grafana provide monitoring.

SLOs and dashboards

Capacity-review cadence

• Weekly: connections per gateway against the 100,000 assumption, memory per connection, partition size outliers, hot conversations.
• Monthly: storage growth against about 440 TB per year before replication, tombstone and compaction health.
• Quarterly: a reconnect-storm drill (drain one zone and measure recovery time), and a load test at twice the assumed peak of 140,000 messages per second.
• Before major events: holidays and large broadcasts shift message rates, so re-run the estimate.

Runbook

Decision record: per-conversation sequencer

Cost view

Gateways (memory per connection), storage and cross-zone traffic dominate. Message payloads are small, so per-message compute is rarely the issue.

Architect's notes

• Measure memory per connection first. The gateway count follows from it, and an unmeasured estimate can be off by a wide margin.
• Make clients part of the design review. Backoff, jitter and sync logic live in the apps, and a bad release can cause the outage.
• Pin down the history policy early. Who can read history, for how long, and what a new group member sees affects storage, encryption and legal obligations.
• Do not let the gateway become a service. Every feature added to the stateful tier makes deploys scarier.
• Defer end-to-end encryption debates with facts. List which features (search, moderation, backfill) depend on server-readable content before deciding.
• Page the right team. Gateways, the store and the push integration usually have different owners, so write the runbook entry points per owner.

Common mistakes

• Ordering by client timestamps. Clock skew reorders messages. Assign a server-side sequence per conversation.
• Acking before persisting. A crash then loses an acknowledged message. Ack only after the durable write.
• Treating push notifications as the delivery channel. Push is best-effort. The stored log and sync are the guarantee.
• Broadcasting presence to every contact. Traffic explodes. Subscribe only to what is visible and debounce.
• Putting business logic and state in the gateways. Deploys then drop connections and lose data. Keep gateways thin and state external.
• No client backoff or jitter. After an outage every client reconnects at once. Use exponential backoff with jitter.
• Claiming end-to-end encryption while the server reads content. Users are misled. Be precise about which parties can read messages.

Key takeaways

• Store, then deliver. The durable per-conversation log is the truth, and connections are a fast lane.
• Connections size the gateway tier. Memory per connection, not message rate, is the limit.
• Order with a sequence number. One owner per conversation assigns seq. Clocks cannot be trusted.
• At-least-once plus idempotent receipt. Retries and de-duplication give the effect users expect.
• Cumulative cursors beat per-message receipts. They are cheaper and survive loss.
• Push is a nudge. The app syncs from its last sequence number when it opens.
• Presence is approximate by design. Use heartbeats, TTLs, subscriptions and debouncing.
• End-to-end encryption has costs. Search, moderation and recovery change when the server cannot read content.

Practice exercises

Exercise 1 — Starter (30 min): For the illustrative numbers in this lesson, recompute the gateway count if each node holds 50,000 connections instead of 100,000 and you want 30% headroom. Then compute messages per second and yearly storage if users send 80 messages per day instead of 40.

Exercise 2 — Intermediate (2 hours): Write the sync protocol precisely: what a device sends when it reconnects after two hours offline, how the server responds for a one-to-one chat and for a group with 5,000 unread messages across 12 conversations, how gaps are detected, and how you bound the response size. Draw the sequence diagram.

Exercise 3 — Advanced (half day): Design the failover of the sequencer for a conversation partition. Describe how a new owner is chosen, how you prevent two owners from assigning the same seq (consider a fencing token checked by the store), what a client sees during the switchover, and how you test it with fault injection.

Related reading

• Message queues - acknowledgements, retries and idempotent consumers that underpin reliable delivery.
• Event streaming - per-partition ordering for a conversation log.
• IDs, clocks and ordering - why sequence numbers beat timestamps.
• Consensus and leader election - safe ownership of a conversation partition and fencing tokens.
• System Design Case Study: News Feed - fan-out trade-offs that also apply to large group channels.