System Design: Building a Notification Service for 50 Million Users

@Service
class TemplateRenderer {

    private final TemplateEngine templateEngine;  // Thymeleaf or Pebble

    public RenderedNotification render(NotificationEvent event, Channel channel) {
        Template template = templateRepository.getTemplate(
            event.type(), channel, event.locale(), event.templateVariant()
        );
        return new RenderedNotification(
            templateEngine.process(template.content(), event.variables()),
            channel
        );
    }
}

Templates are cached in-process (Caffeine, 10-minute TTL) in each worker. Template cache invalidation sends a Kafka event to all worker instances — they evict and reload on next render.

Kafka: The Central Backbone

One topic, multiple consumer groups — one per channel. This is the fan-out mechanism:

notification-events (topic)
├── email-notification-group    → Email Workers (12 instances)
├── sms-notification-group      → SMS Workers (8 instances)
├── push-notification-group     → Push Workers (10 instances)
└── inapp-notification-group    → In-App Workers (6 instances)

Each consumer group processes at its own pace. Email might be slower (due to ESP rate limits) without backing up SMS delivery. The channels are completely decoupled by the queue.

Partition count is the key scaling decision. At 8,333 messages/second peak and aiming for 1,000 messages/second per partition (leaving headroom), we need at minimum 9 partitions. Use 24 partitions — round number divisible by 1, 2, 3, 4, 6, 8, 12 consumers, giving flexibility to scale workers without rebalancing.

Channel Workers

Each worker follows the same structure regardless of channel:

@KafkaListener(
    topics = "notification-events",
    groupId = "email-notification-group",
    containerFactory = "emailKafkaListenerContainerFactory"
)
public class EmailNotificationWorker {

    private final TemplateRenderer templateRenderer;
    private final EmailProvider emailProvider;
    private final DeliveryLogService deliveryLog;

    @KafkaHandler
    public void handle(NotificationEvent event, Acknowledgment ack) {
        String notificationId = event.id();
        try {
            RenderedNotification rendered = templateRenderer.render(event, Channel.EMAIL);
            EmailResult result = emailProvider.send(rendered, event.recipientEmail());

            deliveryLog.record(notificationId, Channel.EMAIL, DeliveryStatus.SENT,
                Map.of("messageId", result.providerId(), "provider", "SES"));
            ack.acknowledge();

        } catch (RetryableException e) {
            // Don't ack — Kafka will redeliver to this consumer group
            log.warn("[{}] Retryable failure on email send, will redeliver: {}", notificationId, e.getMessage());
            // Don't ack — message will be redelivered after max.poll.interval.ms

        } catch (PermanentException e) {
            // Ack the message (remove from main topic) and send to DLQ
            deliveryLog.record(notificationId, Channel.EMAIL, DeliveryStatus.FAILED,
                Map.of("reason", e.getMessage()));
            dlqProducer.send("notification-dlq", event.recipientId(), new DlqEvent(event, e));
            ack.acknowledge();
        }
    }
}

Retryable vs. permanent exceptions:

Retry and DLQ Flow

The DLQ processor runs on a 15-minute schedule for most failures. For transactional notifications (password reset, fraud alert), it runs every 60 seconds — delivery SLA matters more for these.

Delivery Log (TimescaleDB)

Delivery tracking generates time-series data. One row per notification per channel, written at every state transition:

CREATE TABLE notification_delivery_log (
    notification_id  UUID        NOT NULL,
    channel          VARCHAR(20) NOT NULL,
    recipient_id     UUID        NOT NULL,
    status           VARCHAR(30) NOT NULL,   -- SENT, DELIVERED, OPENED, FAILED, ABANDONED
    provider_msg_id  VARCHAR(100),           -- ESP/SMS provider's message ID for callbacks
    error_code       VARCHAR(50),
    attempt_count    INTEGER DEFAULT 1,
    metadata         JSONB,
    occurred_at      TIMESTAMPTZ NOT NULL DEFAULT NOW(),

    PRIMARY KEY (notification_id, channel, occurred_at)
);

SELECT create_hypertable('notification_delivery_log', 'occurred_at');

-- Retention: auto-drop partitions older than 90 days
SELECT add_retention_policy('notification_delivery_log', INTERVAL '90 days');

TimescaleDB's hypertable partitioning handles the write volume (10,000+ inserts/second at peak) and makes time-range queries efficient: "show me all failed email notifications in the last 24 hours" hits one or two partitions instead of scanning 36 TB.

Database Schema

The Trade-offs That Matter

1. Fan-out on write vs. fan-out on read

For campaign sends (10 million recipients), you have two strategies:

• Fan-out on write: When the campaign is submitted, create one notification record per recipient immediately, then process from the queue. Pros: consistent delivery tracking, simple worker logic. Cons: 10M database writes before a single notification is sent.
• Fan-out on read: Store one campaign record. Workers resolve the recipient list at delivery time. Pros: minimal writes upfront. Cons: recipient list must be re-resolved at send time, harder to track per-recipient status.

Recommendation: Fan-out on write, but asynchronously. The campaign scheduler writes notification records in batches of 1,000 to Kafka directly, not to PostgreSQL first. Workers consume from Kafka and persist delivery status. The notification record exists in the delivery log, not as a pre-created row in the notifications table.

2. At-least-once vs. exactly-once delivery

Exactly-once delivery with Kafka requires Kafka transactions and idempotent producers — significant operational complexity. At-least-once with deduplication in the API layer (the deduplicationKey) gives 99.9% of the benefit with 10% of the complexity.

The remaining 0.1% case: network partition causes the Kafka producer to retry after a successful write, sending the event twice. The deduplication cache (Redis with 24-hour TTL) catches this if the deduplicationKey is set. Callers that don't set a deduplicationKey accept at-least-once semantics explicitly.

3. Notification ordering

In-app notifications need to appear in chronological order for a given user. Email and SMS do not — the user sees them in their email client's sort order anyway.

The partition-by-recipientId strategy guarantees ordering within a partition for a given consumer group. For in-app, we additionally attach a sequence number from a per-user Redis counter when writing to the in-app feed. This handles the edge case where two events for the same user land in the same partition but the in-app worker processes them out of order (rare, but possible with concurrent workers).

4. Template rendering location

Render at the API layer (before enqueue) or at the worker layer (after dequeue)?

Rendering at the worker layer is better because:

• Template data can be stale at render time rather than at enqueue time (template variables fetched fresh)
• Failed renders become visible at delivery time with full context for debugging
• Template updates between enqueue and delivery pick up the new template (important for campaigns scheduled days ahead)

The cost: the template variables must be carried in the Kafka event payload, not fetched again at render time. Include all needed data in the event at enqueue time.

Scaling to 500,000 Notifications Per Minute

At 500,000/min (8,333/sec), with 24 partitions, each partition handles ~347 events/sec. Email workers (slowest channel due to ESP rate limits) use 12 partitions. SMS, push, and in-app share the remaining 12 partitions across their consumer groups — each consumer group reads all 24 partitions independently.

Bottleneck analysis at 500K/min:

Email is the bottleneck. The solution isn't more partitions — it's multiple ESP accounts (SendGrid primary, SES fallback), each with their own rate limit, and distributing sends across accounts in a round-robin pattern within the email worker.

What Goes Wrong in Production (And How to Handle It)

1. Preference service is down

Fail open (send the notification, skip preference check) for transactional notifications (password reset, fraud alert). Fail closed (drop the notification) for marketing content. The distinction is in the notification type's config.

2. ESP (email service provider) outage

The circuit breaker opens on the primary ESP. Workers automatically route to the secondary ESP. The DLQ processor drains backed-up notifications when the primary recovers.

3. Template rendering fails for one user

The worker sends the notification to the DLQ with error_type: TEMPLATE_RENDER_ERROR. The DLQ processor alerts engineering. The rest of the campaign continues — one bad template variable doesn't stop 9,999,999 other sends.

4. Kafka consumer lag spikes

Set an alert when consumer group lag exceeds 100,000 messages (about 12 seconds of peak throughput). Root cause is usually: a slow downstream (ESP rate-limited), workers processing too slowly (increase parallelism), or a Kafka partition imbalance (rebalance the consumer group).

Spring Boot Wiring

@Configuration
class NotificationKafkaConfig {

    @Bean
    KafkaListenerContainerFactory<ConcurrentMessageListenerContainer<String, NotificationEvent>>
    emailKafkaListenerContainerFactory(ConsumerFactory<String, NotificationEvent> cf) {
        var factory = new ConcurrentKafkaListenerContainerFactory<String, NotificationEvent>();
        factory.setConsumerFactory(cf);
        factory.setConcurrency(12);  // 12 threads = 12 partitions for email group
        factory.getContainerProperties().setAckMode(AckMode.MANUAL_IMMEDIATE);
        factory.getContainerProperties().setPollTimeout(3_000);

        // Dead letter publishing: after 3 failed attempts, route to DLQ
        var recoverer = new DeadLetterPublishingRecoverer(kafkaTemplate,
            (rec, ex) -> new TopicPartition("notification-dlq", rec.partition()));
        var errorHandler = new DefaultErrorHandler(recoverer,
            new FixedBackOff(1_000L, 3L));
        factory.setCommonErrorHandler(errorHandler);

        return factory;
    }
}

Interview Cheat Sheet

If you're preparing for system design interviews, here's what interviewers are listening for on this question:

The question interviewers are waiting for you to ask: "What's the SLA difference between transactional and marketing notifications?" Answering this before they prompt it shows you understand the business constraints that drive architecture, not just the technology.

Notification systems are where distributed systems complexity meets business requirements that change constantly (new channels, new preference types, new regulatory requirements). The architecture above is designed to absorb those changes: add a new channel by adding a consumer group, add a new preference type by extending the Redis hash, add a new region by adding a Kafka cluster in that region.

The design is the easy part. The hard part is the retries.

Avaneesh Yadav is Engineering Manager at HashedIn by Deloitte, designing enterprise systems at scale. He writes about system design and AI architecture at buildingai.in.