QuickCart Stage 7: Search, Notifications, Uploads and Analytics

Channels differ. Push is instant but needs the app and permission, and tokens go stale: use it for order progress. Email holds long content but is slow to be read and prone to bounces: use it for receipts. SMS reaches any phone but costs per message: use it as a fallback for time-critical items. A template is the message with placeholders, stored per type, channel and language ("Your order {{order_id}} is {{minutes}} minutes away"). The event carries ids and values; the notification service owns the words.

Preferences, quiet hours and two kinds of message

Shoppers choose channels per type, and the preference check is a lookup before any send. Transactional messages (order confirmed, payment failed) honour the channel choice, but a shopper cannot opt out of everything. Time-critical ones (courier at the door) use a fallback and ignore quiet hours. Marketing needs consent and obeys quiet hours (for example none from 21:00 to 08:00 local time, illustrative), which need the shopper's time zone; a held message is rescheduled, not dropped. Consent rules differ by region, and a legal owner decides the wording.

De-duplication and rate limits

Events arrive at least once, so a restart can replay one. A de-duplication key (order, event type, channel) makes the second delivery a no-op. Redis can claim a key atomically with SET and its NX (only if absent) and EX (expiry in seconds) options:

redis-cli SET notif:ord_9917:out_for_delivery:push 1 NX EX 86400
# OK       first claim: send the push
redis-cli SET notif:ord_9917:out_for_delivery:push 1 NX EX 86400
# (nil)    already claimed: skip

One limit: if a worker claims the key and crashes before sending, the message is lost. Either record the key after a successful send and accept a rare duplicate, or store a status (claimed, sent) and let a sweeper retry stale claims. For a delivery update a rare duplicate beats a missed message.

Per-user rate limits protect the shopper from a bug that sends forty pushes: a token bucket per user and channel (see rate limiting) caps marketing at a few messages a day (illustrative). Updates about one order are collapsed, so "10 minutes away" replaces "15 minutes away".

Provider failure, fallbacks and delivery receipts

Providers send the actual push, email and SMS, and they fail: timeouts, throttling, outages. The pipeline copes in layers: retry with exponential backoff and jitter, a bounded number of times; a circuit breaker per provider that stops sending for a while after repeated failures (resilience patterns); a secondary provider for email and SMS (push usually has none, since each platform has its own gateway); and channel fallback, so when push fails permanently (invalid token) the service tries SMS if the type allows. A shopper who disabled SMS is not texted because push failed, except for a short, declared list of critical types.

A delivery receipt is the provider's later report on a message it accepted; "accepted" only means it took the request. Not every provider or carrier returns every status, so a missing receipt means "unknown", not "failed". Receipts delete invalid tokens, suppress hard bounces and fill the delivery log.

Figure 4. One notification, from event to phone.

How to read this: the service decides who, which channel and whether to send before it touches a provider. The receipt arrives later and triggers the fallback.

Every notification ends as delivered, suppressed or failed, and each outcome is recorded for the support question "why did I not get a message?". Transactional and marketing messages use separate queues, so a campaign backlog cannot delay "your courier is here".

File upload and media processing

Store owners photograph products. This is the one pipeline whose input is a large, untrusted file from the internet.

Pre-signed uploads straight to object storage

Photo bytes never pass through the app servers. The API hands the store app a pre-signed URL: a time-limited URL, signed with the server's credentials, that lets its holder perform one operation on one object key. In Amazon S3, according to its documentation, such a URL allows an upload without the uploader having credentials, is limited by the permissions of its creator, replaces any existing object with the same key, and can be valid for up to seven days when generated by the SDKs or CLI. QuickCart uses 15 minutes (illustrative).

1. The store app asks the API to start an upload for product p_18342, with an idempotency key. The API checks the store owns the product and creates an uploads row as pending.
2. The API returns a pre-signed PUT URL for a random key such as incoming/up_7b2e (never a name the client chose) and the content type it signed for.
3. The app uploads straight to object storage, using multipart upload for large files so a dropped connection resumes.
4. Storage emits an "object created" event for the incoming/ prefix to a queue; workers scan, validate, transform and publish, and the app polls the API for status.

A pre-signed PUT URL does not enforce a size range by itself; an S3 POST policy can carry a content-length-range condition, and the scan worker rejects oversize or wrong-type files anyway. Never trust the client's claimed type or size.

The processing pipeline: scan, validate, resize, address

Scan. The worker runs the file through an antivirus engine first (ClamAV, an open-source toolkit with a scanning daemon, is one example). Infected or unparseable files move to a quarantine prefix and the upload becomes rejected.

Validate by decoding. Decode with an image library and read the real format and size from the bytes, not the extension. Reject images over a pixel limit (a small file can decode to an enormous image) and strip metadata (EXIF can carry GPS coordinates).

Resize. Produce the sizes the apps request, for example 200, 600 and 1,200 pixels wide (illustrative), in a format such as WebP where clients support it. The pool scales on queue age. Video transcoding is the same pattern with long jobs (video streaming case study); v1 handles images only.

Content addressing. A content-addressed store names an object by a cryptographic hash of its bytes, such as SHA-256. The worker hashes the validated original (never a client-supplied hash) and writes outputs under media/<hash>/600.webp. Two stores uploading the same manufacturer photo share one object. New bytes mean a new URL, so the CDN can cache for a very long time with no invalidation (CDN and object storage). The cost is deletion: delete only when a reference count reaches zero, and purge the CDN for a legal takedown.

Publish. The worker sets the hash and sizes on the product, which emits a catalogue event that the indexer (Figure 2) picks up. Pipelines chain through events, not calls.

Figure 5. One store photo, from the phone to the CDN.

How to read this: the bytes never pass through QuickCart's API. The queue and workers do the slow work; the catalogue update at the end makes the photo visible.

Failure and retry

S3's documentation says its event notifications are designed to be delivered at least once, so a worker may see an event twice. Workers are idempotent: output keys derive from the hash, and status only moves forward. A crashed worker's message returns to the queue when the visibility timeout expires. A file that always fails goes to a dead-letter queue after a few retries and the upload is marked failed. A sweeper deletes pending rows whose URLs expired. One trap the S3 documentation warns about: writing to the bucket that triggers the notification can loop, so the trigger applies only to the incoming/ prefix.

Stream processing and analytics pipelines

Stream processing: windows, event time and late data

Stream processing computes continuously over events as they arrive. The first idea is the window: you cannot sum an endless stream, so you sum a slice. As Apache Flink's documentation names them, tumbling windows are fixed and do not overlap (orders per minute), sliding windows overlap (orders in the last 10 minutes, updated each minute) and session windows close after a gap of inactivity.

The second idea is two clocks. Event time is when the thing happened, as stamped by the phone; processing time is when the system sees it. They diverge whenever there is delay: a courier's phone in a lift sends events a minute late. Processing-time windows give different answers on replay; event-time windows give the same answer when the same events are replayed, at the cost of waiting.

How long to wait? A watermark is the system's claim that event time has reached some point, so earlier events should have arrived; one common heuristic keeps it a fixed bound behind the latest event time seen. An event arriving after the watermark has passed its window is late. In Apache Flink, at the time of writing, late events are dropped by default unless the window has an allowed lateness, which lets it fire again with a corrected result.

A worked example: one-minute tumbling windows of OrderPlaced events by event time, with a watermark 30 seconds behind the latest event time (illustrative).

Event 5 is a real order. Dropping it leaves the dashboard one short; allowing lateness emits a corrected 3. A live view can accept "about right, quickly"; the warehouse recomputes later.

Exactly-once effect through idempotent sinks

After a failure some events are processed again, so an honest "exactly once" guarantee concerns the effect. Kafka's documentation describes an idempotent producer, so retried sends do not create duplicate log entries; transactions, which write to several partitions atomically; and a read_committed consumer setting that returns only committed transactional messages. Together they give exactly-once for a read-process-write cycle between Kafka topics, including in Kafka Streams; an external destination must cooperate, for example by storing the consumer's offset with its output.

Anything outside the platform (a database, an email, a payment API) needs an idempotent sink: applying the same result twice leaves the same state. The simplest form writes a replacing result under its identity.

-- One row per store per minute window.
CREATE TABLE orders_per_minute (
  window_start timestamptz NOT NULL,
  store_id     bigint      NOT NULL,
  orders       integer     NOT NULL,
  PRIMARY KEY (window_start, store_id)
);

-- The stream job writes the full window result, not an increment.
INSERT INTO orders_per_minute (window_start, store_id, orders)
VALUES ('2026-11-07 12:00:00+00', 17, 3)
ON CONFLICT (window_start, store_id)
DO UPDATE SET orders = EXCLUDED.orders;

Run it once or five times and the row says 3; a corrected window result overwrites it with 4. Write "the answer for this window", never "add 1": an orders = orders + 1 sink double-counts a replayed event, and no broker setting prevents that.

The analytics pipeline: events to stream to warehouse

Analytics answers business questions in four steps. Instrument: apps and services emit small, validated events (page_view, search, add_to_cart, order_placed), each with an event_id for de-duplication, an event_time from the sender, a received_at from the collector (so lateness is measurable) and a schema_version. Collect and stream: a collector validates events, writes them to a topic and replies quickly; consumers (a stream job, a loader, recommendation feedback) are independent. Land and transform: a loader writes raw events to object storage as files partitioned by day (the data lake); a batch job de-duplicates by event_id and loads modelled tables into the warehouse (Stage 3). Serve: dashboards read the warehouse for history and a small metrics store for real-time numbers.

{
  "event_id": "e_5f1c0a9d",
  "type": "add_to_cart",
  "event_time": "2026-11-07T12:00:58Z",
  "received_at": "2026-11-07T12:01:20Z",
  "schema_version": 3,
  "user_pseudo_id": "u_a91f",
  "session_id": "s_33c2",
  "properties": { "product_id": "p_18342", "store_id": "store_17", "source": "search" }
}

Use a pseudonymous id, never an email or phone number: events are copied widely and kept long, so direct identifiers make privacy deletion hard. At an illustrative three events per page view, 40,000,000 page views a day is 120,000,000 events: about 1,390 per second on average and about 6,900 at the 5 times peak. The sizing question is storage and retention.

Figure 6. QuickCart's analytics pipeline.

How to read this: one stream feeds two paths. The upper answers within seconds and may be slightly wrong; the lower is slower and complete. Both can be rebuilt by replaying the stream.

Batch versus stream, lambda versus kappa

Batch processing computes over a bounded dataset on a schedule; streaming computes over events continuously. Batch is simpler, easy to re-run and cheap per event; streaming gives freshness but needs the event-time, late-data and idempotency discipline above.

The lambda architecture runs both over the same input and merges them at query time; Jay Kreps's essay "Questioning the Lambda Architecture" names its pain as maintaining "code that needs to produce the same result in two complex distributed systems". The kappa alternative keeps one stream path and reprocesses by replaying the retained log into a new output table (event streaming). QuickCart takes a hybrid: a few operational numbers go through the stream job, everything else is batch from the lake.

Figure 7. Which path does a number take?

How to read this: only numbers that drive an immediate action and tolerate small error earn a stream job. Revenue comes from the Stage 6 ledger, never from clickstream events.

Metrics for the business

Define each metric precisely and name its source, or two dashboards will disagree. Orders per daily active user is 500,000 / 2,000,000 = 0.25 (a ratio, not a conversion rate). Search success rate is searches followed by a click or add-to-cart, divided by all searches. Delivery time uses the median and 90th percentile, because averages hide pain (observability and SLOs). Revenue and refunds are sums of ledger entries, never events.

Decisions and trade-offs

In practice: QuickCart at this stage

The situation

QuickCart is at about 5,000,000 monthly active users and growing. Search is wrong after catalogue edits, a Saturday offer email sent through the order worker delayed orders by minutes, store owners say photo uploads time out on mobile, and operations wants an hourly orders-by-store view the nightly export cannot supply. Eight engineers own the work.

Starting architecture

It is Figure 1, plus the Stage 5 event stream and outbox, which are the foundation.

Options considered

The decision

Rollout and verification

Ship each pipeline beside the old behaviour. Search: build products_v2, compare its top 10 with the live index for a sample of real queries (say 1,000, illustrative), then switch the alias. Notifications: dry-run mode, logging "would send", then one type for 5 percent of shoppers. Uploads: direct upload for 10 percent of stores, watching failures and dead-letter depth. Analytics: run the stream job beside the nightly export and reconcile orders per day. Dashboards: indexer lag, zero-result rate, notification queue age per priority, per-provider error rate, media DLQ depth and stream-job lag.

The 2 a.m. failure

At 02:00 the email provider starts timing out. Workers retry without a cap, and because the marketing digest and order receipts share one topic and consumer group, receipts queue behind 400,000 digest messages (illustrative). The page says "oldest transactional notification older than 60 seconds". First three checks: per-provider error rate and latency, consumer lag per queue (is transactional waiting behind marketing?) and circuit breaker state (it should have opened). Mitigation: open the breaker by hand, switch email to the secondary provider, pause the marketing queue and let receipts drain. Follow-ups: separate queues and consumer groups by priority, capped retries with jitter, and an alert on queue age rather than depth.

Architecture after this stage

Figure 8. QuickCart after Stage 7: the four new pipelines.

How to read this: muted boxes existed before; accent boxes are new in Stage 7. Each row is one pipeline, left to right, from a source of truth to a derived view.

What QuickCart looks like now

Architect's notes

• Decide the lag you promise before choosing the mechanism. "Stock changes appear in search within 5 seconds" can be alerted on; "near real time" cannot.
• Keep one owner for each fact. The cart asks inventory about stock, not the index or a cache.
• Separate priorities early. Transactional and marketing notifications on one queue is an avoidable incident.
• Defer machine learning. Popularity, lift and a hand-tuned blend are measurable baselines.

Common mistakes

• Dual writing to the database and the index. It drifts silently; write once and publish an event.
• Recommending by raw co-occurrence counts. Compare with the base rate.
• Sending notifications from the order path, or on one queue for all kinds. A provider slowdown becomes a checkout slowdown, and marketing delays order updates.
• Treating "accepted by the provider" as "delivered", or trusting an uploaded file's name, type or size. Use receipts; decode and scan on the server.
• Counting by adding in a sink, or computing money from events. A replay double-counts; money comes from the ledger.

Key takeaways

• Derived views come from pipelines over durable facts, not from side effects of the request that created the fact.
• A search index is a rebuildable projection, kept in sync by outbox or CDC events, versioned upserts and an alias-switch reindex.
• Ranking blends text relevance with business and personal signals; recommendations work in two stages, with business rules no model overrides.
• A notification service owns preferences, quiet hours, de-duplication, rate limits, fallbacks and receipts, and keeps transactional and marketing traffic apart.
• Pre-signed uploads keep bytes off the app tier; scan, decode, resize and hash make the file safe, and content addressing makes CDN caching immutable.
• Streams need event time, watermarks and a late-data policy; "exactly once" means an idempotent effect, not a broker setting.
• Choose batch or stream per metric; money comes from the ledger.

Practice exercises

Exercise 1 (30 min): Add "Lactose-free milk 1 L" to the ranking table (text 0.89, popularity 0.3, not reordered). Compute its score with and without the personal signal and say where it ranks.

Exercise 2 (1 hour): For the four order notifications, tabulate kind, channels, fallback, quiet-hours behaviour and de-duplication key.

Exercise 3 (2 hours): Extend the window example with a sixth event at 12:01:50 and a seventh at 12:00:40 that arrives after it. Which windows close, and what does each late event do under "drop" and "allow lateness"? Write the final upserts.

Exercise 4 (half day): Design the takedown flow for a photo shared by three products through one content-addressed object: the reference check, the deletion order, what each failure leaves visible, and how you prove completion.

Related reading

• QuickCart Stage 5: The Order Pipeline - the outbox, event stream and workers this stage builds on.
• QuickCart Stage 6: Payments and the Ledger - why revenue comes from the ledger.
• QuickCart Stage 8: Split the Monolith - the next stage, which gives these pipelines service boundaries.
• Photo sharing case study - the same upload pipeline for user photos.
• Search and indexing - analysis, ranking and index sync in depth.
• Event streaming - consumer groups, delivery semantics, retention and replay.
• Building a Notification Service for 50 Million Users - a larger notification design with code.

Official documentation

• Amazon S3 Event Notifications - at-least-once delivery.
• Uploading objects with presigned URLs (Amazon S3) - how they work and expire.
• Apache Kafka documentation: message delivery semantics - idempotence, transactions, external systems.
• Apache Flink: windows - window types and late elements.
• Elasticsearch match query - fuzziness and analyzers.
• Questioning the Lambda Architecture (Jay Kreps) - lambda and the replay alternative.