System Design Case Study: Photo Sharing App

• Metadata database (a relational database such as PostgreSQL is a reasonable default at this scale): photos(photo_id, owner_id, status, caption, object_key, width, height, created_at, moderation_state) plus a tags table. An index on (owner_id, created_at DESC) serves the profile page.
• Object storage (S3-style): originals and derivatives, with keys such as orig/p_8f3a and thumb/p_8f3a/256.jpg. Objects are immutable. An edit creates a new object.
• Search index (an inverted-index engine such as Elasticsearch or OpenSearch): caption, tags and owner, populated asynchronously. It is a derived copy, never the source of truth.

The database stores a key, not the bytes. Putting megabytes into database rows bloats backups, replication and caches.

The relationships between the core records look like this.

How to read it: PHOTOS is the only table the write path must keep correct. DERIVATIVES rows can be rebuilt from the original, and the search index is built from PHOTOS plus PHOTO_TAGS.

High-level design

The upload sequence in detail:

Notice that the storage event, not the client's complete call, is the trigger for processing. If the client crashes after uploading, the photo still gets processed. The complete call exists so the API can confirm the object exists and flip the row to uploaded. A background sweeper handles rows stuck in pending past the URL expiry by deleting the row and any orphan object. Two details keep this safe: status updates must only move forward (a late complete must not overwrite ready), and the storage event should be scoped to the orig/ prefix so the derivatives a worker writes do not trigger more events.

Reference deployment topology

The logical design above becomes the following zones when deployed. Everything stateless runs in at least two availability zones.

How to read it: user-facing traffic enters at the edge, the object store and the queue decouple the app tier from the async tier, and only the workers write to the search index (the API only queries it). Losing the async tier delays thumbnails but does not stop viewing.

Deep dive 1: the asynchronous processing pipeline

A worker receives an event and must: validate that the file really is an image (do not trust the extension or the Content-Type the client claims), strip or normalise metadata, correct orientation, produce derivatives, call moderation, and publish to search.

Make every step idempotent. Queues deliver at least once, so a worker may process the same event twice. Writing thumb/p_8f3a/256.jpg twice is harmless because the key is deterministic. Updating status = ready is harmless too. Avoid steps like "increment a counter" unless you guard them.

Separate queues by cost. Thumbnailing is CPU heavy and predictable, while moderation may call a slower ML service. On one shared queue, a moderation backlog delays thumbnails. Fan out one event to several queues or consumer groups (see message queues).

Dead-letter queue. A corrupt file fails every retry. After a few attempts with backoff, move the message to a dead-letter queue, set the photo to failed, and alert on DLQ depth, so one poison message cannot waste the pool.

Autoscale on queue depth or age, not CPU alone. The metric that matches the user's pain is "oldest message age".

Deep dive 2: delivery through the CDN

At about 58,000 reads per second at peak, serving from our own servers would be wasteful. Photos are immutable, so they are ideal CDN content: give each derivative a versioned, content-based URL and a long Cache-Control: max-age with immutable. An edit produces a new URL, which avoids the hard problem of invalidation. Deletion is different, because the old URL may still be cached (see below).

Choose the cache key carefully. If your URL includes a user-specific token, every user gets a separate cache entry and the hit ratio collapses. Use a stable URL for public photos. For private photos, use signed URLs or signed cookies with a short expiry, and accept a lower hit ratio for that class.

Delivery ladder. The browser cache is checked first, then the CDN edge, then an optional shield tier (an intermediate cache that collapses simultaneous edge misses into one origin request), then object storage. A popular photo falling through to storage means a misconfigured CDN or a viral photo that many edges missed at once.

Deletion is the awkward case. The database row is gone immediately, but edges may keep serving the cached image until its TTL ends. Options: a CDN purge (supported by most CDNs but not instant everywhere), short TTLs for sensitive content, or an authorization check at the edge. For legally required takedowns, combine purge with deletion from storage and accept a brief propagation window. Say that window out loud in an interview.

Deep dive 3: moderation and search

Moderation hook. Treat moderation as a consumer of the same event. The worker sends the image (or a derivative) to a classifier and records a label and score. Three outcomes are typical: clearly fine (publish), clearly unsafe (block and notify), uncertain (queue for human review). While in review the photo may be visible only to its owner.

The design question is whether to publish first or hold until checked. Publishing first is faster for users but exposes the platform briefly. Holding until checked is safer but adds latency and a hard dependency. Many designs let a new account's photos wait for checks while trusted accounts publish first, and run checks anyway. State the policy explicitly, because it changes the state machine: pending, uploaded, processing, ready, held, failed, deleted. The decision flow below is the central trade-off of this design.

How to read it: trust decides whether you pay latency up front or exposure risk later, and the classifier's availability decides whether holding is even possible.

Search. The indexer consumes a "photo ready" event and upserts a document. Because the index is derived, you can rebuild it from the database if it is lost or its schema changes. The risk is drift: if you dual-write (API writes to DB and index) and one write fails, they diverge. Prefer one write to the database and an event (or change-data-capture) that updates the index. On delete, publish a delete event, and also filter results by checking the database for visibility, so a lagging index cannot leak deleted photos. See search and indexing for the mechanics.

Bottlenecks and scaling

• Metadata reads. At about 58,000 image reads per second, the database is not on the image path, but page loads also fetch metadata. Cache hot photo rows with the cache-aside pattern, and add read replicas for profile listings.
• Metadata writes. About 115 writes per second at peak is easy for one primary. Shard by owner_id only if growth or write volume eventually demands it (see sharding). Sharding by owner keeps a profile page on one shard.
• Object storage. Managed object storage scales horizontally, but request rates per key prefix and per-account limits exist, so spread keys and read your provider's documented limits rather than assuming none.
• Worker pool. Scale on queue age. Set a maximum so a backlog cannot exhaust budget or overwhelm downstream services.
• Egress and storage cost. Egress usually dominates. A good hit ratio at the CDN, modern formats such as WebP or AVIF where supported, and appropriate image sizes cut it. Move old originals to colder storage tiers when access drops, with lifecycle rules.

Failure modes

Durability deserves its own sentence. Managed object stores typically replicate data across several devices and zones, but replication does not protect against your own bugs or a malicious delete. Soft deletes (mark first, purge after a delay) and versioning protect against those.

Trade-offs worth stating

Interview cheat sheet

In practice: operating the reference design

A design is only finished when someone can run it. Assume a team of eight engineers owns the platform on Kubernetes, with PostgreSQL, Redis, an S3-compatible object store, a managed CDN, OpenSearch, Prometheus and Grafana.

SLOs and dashboards

Alert on burn rate of the first three, page on durability mismatches immediately, and keep hit ratio as a ticket-level signal because it drives cost rather than correctness.

Capacity-review cadence

• Weekly: queue age percentiles, worker saturation, DLQ contents, replica lag.
• Monthly: storage growth against the 2.5 PB per year forecast, egress per view, CDN hit ratio by content class, search index size.
• Quarterly: load test at twice the assumed peak (about 230 uploads per second and 116,000 reads per second), restore drill for the metadata database, lifecycle-rule audit.
• Before any campaign or launch: re-run the estimate with the marketing forecast.

Runbook

Decision record: publish policy

Cost view

Egress dominates, then storage, then compute. Request counts matter too, because each upload creates several objects.

Architect's notes

• Measure the size mix before choosing derivatives. Log which sizes clients actually request for a month. Sizes nobody requests are pure storage cost.
• Ask who owns takedowns. In the design review, name the person who can order a purge at 3 a.m. and the audit trail they leave.
• Orphans accumulate quietly. Abandoned multipart uploads and pending rows cost money. Put a lifecycle rule on them on day one.
• Treat the classifier as a dependency with an SLO. Its outage must degrade to owner-only visibility, not to a blocked upload path.
• Defer sharding. At about 115 writes per second, a single PostgreSQL primary with replicas is enough. Set the revisit trigger as a number, not a feeling.
• Test the restore, not the backup. Durability claims are only as good as the last restore drill.

Common mistakes

• Uploading through the API servers. Large bodies tie up connections and memory and force you to scale app servers for bandwidth. Use presigned direct uploads.
• Storing image bytes in the relational database. Backups, replicas and caches balloon. Store keys, keep bytes in object storage.
• Doing thumbnailing inside the upload request. Latency and timeouts follow. Acknowledge fast and process asynchronously.
• Non-idempotent workers. At-least-once delivery will duplicate work. Use deterministic keys and conditional updates.
• Forgetting deletion and takedown paths. Cached copies and index entries outlive the row. Plan purge, delete events and visibility checks.
• Making the search index the source of truth. It will drift or be lost. Treat it as rebuildable from the database.
• Trusting the client's file type or size. Validate content server-side after upload, and constrain size at upload time where the storage service supports it (for example with a presigned POST policy).

Key takeaways

• Keep bytes off the app tier. Presigned URLs let clients write directly to object storage.
• Acknowledge fast, process later. A queue and workers separate the user-visible path from slow derived work.
• Idempotency is mandatory. At-least-once delivery means every pipeline step must tolerate repeats.
• The CDN does most of the reading. Immutable content with stable URLs gives a high hit ratio.
• Derived data is rebuildable. Thumbnails and search entries can be regenerated from originals and metadata.
• Deletion is harder than creation. Plan for purge lag, soft deletes and visibility checks.
• Scale on the right signal. Queue age reflects user pain better than CPU does.
• Name your policies. Publish-then-moderate versus moderate-then-publish changes the whole state machine.

Practice exercises

Exercise 1 — Starter (30 min): Draw the photo state machine with every state in this lesson (pending through deleted). Label each transition with the event or actor that causes it, and mark which transitions the sweeper performs.

Exercise 2 — Intermediate (2 hours): Redo the estimate for 50 million daily users with 200 million uploads per day and an average original of 2 MB. Compute uploads per second at peak, daily and yearly storage growth, and the egress per day if the CDN hit ratio is 95% versus 80% and each view transfers 150 KB. State which assumption moves the cost most.

Exercise 3 — Advanced (half day): Design the takedown flow for a legally required removal: database, object storage, CDN purge, search index, backups, and any derived caches. Write the order of operations, what happens if each step fails, and the maximum time the content can remain visible. Then decide what you would monitor to prove the takedown completed.

Related reading

• System Design: The Big Picture - the high-level tour this case study expands on.
• CDN and object storage - the mechanics of presigned URLs, multipart upload and edge caching.
• Message queues - acknowledgements, retries and dead-letter queues used by the pipeline.
• Search and indexing - how the inverted index is built and kept in sync.
• System Design Case Study: News Feed - the next case study, built on top of content like this.