System Design: CDN and Object Storage

Versioned URLs beat invalidation

When you change a file, cached copies around the world are stale. You have two strategies:

1. Invalidation (purge). Ask the CDN to delete cached copies of a path. It works, but it takes time to propagate, may be limited or charged by volume, and browsers may still hold their own copies.
2. Versioned or content-addressed URLs. Put a version or hash in the file name: /photos/abc_v2.jpg or /static/app.3f9c2e.js. When content changes, the URL changes, so there is nothing to purge.

For immutable uploads, such as a photo that is never edited, the object key can itself be unique and permanent.

Cache keys

A cache key is what the CDN uses to decide whether two requests are "the same". By default it is usually the URL, often including the query string.

• Tracking or session parameters (?utm_source=..., ?session=...) can make every request a separate entry and wreck the hit ratio. Configure the CDN to ignore irrelevant parameters.
• If the response differs by a header (for example Accept-Encoding or an image format preference), the key must include it or the response must use Vary correctly. Otherwise a user can get the wrong variant.
• Never cache personalised responses under a shared key.

Object storage concepts

Object storage stores data as objects inside containers called buckets. An object is the file's bytes, a key (its name, such as listings/8841/photos/abc.jpg), and metadata (content type, custom tags). There are no real folders; slashes in keys are just part of the name, even though tools display them as folders. Examples include Amazon S3, Google Cloud Storage and Azure Blob Storage, plus self-hosted systems that speak S3-compatible APIs.

What makes it different from a disk or a database:

• Whole-object operations. You put, get, delete or overwrite entire objects over HTTP. To change an object, you write a new version.
• Huge scale and low cost per gigabyte. Designed for billions of objects.
• Durability by redundancy. Providers typically store data redundantly across multiple devices and facilities and publish a durability figure. Do not confuse durability (will my data still exist?) with availability (can I read it right now?). Durability does not protect you from your own accidental delete unless you enable versioning or backups.
• Consistency semantics vary by provider and over time. Rely on current documentation rather than old folklore.
• Not a database. There is no rich querying. Keep searchable metadata in a database and use the object key as the link.

Presigned URLs

How does a user upload a photo to a private bucket without giving them your credentials, and without pushing all the bytes through your servers? With a presigned URL: a temporary URL, created by your server with its credentials, that grants permission to perform one specific operation on one specific object until it expires. The app checks that the user is allowed to upload, generates a presigned PUT (or POST) URL valid for a short time such as ten minutes, and hands it back. The client then sends the file directly to object storage. Keep the expiry short, scope the URL to one key, and constrain content type and size where the provider allows it.

Multipart upload

Large files are uploaded in parts. In a multipart upload, the file is split into chunks (several megabytes each), uploaded in parallel, and then assembled by the storage service. If one part fails, only that part is retried; parts upload in parallel; and single-request size limits are avoided (limits differ by provider). Also arrange cleanup: abandoned multipart uploads keep occupying storage until aborted, so use a lifecycle rule (an automatic storage rule such as "abort incomplete uploads after seven days").

The upload and download flow

The app tier handles only small JSON messages; the large bytes move straight between client, storage and CDN. In words: the app authenticates the user and creates a pending row with a unique object key; the client uploads straight to storage; storage emits an event that enqueues a job (message queues); a worker validates the file, strips sensitive metadata where appropriate, writes several sizes back to storage, and marks the row ready; pages then reference CDN URLs, not the bucket.

The row's status is the contract: the site shows only Ready photos, and background jobs clean up Expired ones.

Why keep the original and generate derivatives? Different screens need different sizes. Serving a 12-megapixel original to a phone showing a 400-pixel thumbnail wastes bandwidth and money. Pre-sized variants (or an image service that resizes on demand and caches the result) fix that.

Serving images, video and private content

Images. Use efficient formats where browser support allows, serve appropriately sized variants, set long immutable lifetimes on versioned URLs, and lazy load images below the fold.

Video. Do not serve a giant single file for streaming. The video is transcoded into several quality levels and split into short segments, and the player switches quality as the network changes (adaptive bitrate streaming; HLS and DASH are common formats). Segments are ordinary static files, ideal for CDN caching. The full design is in the photo sharing case study and the video streaming case study later in the course.

Private content. Public CDN URLs can be shared by anyone who has them. For restricted content use signed URLs or signed cookies supported by your CDN, with expiry times. Check how your CDN builds the cache key: if each signed URL counts as a separate key, caching is less effective, which is one reason signed cookies suit assets made of many files.

Cost levers

Media systems often spend more on bandwidth and storage than on compute. Pricing models vary, so check your provider.

In practice: photos and virtual tours for a real-estate listing site

The situation

A real-estate listing site shows about 300,000 active listings, each with about 30 photos, a 360-degree panorama set and sometimes a short walkthrough video. Assume 9 million photos averaging 6 MB as originals, roughly 54 TB. A Node.js and TypeScript API runs on Kubernetes with PostgreSQL for listings; originals sit in an S3-compatible bucket behind a CDN; Prometheus and Grafana provide metrics and Terraform manages infrastructure. A team of six engineers supports the web and mobile apps, and agents upload from phones on cellular networks.

The pain, with illustrative numbers: the listing page transfers about 14 MB of photos because it serves originals. Uploads of 40 MB batches go through the API, which times out on slow connections. The CDN hit ratio is about 55 percent because image URLs carry session and tracking query strings. The monthly egress line is the largest item on the cloud bill, and photos of sold listings stay in the hot tier forever.

Starting architecture

Uploads consume API capacity, and the browser downloads originals through a CDN whose cache keys are fragmented.

Options considered

The decision

The measurable targets: first-view image transfer under about 1.5 MB, CDN hit ratio above 90 percent, and upload success rate above 99 percent on mobile.

Rollout and verification

Roll out in stages: new uploads first, behind a flag for a subset of agents, while old listings keep the old path. Then run a throttled backfill that creates variants for existing photos, newest listings first, rate limited so it cannot saturate the workers or storage requests. Switch the listing page to variants for a small share of traffic and compare. Dashboards: CDN hit ratio and origin requests per second, bytes per page (from real-user monitoring), upload success rate and part retry count, queue depth and oldest message age, worker failure rate, and a daily egress cost graph with a budget alert.

The 2 a.m. failure

A web deploy at 01:40 adds a cache-busting query parameter to every image URL. At 02:10 the page says "CDN hit ratio below 60 percent, origin request rate high".

Detection: the hit-ratio panel drops while origin requests per second jump and an egress budget alert fires. The first three checks: fetch one image URL twice with curl -s -o /dev/null -D - "<url>" and look at the Age and cache-status headers (the repeat should be a hit), then change only the cb value and confirm it becomes a miss, which proves the parameter is part of the cache key; list top requested URLs in the CDN logs grouped by path and count of distinct query strings; and review what changed in the deploy. Mitigation: add the parameter to the CDN's ignored-parameters list (or roll back the deploy), then watch hit ratio recover. Follow-up: keep cache-key rules in Terraform, add a CI check that image URLs are stable across builds, and alert when hit ratio stays under a set threshold for ten minutes.

Architect's notes

• Measure bytes per page before designing variants. Use real-user monitoring to see what mobile users actually download. Variants are a means; page weight is the goal.
• Ask for the retention policy. Who decides how long sold-listing photos stay, and who pays for retrieval from colder tiers? Without an answer, lifecycle rules become a late argument.
• Egress is a product decision. Autoplaying walkthrough videos cost real money. Ask product whether the default can be a poster image with play on tap.
• Backfills are projects. Estimate the number of objects and per-request costs before starting, and make the job resumable and rate limited.
• Decide who is paged. Hit ratio and origin errors sit between the platform team (CDN, bucket) and the application team (URLs). Agree ownership and a single alert path.
• Defer on-demand transformation. Fixed variants are simple to cache and reason about. Revisit only when the design genuinely needs arbitrary crops.

Common mistakes

• Serving files through the application servers. Large downloads tie up app capacity. Use presigned URLs for uploads and a CDN for downloads.
• Short TTLs on immutable files. You throw away the main benefit. Use unique URLs and long TTLs.
• Relying on purges for correctness. Invalidation is slow and limited. Prefer versioned URLs.
• Polluted cache keys. Random query parameters or unneeded headers fragment the cache. Normalise keys.
• Caching personalised responses under a shared key. It leaks data between users. Vary correctly or do not cache.
• Public buckets by accident. A misconfigured bucket exposes data. Keep buckets private, serve through the CDN with controlled access, and review permissions.
• Presigned URLs with long expiry and broad scope. A leaked URL becomes a standing credential. Keep them short-lived and specific.
• Forgetting lifecycle rules. Abandoned multipart uploads, temporary files and old derivatives pile up silently.

Key takeaways

• Bytes should skip your app servers. Use object storage for files and a CDN for delivery.
• A CDN moves copies close to users. It lowers latency, absorbs spikes and shields the origin.
• Pull is the usual model. The edge fetches from the origin on a miss; push suits predictable large releases.
• Version your URLs. Content-addressed or versioned names allow long TTLs and avoid purges.
• Mind the cache key. Normalise query strings and use Vary correctly so the hit ratio stays high and responses stay correct.
• Object storage is durable and cheap, not a database. Store bytes there and metadata in a database, linked by the object key.
• Presigned URLs and multipart uploads scale uploads. Clients upload directly, with short-lived scoped permission and resumable parts.
• Watch egress and tiers. Hit ratio, image sizing and lifecycle rules control the bill.

Practice exercises

Exercise 1 — Starter (30 min): For each of these files, write the Cache-Control header you would send and why: a logo at a fixed URL, a JavaScript bundle named with a content hash, an agent's profile photo at /agents/42/avatar, a private contract PDF, and a listing photo with a unique permanent key.

Exercise 2 — Intermediate (2 hours): Estimate the origin traffic for a listing site with assumed numbers: 1 million listing views per day, an average of 10 photos each, an average variant size of 150 KB, and CDN hit ratios of 55% and 92%. Treating every miss as one full origin fetch, calculate how many bytes per day reach the origin in each case, and how much origin traffic the better hit ratio saves. State your assumptions and show the arithmetic.

Exercise 3 — Advanced (half day): Using any S3-compatible service or a local emulator, build the upload flow: an API that returns a presigned URL, a client script that uploads a file directly, and a worker that creates a resized variant when the object appears. Add a lifecycle rule that aborts incomplete multipart uploads. Then write a failure analysis: what state is the database in if the client uploads but the worker crashes, and how does your design recover?

Related reading

• System Design: Caching Basics — the general ideas behind TTL, hit ratio and staleness that CDNs apply at the edge.
• System Design: Load Balancing — how traffic reaches the origin when the CDN misses.
• System Design: The Request Lifecycle — DNS, TLS and HTTP steps that happen before the edge responds.
• System Design: Message Queues — how upload events trigger background processing reliably.
• System Design: Photo Sharing Case Study — the full end-to-end design that builds on this flow.