System Design: API Design

Idempotency and idempotency keys

An operation is idempotent if doing it once or many times has the same effect. GET, PUT and DELETE are idempotent by definition. POST usually is not, and that is where trouble starts.

A member taps "Book" on a gym's weak wifi. The request reaches the server, the response is lost, and the app retries. Did the first one succeed? The client cannot tell. If it retries it may book twice; if it does not, it may book nothing. Networks make this unavoidable, so the API must give the client a safe way to retry.

The standard answer is an idempotency key: the client generates a unique value (a random UUID) and sends it in a header, Idempotency-Key: 6f1c2d3a-.... The server remembers the key and the result.

How to read it: the second request does no booking work. The saved result is returned as if it were the first.

• The check-and-store of the key must be atomic with creating the booking (all-or-nothing, no gap), or two simultaneous retries could both pass the check.
• Keep keys for a bounded time, long enough to cover realistic retries (hours to a day), then expire them.
• If the same key arrives with a different body, reject it: that is a client bug.
• Return the original response for a repeat, so the client cannot tell the difference.

This is how an "exactly once effect" is built from at-least-once delivery: the client may send many times and the server makes repeats harmless.

Versioning

Clients cannot all upgrade at once, and you will need to change things.

A practical combination is a major version in the path (/v1) and a strict habit of additive changes. Whether a change needs a new version is a decision you can draw.

How to read it: most changes should exit at the first or second question. A new major version is the exception, and you only turn off the old one when usage has fallen.

Clients should ignore unknown fields. When a breaking change is needed, run both versions, publish a deprecation date, and watch /v1 usage fall before retiring it.

A consistent error format

Errors are part of the contract. Every error should look the same, so clients write one handler.

{
  "error": {
    "code": "class_full",
    "message": "The class has no spots left.",
    "details": [{ "field": "class_id", "capacity": 20 }],
    "request_id": "req_8c1f"
  }
}

Include a stable machine-readable code (clients branch on it, so never reuse it with a new meaning), a human message, optional field-level details, and a request_id that support can find in logs. Never leak stack traces, SQL or internal hostnames. Pair the body with the right status code (here 409).

Authentication, authorisation and rate limits

Authentication proves who the caller is, typically with Authorization: Bearer <token>. Authorisation decides what that caller may do ("can member 42 cancel booking b81?"). Check both on every request, on the server, over HTTPS; never trust the client to enforce rules.

Rate limiting protects you from abuse and accidents. Retry-After is a standard header, commonly sent with 429 Too Many Requests (or 503), giving the seconds to wait or a date. Many APIs also send informational headers showing the limit, what remains and when it resets. Names vary between providers (X-RateLimit-* is a widespread convention, and a standard is in progress), so document yours. The Rate Limiting lesson covers the algorithms.

Choosing REST, gRPC or GraphQL

There is no winner. Many systems use REST at the public edge, gRPC between internal services, and sometimes GraphQL for one rich client. Pick for the consumer: who calls this, from where, and how often?

Example: a presigned upload for studio photos

Studios upload photos. Pushing 2 MB files through your application servers ties them up moving bytes. A common design lets the client send the file straight to object storage using a presigned URL: a time-limited URL that grants permission to upload one specific object. The API only handles small metadata calls.

POST /v1/photos creates a record in state pending (accept an Idempotency-Key here too). complete verifies the object and enqueues thumbnail work, and GET /v1/photos/{id} shows status moving through pending, processing and ready. Return 202 Accepted for work that is accepted but not finished. Enforce the size limit in several places: state it when creating the record, apply any size condition your storage provider supports (support varies, so check its documentation), and verify the real size at complete.

In practice: a fitness-class booking service with a partner API

The situation

A fitness-class booking service lets members reserve spots in yoga and spin classes at partner studios. Seven engineers run it. A mobile app is the main client, and about 30 partner platforms (corporate-wellness tools and studio websites) use the same HTTP API. The stack: a Kotlin service on Kubernetes, PostgreSQL, Redis, and Nginx ingress.

Three problems hurt. Duplicate bookings: assume 0.8% of bookings are duplicates, produced by mobile retries on poor gym wifi, creating around 300 support tickets a month. Partner syncs: the class-list endpoint uses offset pagination, p95 at page 500 is about 2.5 seconds (assumed) and partners report missing and repeated classes when schedules change mid-sync. Change risk: a partner wants a field renamed, and nobody knows which of 30 integrations would break.

Starting architecture

The data model behind the contract is small, and the booking resource is the one with the double-booking risk.

Options considered

The decision

The capacity check uses one atomic statement, UPDATE class_sessions SET booked = booked + 1 WHERE id = ? AND booked < capacity, so two members racing for the last spot cannot both win. If it updates zero rows, the API returns 409 with class_full.

Rollout and verification

Run in observe-only mode first: accept the key, record it, change nothing, and measure how many duplicates it would have caught. Enforce for the mobile app (which you control) before partners. Add cursor parameters beside offset, announce a retirement date with the Sunset header (a standard HTTP header for announcing when a resource will go away), and track usage per partner key. Dashboards: duplicate-booking rate (target near zero), idempotent-replay rate, p95 of list calls by page depth, traffic by API version and partner, and 409 and 422 rates. Add contract tests that call the API as a partner would, so a renamed field fails a build rather than an integration.

The 2 a.m. failure

A nightly cleanup job deletes idempotency keys older than 24 hours. It silently failed for several weeks after a deploy changed its schedule, and the first successful run then issues one huge DELETE of the backlog at 02:00. The page reads "POST /v1/bookings p99 above 3 s". The delete creates a large number of dead rows and heavy disk and write-ahead-log activity on the same table and index that every booking write touches.

First three checks: the booking latency and disk I/O panels, SELECT pid, state, wait_event_type, left(query, 60) FROM pg_stat_activity WHERE state <> 'idle' ORDER BY query_start; to spot the long-running delete, and a count of rows in the key table. Mitigation: cancel the job with SELECT pg_cancel_backend(pid);, then let the system recover. Follow-up: delete in small batches of a few thousand rows, or partition the key table by day and drop old partitions, add an index on created_at, and alert on the table's size.

Architect's notes

• Retries are the default, not the exception. Mobile networks and partner schedulers retry. Design every POST that creates something to be safely repeatable.
• The contract is a product. Ask in the review: who are the consumers, which fields do they depend on, and how will we know before we break them? Per-partner usage metrics answer that.
• Prefer cursors for anything partners sync. Deep offset pages are a latent outage, and repeated or missing rows are a correctness bug that is hard to see.
• Keep idempotency state cheap to expire. Put a retention window on it from day one and plan cleanup as a first-class job.
• Own the deprecation. Someone must run the sunset conversation with partners. Defer /v2 until additive change truly cannot express the need.

Common mistakes

• Verbs in URLs and inconsistent naming. /getBooking and /Bookings make every endpoint a surprise. Fix: nouns, plural collections, standard methods.
• Returning 200 for errors. Monitoring and retries cannot tell success from failure. Fix: accurate 4xx and 5xx codes plus a structured body.
• Offset pagination on big, live lists. It slows with depth and repeats or skips items. Fix: cursors with a stable sort and a capped page size.
• Unsafe retries on POST. Duplicate charges or bookings. Fix: idempotency keys stored atomically.
• Breaking changes without a version. Renaming a field breaks apps you cannot force to update. Fix: additive changes, versioned majors, deprecation notices.
• Unbounded results and unindexed filters. One query can tie up the database. Fix: cap page sizes and filter only on indexed fields.
• Leaking internals. Exposed database IDs and stack traces couple clients to your implementation. Fix: stable public identifiers and sanitised errors.

Key takeaways

• An API is a contract. Changing it later is costly, so design for predictability, safety and evolution.
• Resources are nouns, methods are verbs. Use standard methods and accurate status codes; 4xx and 5xx mean different things to clients and monitors.
• Cursors beat offsets at scale. Cursor pagination stays fast at any depth and is stable while data changes.
• Cap and index. Limit page sizes and allow filtering only on fields the database can serve efficiently.
• Idempotency keys make retries safe. The client reuses a unique key and the server returns the saved result.
• Evolve by adding. Add optional fields freely; use a new version only for genuine breaking changes.
• Errors need a stable shape. A machine-readable code, a message and a request ID serve clients and support.
• Choose the style for the consumer. REST for broad public use, gRPC between internal services, GraphQL when varied clients need flexible queries.

Practice exercises

Exercise 1 — Starter (30 min): Design the REST endpoints for a comments feature on a class page: create, list, edit and delete a comment. For each, write the method, URL, success status code and one likely error status code. Check that every URL is a noun and every list endpoint is paginated.

Exercise 2 — Intermediate (2 hours): Write a full specification for GET /v1/classes using cursor pagination. Describe the request parameters (limit, cursor), the response shape, what the cursor encodes (for example, a start time plus an ID as a tie-breaker), what happens at the last page, and what the server does if the cursor is malformed or expired. Then write the database query shape (in words or SQL) and explain why an index on the sort fields makes it fast.

Exercise 3 — Advanced (half day): Design an idempotent POST /v1/payments endpoint. Draw a sequence diagram for a client retrying after a lost response, and a second for two simultaneous retries arriving at different servers. Specify where the key is stored, how the check is made atomic, how long keys live, and what happens when the same key is sent with a different body. List two failure modes (for example, the server crashes after charging but before saving the result) and how you would handle each.

Related reading

• The Request Lifecycle - the previous lesson, covering the HTTP journey your API sits on.
• Load Balancing - the next lesson: spreading API traffic across servers.
• Rate Limiting - the algorithms behind 429 responses and limit headers.
• API Gateway and Service Mesh - where authentication, routing and limits are often enforced.
• Case study: Photo Sharing - the upload flow in this lesson, designed in full.