System Design: IDs, Clocks and Ordering

The flowchart below is the order in which a design review usually asks the questions.

Read it top to bottom: a single writer makes the problem disappear; otherwise the 64-bit requirement is what pushes you toward Snowflake-style IDs and their operational duties. Whatever you pick, if the ID is shown to customers, add a separate unguessable public identifier rather than exposing the internal key.

The Snowflake duties: clocks and worker IDs

A Snowflake-style generator depends on the machine clock, so it must handle a clock that moves backwards, or it may issue an ID it already issued.

A small regression is waited out. A large one means the generator stops issuing and tells the platform it is unhealthy, because guessing is how duplicates are born.

Worker IDs need an owner too. Common approaches: assign them from a coordination service such as ZooKeeper or etcd at startup, derive them from a stable instance number (for example a Kubernetes StatefulSet ordinal), or lease them with an expiry so a crashed worker's ID is not reused while it might still be alive.

Why you cannot trust the wall clock

A physical clock is a quartz crystal counting oscillations. It runs slightly fast or slow, by an amount that changes with temperature. That error is drift: two machines that agree at noon disagree a little by evening.

To limit drift, machines use NTP (Network Time Protocol) to ask a time server for the correct time. NTP is good but not perfect: over a network the error is often milliseconds to tens of milliseconds, and during network trouble it can be far worse. Time can be stepped (jumped) or slewed (sped up or slowed down gradually). Leap seconds add surprises, and virtual machines that pause and resume can wake with a badly wrong clock.

The difference between two machines' clocks at one instant is clock skew, and it creates concrete bugs:

B really wrote after A, but because B's clock runs behind, "last write wins by timestamp" keeps A's value and discards the newer write. Nobody sees an error; data is simply lost. This is why timestamp-based conflict resolution needs care, as revisited in multi-region and disaster recovery.

The practical rules:

• Use monotonic clocks for durations. They never jump backwards. Use the wall clock only for "what time is it for a human".
• Never compare timestamps from two machines and conclude causality unless you bound the error.
• Make leases skew-tolerant. A lock that "expires at 10:00:05" only means something if everyone agrees on 10:00:05, so lease protocols measure durations locally, with a safety margin.

Happens-before and logical clocks

If wall clocks cannot order events, what can? In 1978 Leslie Lamport observed that for most purposes we only care about causality: could event A have influenced event B? He defined the happens-before relation, written A → B:

1. If A and B are on the same process and A occurs first, A → B.
2. If A is the sending of a message and B is its receipt, A → B.
3. It is transitive: if A → B and B → C, then A → C.

If neither A → B nor B → A, the events are concurrent. That does not mean simultaneous; it means no information could have flowed from one to the other. Concurrent events have no meaningful order, and a correct system must not pretend otherwise.

Lamport clocks

A Lamport clock is a counter on each process: increment before each local event; when sending, increment and attach the counter; on receipt, set your counter to max(local, received) + 1.

The guarantee: if A → B then the Lamport timestamp of A is less than that of B. The reverse is false: if timestamp(A) < timestamp(B), A might have happened before B, or they might be concurrent. Breaking ties with the process ID gives a total order consistent with causality, which many algorithms (mutual exclusion, replicated logs) use.

Vector clocks

To detect concurrency, each process keeps a vector with one counter per process. On a local event, increment your own entry. On sending, attach the vector. On receiving, take the element-wise maximum, then increment your own entry. Comparing vectors tells the truth: if every entry of V1 is at most the matching entry of V2 (and one is smaller), V1 happened before V2; if neither dominates, the events are concurrent.

Dynamo-style stores used this idea to detect that two replicas accepted conflicting writes, then handed both versions to the application to merge. The cost is size: a vector grows with the number of writers, so real systems prune or use variants such as dotted version vectors.

Hybrid logical clocks and TrueTime

Lamport and vector clocks bear no relation to real time, which is awkward when people ask for "everything from last Tuesday". A hybrid logical clock (HLC) combines a physical timestamp with a small logical counter: use the maximum of the local physical time and the highest timestamp seen so far, and if that equals the previous value, bump the counter. The result stays close to wall-clock time, never goes backwards, and respects causality: a receiver's HLC is always greater than the sender's. Several distributed databases use HLC-style timestamps.

Google's Spanner takes another route. Its TrueTime API returns an interval designed to contain the true time, backed by GPS receivers and atomic clocks. Before committing, a transaction waits long enough that different transactions' intervals cannot overlap confusingly. The lesson travels well: if you can bound clock uncertainty, you can wait it out; if you cannot, do not rely on physical time for correctness.

Choosing an ordering guarantee

Ordering is a spectrum, and stronger ordering costs more coordination.

Most business needs stop at the second box. A chat app needs messages within one conversation ordered, not a global order; partition by conversation and you avoid global coordination. See event streaming for how partitions provide exactly this.

In practice: order IDs and event ordering for a multi-service retail platform

The situation

A retail platform has an order service, payment service, inventory service and shipping service, owned by four teams of about six engineers each. Orders live in a single PostgreSQL primary with an auto-increment order_id, and services publish events to Kafka. Two problems have reached the architecture review. First, the database is nearing its write ceiling, so the plan is to shard orders across eight PostgreSQL shards, which breaks the single sequence. Second, customer support complains that the order timeline sometimes shows "shipped" before "paid". Each service stamps events with its own created_at, and with a skew of a few hundred milliseconds between hosts (an assumption), a payment confirmation and the shipping label it triggers can be stamped in the wrong order. Constraints: no downtime during the change, a public order number that must not reveal daily volume, and a six-week window before peak season.

Starting architecture

Options considered

The decision

Rollout and verification

Ship the new columns first and backfill order_version from the existing event log. Dual-write the new ID and the old sequence value behind a feature flag, canary on one pod, then ramp. Run the timeline in shadow mode, computing both orderings and logging every order where they disagree; the disagreement count should fall to zero for new orders. Dashboards to watch: ID generation rate per worker, id_clock_regression_total, duplicate-key errors on the order table (target zero), consumer lag per partition, and a daily check that order_version has no gaps per order. Load test generators at a multiple of peak order rate before touching sharding.

The 2 a.m. failure

The page reads "Order creation error rate above 2 percent on order-service-3". The cause is a node whose NTP daemon stepped the clock back 1.8 seconds after a VM pause. The generator detects the regression, exceeds its small wait threshold, and marks itself unhealthy, so one pod stops issuing IDs while the rest continue.

First three checks: kubectl logs order-service-3 | grep clock_regression to confirm the cause; chronyc tracking on the node for the offset and last step; and a query for duplicate-key violations on the orders table to prove no duplicate was issued (expected zero). Mitigation: cordon the node and let the pod reschedule onto a healthy host; because the worker ID is the StatefulSet ordinal, the new pod must not issue IDs until its clock passes the last timestamp persisted by the old one, so the generator stores a high-water mark periodically and refuses to start below it. Follow-up fix: configure chrony to slew rather than step after startup, alert on offset above a threshold before it becomes a step, and add the high-water mark to the test suite.

Architect's notes

• Measure skew before arguing about clocks. Export each host's offset from its time source and graph the worst case; the number decides whether you need logical ordering at all.
• Ask in the review: "who assigns the order?" For every event stream, name the single component that owns the sequence. If the answer is "the consumer's clock", the design is wrong.
• A per-entity version number is the cheapest ordering you can buy. It costs one column and an increment inside the transaction that already exists, and it removes most arguments about timestamps.
• The cost trap is wide keys. A 16-byte key repeats in every secondary index and foreign key, so storage and cache footprint grow faster than the table itself.
• Public IDs are a product decision, not just an encoding. Decide early whether customers may guess, enumerate or type them; changing the format later touches emails, support scripts and partner integrations.
• Defer HLC and vector clocks until you have a real multi-writer conflict problem. Most services need only partition order plus a version.

Common mistakes

• Using now() timestamps to order events from different machines. Skew can reverse real order. Use a logical clock, a per-entity sequence from one owner, or a bounded-uncertainty clock.
• Random UUID v4 as the key of a huge write-heavy table. Random inserts fragment the index. Prefer a time-ordered ID.
• Ignoring backwards clock jumps in an ID generator. It can issue duplicates. Detect the jump and block or fail.
• Exposing sequential IDs publicly. They let outsiders count your records. Use unguessable public IDs and keep internal IDs private.
• Treating "same millisecond" as concurrency. Concurrency means no causal path, not equal timestamps; different timestamps do not prove causality either.
• Measuring timeouts with the wall clock. A clock step can fire or suppress timeouts. Use the monotonic clock.
• Assuming Lamport order equals real order. It only guarantees that causes get smaller numbers than their effects.

Key takeaways

• There is no global clock. Clocks drift and skew, and NTP narrows but never removes the error.
• Uniqueness, time-sortability and causality are different goals. An ID scheme covers the first two; logical clocks or a log cover the third.
• UUID v4 is simple and unguessable but index-unfriendly. Time-ordered IDs keep inserts local.
• Snowflake-style IDs trade coordination for clock dependence. Handle worker ID assignment and backwards jumps explicitly.
• Happens-before is the real definition of order. Events with no causal path are concurrent.
• Lamport clocks give a causality-consistent total order; vector clocks detect concurrency.
• Hybrid logical clocks keep timestamps close to real time while respecting causality.
• Buy only the ordering you need. Per-partition order is cheap; total order needs consensus.

Practice exercises

Exercise 1 — Starter (30 min): A Snowflake-style ID has 41 bits of milliseconds, 10 bits of worker ID and 12 bits of sequence. Calculate (a) how many IDs one worker can issue per second at most, (b) how many years the timestamp field lasts, and (c) the maximum number of workers. Show your arithmetic and state which numbers are exact and which are rounded.

Exercise 2 — Intermediate (2 hours): Three processes P1, P2, P3 exchange messages: P1 sends m1 to P2, then P2 sends m2 to P3, then P3 does a local event, and separately P1 does a local event after sending m1. Draw the timeline and compute Lamport timestamps and vector clocks for every event. Identify one pair of events that is concurrent and explain why the Lamport timestamps alone would not reveal this.

Exercise 3 — Advanced (half day): Design the ID and ordering strategy for a multi-tenant order system with 20 database shards and several application regions. Choose an ID format, explain how worker IDs are assigned and how a backwards clock jump is handled, decide which entities need causal order versus per-key order, and describe what the system does when two regions update the same order concurrently. Write a one-page design note and list two failure scenarios with their mitigations.

Related reading

• System Design: Consensus and Leader Election — how a replicated log gives a true total order when you need one.
• System Design: Event Streaming — per-partition ordering in practice.
• System Design: Multi-Region and Disaster Recovery — what clock skew means for conflict resolution across regions.
• System Design: Case Study, URL Shortener — ID generation applied to a real design.