Skip to content

    System Design

    Hot Partitions, Hot Keys and Resharding: Skew, Splits and Safe Moves

    Why one partition runs hot while the rest idle, how to detect skew, how salting, caching and load-aware placement help, and how to reshard a live system safely.

    By Published 13 min read

    Share

    Standard lessonadvancedBackend System Design

    Before this: Consistent Hashing

    What you'll learn: By the end of this lesson you will be able to explain how skew makes one partition the bottleneck of a whole cluster, name the common causes (celebrity keys, time-based keys, sequential keys), detect a hot partition with per-partition numbers, choose among salting, caching, load-aware placement and rate limits, compare fixed-partition and dynamic-split resharding, and plan an online reshard with a verification step and a way back.

    A supermarket opens eight checkout lanes, then assigns shoppers to lanes by the first letter of their surname. On a normal day the queues look even. On the day a local club with 300 members called Smith holds its annual shop, lane 8 has a queue to the door and lanes 1 to 7 are empty. Opening a ninth lane changes nothing: the club members still all go to lane 8.

    Partitioned data stores fail the same way: adding machines helps only if the load spreads over them. This lesson covers why it often does not, what to do about a key or range that is too busy, and how to change the number of partitions without stopping the system. Partitioning basics are in sharding and partitioning, and the table design side is in Cassandra and DynamoDB modelling. Consistent hashing has its own lesson in this module and appears here only by name.

    Verified on 2026-11-19 against the Amazon DynamoDB, MongoDB, Redis, Vitess, CockroachDB and Apache Cassandra documentation and the Dynamo paper. No code on this page was run.

    The core idea

    A partitioned system is as fast as its busiest partition. A hot partition is a partition that receives much more load than the average; a hot key is a single key that causes it. Skew is about load, not size: a partition can hold average data and still take most of the traffic. The fixes work at three levels: change the key so load spreads, change placement so hot items do not share a node, or change the number of partitions.

    Three illustrations run through the lesson, each labelled:

    • Minimal example: a key made from the date, so every write of the day lands in one partition.
    • Realistic example: a ShopStream flash sale on one product's stock counter.
    • Failure example: a backfill that overwrites a newer write during resharding.

    How skew arises

    Four patterns cover most cases.

    Pattern What happens Documented example
    Celebrity key One item or tenant gets far more traffic than the rest DynamoDB rates a device ID as bad uniformity when "one is by far more popular than all the others"
    Time-based key Every new write has the current time in its key, so all of it lands on the newest partition DynamoDB rates an item creation date "rounded to the nearest time period" as bad
    Sequential key An always-increasing key sends every insert to the last range MongoDB: "all new inserts are routed to the chunk with maxKey as the upper bound"
    Low-cardinality key Few distinct values, so few partitions can ever be used DynamoDB rates a status code with few values as bad; MongoDB says low cardinality "reduces the effectiveness of horizontal scaling"

    DynamoDB states the ceiling plainly: every partition is designed to deliver at most 3,000 read units and 1,000 write units per second, and a key design that does not distribute requests can create hot partitions that cause throttling. Minimal example: a table keyed by order date. Suppose a day brings 1,500 writes per second at peak (an illustrative figure). All of them go to the partition for today's date, which tops out at 1,000, so about a third of the writes are throttled while every other partition is idle.

    A time-based key sends every write to one partitionWriters to Router; Router to Partition 1 (same key prefix); Router to Partition 2; Router to Partition 3Writers: all keys start with today's dateWritersall keys startwith today's dateRouter: hash of the keyRouterhash of the keyPartition 1: takes every writePartition 1takes every writePartition 2: idlePartition 2idlePartition 3: idlePartition 3idlesame key prefix

    Figure 1. Skew from a time-based key. How to read this: the route to partition 1 carries all the traffic. The dashed red routes are the capacity that exists and is never used.

    Detecting skew

    Cluster averages hide it: a cluster at 25 percent average load can have one partition at 100 percent. Watch three signals: throttling or queueing on one partition while others idle, per-partition request and byte rates, and a top-N list of keys. DynamoDB's CloudWatch Contributor Insights gives the last: it reports "the most accessed and throttled items" in a table, and in its graphs closely clustered lines mean a balanced workload while an outlier means a hot item.

    flowchart TD accTitle: From symptom to kind of skew accDescr: A decision flow for a slow partitioned store. If load per partition is even, look elsewhere. If one partition is hot, check whether one key dominates. If one key dominates, it is a hot key and needs splitting, caching or rate limits. If many adjacent keys are hot, it is a hot range from a time or sequential key and needs a better key or splitting of the range. S["Slow or throttled store"] --> Q1{"Is load per partition uneven?"} Q1 -->|no| E["Look elsewhere: query, index, network"] Q1 -->|yes| Q2{"Does one key dominate
    the hot partition?"} Q2 -->|yes| H["Hot key:
    split, cache, rate limit"] Q2 -->|no| Q3{"Are the hot keys adjacent
    in time or sequence?"} Q3 -->|yes| R["Hot range:
    change the key or split the range"] Q3 -->|no| M["Several heavy keys share a node:
    move them apart"]

    Figure 2. Telling the kinds of skew apart. How to read this: the second question separates a hot key, which needs a per-key fix, from a hot range, which needs a different key design.

    Mitigation

    Split the key (salting). Add a suffix so one logical key becomes several physical keys. DynamoDB's write-sharding guidance appends a random number, for example 1 to 200 on a date, which spreads writes, or a calculated suffix such as a hash of the order ID, which lets a single item still be found with GetItem. The cost is on reads: to read a whole day you query every suffix and merge. The cache lesson applies the same idea to a hot cache key.

    Cache. A hot key that is read far more than written is a cache problem first: a short-lived local copy takes the reads off the partition. It does nothing for a hot write key.

    Place by load. A system can move hot items apart. DynamoDB's adaptive capacity "rebalances your partitions such that frequently accessed items don't reside on the same partition", and a single busy item can end up alone in a partition, but the ceiling then remains 3,000 reads and 1,000 writes per second for that item's key. Placement fixes neighbours, not one item that is itself too busy.

    Limit. If one tenant or client causes the load, a rate limit protects everyone else, at the price of refusing some of its requests.

    Salting one hot key into four bucketsWriters to sku-42#0; Writers to sku-42#1; Writers to sku-42#2; Writers to sku-42#3; sku-42#0 to Reader; sku-42#1 to Reader; sku-42#2 to Reader; sku-42#3 to ReaderWriters: pick a bucketWriterspick a bucketsku-42#0: partition Asku-42#0partition Asku-42#1: partition Bsku-42#1partition Bsku-42#2: partition Csku-42#2partition Csku-42#3: partition Dsku-42#3partition DReader: queries all four, mergesReaderqueries all four,merges

    Figure 3. Write sharding. How to read this: each write goes to one of four derived keys, so no partition sees more than about a quarter of them. The reader pays for it by collecting all four.

    Mitigation Gain Cost Wrong when
    Salt the key Spreads writes Reads fan out; single-key atomic operations are lost You need an exact value read from one place
    Cache Removes read load Staleness; no help for writes The key is write-heavy
    Load-aware placement Separates hot neighbours Does not raise a single key's ceiling One key exceeds a partition's limit
    Rate limit Protects other users Refuses some traffic The load is legitimate and must be served

    Resharding: changing the number of partitions

    Sometimes the answer is more partitions. Three strategies appear in the documentation.

    A fixed number of partitions, many more than nodes. Redis Cluster has 16,384 hash slots; a key's slot is the CRC16 of the key modulo 16,384, and each node owns a range of slots. Adding a node means moving slots to it, and the documentation says moving slots "does not require stopping any operations". The Dynamo paper's third partitioning strategy divides the hash space into Q equal partitions, decouples partitioning from placement, and notes that fixed ranges let a partition be relocated "as a unit by simply transferring the file"; the cost it names is that changing membership "requires coordination". The key-to-partition mapping never changes; only the partition-to-node map does.

    Dynamic splitting. A partition splits when it grows. CockroachDB's documentation says that "as soon as a range reaches the default range size, it splits into two ranges", and that it rebalances ranges as nodes come and go. MongoDB divides a collection into chunks and says a shard key's cardinality limits "the maximum number of chunks the balancer can create". Splitting by size does not by itself divide the load of one key, because one key lives in one range.

    Consistent hashing with virtual nodes. Cassandra's documentation says virtual nodes assign "multiple tokens in the token ring to each physical node", and that a new node "accepts approximately equal amounts of data from other nodes in the ring". Its own lesson in this module covers the ring.

    When the key itself is wrong, no partition count helps. MongoDB 5.0 and later can change a collection's shard key with reshardCollection, which redistributes the data. That is resharding in its heaviest form.

    Strategy Gain Cost Wrong when
    Fixed partitions Cheap moves of whole partitions; stable mapping The partition count is a ceiling you chose early You picked far too few
    Dynamic splitting Grows with the data Splits can lag a sudden spike One key is the hot one
    Consistent hashing Small data movement on node changes Balance depends on tokens Range scans dominate
    Change the shard key Fixes a wrong key Moves nearly all the data A salt would do

    Online resharding: steps and risks

    An online reshard has to copy live data and switch over without losing writes. Vitess documents one complete workflow, which makes a useful reference even if you do not use it:

    1. Create a workflow naming the source and target shards.
    2. Copy and catch up: the data is copied, then changes made since are streamed; status reports the percentage copied and then the replication lag.
    3. Verify with VDiff, because the documentation says to "confirm that data has been copied over correctly".
    4. Switch traffic with SwitchTraffic, which can switch replica, read-only and primary traffic in any order.
    5. Reverse replication: by default a stream is started "with the current target as the source", so traffic can be switched back.
    6. Complete, which removes the source and is destructive.

    MongoDB's reshardCollection has a similar shape and publishes its limits: applications should tolerate a two-second period in which writes to the collection are blocked, the minimum duration is always 5 minutes, each recipient shard needs free storage of at least twice the collection and index size divided by the shard count, and I/O should be below 50 percent and CPU below 80 percent. The database does not enforce those limits, and running short can mean running out of space or slower operations.

    sequenceDiagram accTitle: Online resharding with copy, verify and cutover accDescr: The source shard keeps serving. A backfill copies existing data to the target while changes made during the copy are streamed. After verification, writes are paused briefly, the last changes are applied, the router flips to the target, and a reverse stream keeps the source current so the move can be undone. participant App as Application participant R as Router participant S as Source shard participant T as Target shard App->>R: writes and reads R->>S: served by the source S->>T: backfill existing rows S->>T: stream changes made during the copy Note over S,T: verify: counts and checksums match R->>R: pause writes briefly S->>T: apply the last changes R->>T: flip routing to the target T->>S: reverse stream, for rollback

    Figure 4. A safe cutover. How to read this: the source serves until the flip. The pause is short because the target is already almost caught up, and the reverse stream is what makes the move reversible.

    The risks are in the seams. Dual writes (the application writing to old and new itself) can diverge when one write fails or two arrive in opposite orders; a change-stream copy, as above, avoids making the application own that consistency. A backfill racing live writes can copy an old row over a newer one (the failure example). A stale router sends writes to the wrong place. No verification lets a silent difference surface weeks later. No way back turns a bad cutover into an outage.

    Realistic example: a flash sale on one stock counter

    This is an educational scenario on ShopStream. A stock counter for one product is a hot key. At peak the shared ShopStream figure is about 58 checkouts per second; a flash sale at ten times that (illustrative) is about 580 orders per second. If an order has three lines (illustrative) and each line decrements that product's counter, the counter sees about 1,740 writes per second, above the 1,000 write units per second that a DynamoDB partition is designed to deliver. No amount of added capacity changes that for one key.

    Failure example: the backfill that undid a write

    During a reshard, a backfill reads row X (version 1) from the source and is slow to write it. Meanwhile a live write updates X to version 2 on the target. The backfill then writes version 1 on top. The target now holds old data, and checks on row counts still pass. Prevention: the backfill must write only if the target row does not exist or has an older version, and the verification must compare contents, not just counts.

    sequenceDiagram accTitle: A backfill overwrites a newer live write accDescr: The backfill reads version 1 of a row from the source. A live write then stores version 2 on the target. The backfill then writes version 1 to the target, replacing the newer value, so the target holds stale data while row counts still match. participant B as Backfill participant S as Source participant L as Live write participant T as Target B->>S: read row X, version 1 L->>T: write row X, version 2 B->>T: write row X, version 1 Note over T: newer data replaced, counts still match

    Figure 5. An unguarded backfill. How to read this: the order of the last two steps is the whole bug. A version check on the backfill write removes it.

    In practice: protecting one product's stock during a flash sale

    This is an educational scenario on ShopStream with illustrative numbers.

    This is a realistic scenario, not a case history.

    The situation

    A promotion on one product is planned for next month. A rehearsal at the expected load shows throttled writes on the product's stock key while the rest of the table is quiet.

    Starting architecture

    Flash sale, starting architecture: one stock keyCheckout service to Stock partition (decrement); Stock partition to Other partitions (unused)Checkout service: 580 orders/sCheckoutservice580 orders/sStock partition: key sku-42Stockpartitionkey sku-42Other partitions: quietOtherpartitionsquietdecrementunused

    Options considered

    Option Gain Cost Wrong when
    A. Rely on the store's adaptive placement No code Cannot exceed the one-key ceiling Load is above it
    B. Split stock into four buckets Each bucket sees about a quarter Reads sum four keys; a bucket can empty first Exact real-time stock matters more than throughput
    C. Queue decrements to one writer Exact, ordered Throughput limited by one writer; added latency Shoppers need an instant answer

    The decision

    Option B, with a fallback: a decrement that finds its bucket empty tries the other three before reporting "sold out". The product page shows a cached, approximate total. Revisit if oversell reconciliation shows more than a handful of corrections per sale.

    Rollout and verification

    Replay the rehearsal load against the new key design. After each run check the invariant: the four buckets plus units sold equal the initial stock. Watch per-bucket write rate and throttle count; the goal is no throttling and no bucket far above a quarter of the load.

    The 2 a.m. failure

    The page reads "Sold out shown, stock remaining". One bucket emptied early and a client that skips the fallback reported "sold out" while the other buckets still held stock.

    First checks: per-bucket remaining counts, and the share of "sold out" responses that did not try the other buckets. Mitigation: enable the fallback path server-side so clients cannot skip it, and rebalance units into the empty bucket. Follow-up: a periodic reconciler that tops up empty buckets from full ones.

    Architect's notes

    • Alert on the busiest partition, not the average, and ask for the top keys in every capacity review.
    • Decide the key's growth direction before launch: time, a counter or a tenant?
    • Pick a partition count with headroom. A fixed count is easy to use and hard to raise.
    • Rehearse the reshard, including rollback, and keep the reverse path until the new layout has run through a peak.

    Common mistakes

    • Fixing skew by adding nodes. More nodes do not help one key.
    • Salting a key that needs an exact read without planning the fan-out.
    • Trusting row counts as verification.
    • Dual writes without idempotent, versioned writes.
    • Resharding a cluster already at its limit. The copy needs spare I/O and storage.

    Key takeaways

    • A partitioned system is as fast as its busiest partition; measure per partition.
    • Skew comes from celebrity keys, time-based keys, sequential keys and low-cardinality keys.
    • Salting, caching, load-aware placement and rate limits each fix a different kind of skew, at a cost.
    • A single key has a ceiling that more partitions cannot raise; only splitting the key or cutting its load can.
    • Fixed partitions move whole pieces and dynamic splitting grows with size; neither divides one hot key.
    • An online reshard is copy, catch up, verify, switch, and keep a way back.

    Practice exercises

    Exercise 1 (30 min). For a key of the form region#date, list which of the four patterns it falls into and propose a better key.

    Exercise 2 (1 hour). Using a per-partition limit of 1,000 writes per second, compute the minimum number of salt buckets for a key that receives 5,000 writes per second, and say what each read now costs.

    Official documentation

    Ask about this article

    Get answers grounded in this post. AI-generated — based on this article, and may be imperfect.

    Scaled AI Weekly

    Enjoyed this? Get more like it every Monday.

    Real architecture decisions, LLMOps patterns that survive production, and engineering leadership advice — from 12+ years of building at enterprise scale. Free. No spam. Unsubscribe anytime.

    Get new articles by email

    Free. We email a confirmation link first, and you can unsubscribe from any email.

    Join engineers building production AI systems

    Comments