Auto-Scaling and Kubernetes HPA: Metrics, Cooldowns, Scale-In

What you'll learn: By the end of this lesson you will be able to describe an autoscaler as a control loop with delays, work out the replica count the Kubernetes Horizontal Pod Autoscaler (HPA) will choose, pick a metric that moves with capacity, tell a cooldown, an instance warm-up and a stabilization window apart, and name the ways scaling in goes wrong.

A supermarket opens extra checkout lanes when queues grow and closes them when the shop empties. A good manager watches the queue, waits a few minutes before trusting a sudden rush, and never closes a lane while a customer is still scanning at it. An autoscaler is that manager: a measurement, a target, a rule for how many lanes to open, and some patience.

Scaling Basics and ShopStream Stage 2 cover auto scaling in one section each. This lesson goes deeper: how the decision is made, which metric to feed it, which timers stop over-reaction, and where scaling in does damage.

Verified against the Kubernetes documentation (pages read on 2026-11-08, feature states quoted as the pages give them for v1.37) and the Amazon EC2 Auto Scaling User Guide (read on 2026-11-08).

The core idea

An autoscaler is a feedback loop: measure a signal, compare it with a target, change the number of instances, then wait to see the effect. Every part of that sentence can go wrong. The signal may not move when capacity changes. The change takes time to arrive. Without patience, the loop overshoots and oscillates.

Three labelled illustrations run through the lesson: a minimal example (the HPA formula, with a TypeScript model that was run), a realistic example (ShopStream's Saturday promotion) and a failure example (scale-in that kills requests in flight).

Where the autoscalers sit

In Kubernetes, the HPA changes the number of Pod replicas of a workload, a node autoscaler (the docs name Cluster Autoscaler and Karpenter) adds or removes machines, and the Vertical Pod Autoscaler changes a Pod's resource requests. They chain: the HPA asks for more Pods, Pods that fit nowhere stay Pending, and the node autoscaler provisions nodes for them. Node autoscalers look only at requests, not real usage, so correct requests matter.

Figure 1. The two scaling layers. How to read this: the HPA reads usage and sets the replica count. If the new replicas cannot be scheduled, a node autoscaler adds machines. Each hop between the traffic and a ready Pod takes time.

On AWS the same ideas apply to an EC2 Auto Scaling group with scaling policies; the lesson names the product when a rule is specific to it.

Autoscaling is a chain of delays

Every step between "load rose" and "a new instance serves traffic" adds delay, and the loop sees the effect only after the last one.

Figure 2. Reactive scaling is always late. How to read this: each arrow adds delay. The HPA's evaluation interval is 15 seconds by default (--horizontal-pod-autoscaler-sync-period), and the others depend on your metrics pipeline, image size, node availability and start-up time. Headroom must cover the whole chain.

Two consequences: reactive scaling needs spare capacity while new capacity arrives (a 100 percent target is a bad target), and the loop must not act on distorted numbers such as those from instances still warming up. The timers below address that.

The HPA decision, step by step

The HPA documentation gives the core rule: desiredReplicas = ceil[currentReplicas x (currentMetricValue / desiredMetricValue)]. Scaling is skipped when the ratio is within a tolerance of 1.0, 0.1 by default; the docs mark a per-HPA tolerance (behavior.scaleUp.tolerance, behavior.scaleDown.tolerance) stable since v1.37. The surrounding rules matter as much as the formula:

• Several metrics: the calculation runs per metric and the largest replica count wins. If a metric cannot be read and the others suggest scaling down, scaling is skipped, but scaling up still works.
• Pods not ready: when scaling on CPU, a Pod still initializing is set aside, and on a scale-up the controller assumes it uses 0 percent of the target, so it does not keep adding Pods while earlier ones start. Missing metrics are assumed at 100 percent on a scale-down and 0 percent on a scale-up. Two cluster-wide settings govern this: initial readiness delay (30 seconds) and CPU initialization period (5 minutes).
• Utilization is relative to requests: if a container lacks a CPU request, utilization is undefined and the autoscaler takes no action for that metric.

Figure 3. The HPA pipeline. How to read this: the formula is only the first step. Tolerance, the stabilization window and the rate policies decide how fast the answer is applied; the min and max have the last word.

Minimal example. Six Pods run at 90 percent average CPU against a 60 percent target. 6 x 90 / 60 = 9, so the HPA asks for 9. At 64 percent the ratio is about 1.07, inside the 0.1 tolerance, so nothing changes. A model of the rule in TypeScript (a teaching model, not the controller's code; run with Node):

function desiredReplicas(current: number, metric: number, target: number, tolerance = 0.1): number {
  const ratio = metric / target;
  if (Math.abs(ratio - 1) <= tolerance) return current; // within tolerance: no action
  return Math.ceil(current * ratio);
}

console.log("90% vs 60%, 6 pods ->", desiredReplicas(6, 90, 60));
console.log("64% vs 60%, 6 pods ->", desiredReplicas(6, 64, 60));
console.log("20% vs 60%, 12 pods ->", desiredReplicas(12, 20, 60));
// 9 pods: 6 ready at 90%, 3 not ready, assumed 0% while scaling up.
console.log("with 3 pods not ready ->", desiredReplicas(9, (6 * 90 + 3 * 0) / 9, 60));

Output from a real run:

90% vs 60%, 6 pods -> 9
64% vs 60%, 6 pods -> 6
20% vs 60%, 12 pods -> 4
with 3 pods not ready -> 9

The last line is the not-ready rule at work: with 3 of 9 Pods still starting and assumed idle, the average falls to 60 percent, so the controller does not order a second round of Pods.

Choosing the metric

The EC2 target-tracking page states the test: the metric must describe how busy an instance is, and rise or fall in proportion to the number of instances. Only then does "double the instances" mean "halve the metric", which the formula assumes. It lists metrics that fail: the load balancer's total RequestCount, load balancer Latency, and raw SQS ApproximateNumberOfMessagesVisible (a custom metric of messages per instance can work).

Figure 4. Metric choice by what the tier does. How to read this: ask what saturates first. A raw latency number says users hurt, not how many instances would fix it.

AWS predefines ASGAverageCPUUtilization, ASGAverageNetworkIn, ASGAverageNetworkOut and ALBRequestCountPerTarget. EC2 metrics arrive every five minutes by default; the guide recommends detailed monitoring (one minute).

Three timers that look alike and are not

Teams often say "cooldown" for all of these. They protect against different mistakes.

The stabilization window turns a noisy series into a rolling maximum. A model, with a window of four evaluations (illustrative; the real default is 300 seconds, about 20 evaluations at 15 s):

sync 1: raw 12, stabilized 12
sync 2: raw 9, stabilized 12
sync 3: raw 7, stabilized 12
sync 4: raw 6, stabilized 12
sync 5: raw 6, stabilized 9
sync 6: raw 6, stabilized 7

The raw recommendation drops to 6 at sync 4, yet the count stays at 12 until the high values leave the window: scale out fast, scale in slowly. The HPA docs call the alternative thrashing or flapping and name the window and tolerance as remedies; AWS advises an adequate margin between scale-out and scale-in thresholds. AWS's rule for several target-tracking policies has the same bias: scale out if any wants to, scale in only if all agree. Instances still warming up count toward the next scale-out, so repeated breaches produce one activity, not a pile.

Scaling in: where the damage happens

Adding capacity rarely hurts anyone. Removing it does, because the instance being removed may be in the middle of work.

Figure 5. The shutdown path of a Pod. How to read this: from the Kubernetes Pod lifecycle docs. The default grace period is 30 seconds. Removal from the endpoints runs at the same time as the Pod's own shutdown, not before it, so the application must keep serving what it already accepted and stop taking new work.

What the sources say, and what follows:

• Graceful termination is not automatic. The Pod gets a TERM signal and terminationGracePeriodSeconds (30 seconds by default); after that the kubelet forces shutdown. An application that ignores the signal, or exits at once, drops in-flight requests. A preStop hook that overruns the grace period gets only a 2-second extension, so raise the grace period to fit it. Because endpoint removal runs concurrently with shutdown, allow for a short window in which a balancer may still send traffic, and test it.
• EC2 has the same problem. Elastic Load Balancing waits 300 seconds for deregistration (connection draining) by default, and lifecycle hooks can add delay.
• Scale-in protection controls which EC2 instances may be terminated, but does not stop health-check replacement, Spot interruptions or manual termination.
• Node consolidation is disruptive too: the Kubernetes docs say removing non-empty nodes terminates their Pods, which are recreated elsewhere.
• Survivors take the load, and caches and warm connections leave with the removed instances.

Beyond reacting: scheduled and predictive scaling

Reactive scaling cannot beat Figure 2 for a spike you can predict. EC2 offers scheduled actions and predictive scaling, which needs at least 24 hours of metric data, analyses up to 14 days, forecasts 48 hours ahead and refreshes every 6 hours. It starts in forecast-only mode so you can judge it, and in forecast-and-scale mode it only adds capacity: dynamic policies still do the scale-in. With several policies active, the group takes the maximum.

Scale to zero exists in limited forms: Kubernetes allows minReplicas: 0 only with object or external metrics (beta since v1.37), and an EC2 group can reach zero when its metric emits real zeros and its minimum is 0. The first request then pays a cold start.

Limits that autoscaling cannot remove

The Kubernetes docs say making every Pod schedulable is not always attainable because of configured limits or missing cloud capacity. More Pods also open more database connections, as in ShopStream Stage 2 and Capacity Planning and Cost. And when retries multiply load, scaling out feeds them: see Backpressure and Load Shedding and Retry Storms.

In practice: ShopStream's Saturday promotion

This is an educational scenario on the course's running example. Every number is illustrative unless stated.

This is a realistic scenario, not a case history.

The situation

ShopStream runs on Kubernetes. A push notification, "Free delivery today", is expected to double the browsing peak from about 1,150 to about 2,300 requests per second within minutes. One app Pod handles about 200 requests per second at saturation (the figure Stage 2 uses). The database carries about 400 connections in this scenario, with 20 per Pod.

Starting architecture

Figure 6. The starting point. How to read this: the HPA scales on CPU only, up to 20 Pods. Twenty Pods would hold 400 connections, which is the whole database budget.

Options considered

The decision

Option C, with limits chosen from the database, not from the Pod count the traffic seems to want. A requests-per-Pod target of 150 (75 percent of the 200 saturation figure) gives 2,300 / 150 = 15.3, so 16 Pods; set minReplicas 8 (8 x 150 = 1,200, enough for the normal peak) and maxReplicas 16 (16 x 20 = 320 connections, under the 400 budget). The proportional rule on the spike: 8 Pods each taking 287.5 requests per second give 8 x 287.5 / 150 = 15.3, rounded up to 16. A scheduled raise of the minimum before the push (Option C) has the Pods ready in advance.

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: shopstream-app
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: shopstream-app
  minReplicas: 8
  maxReplicas: 16
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 60
  - type: Pods
    pods:
      metric:
        name: requests-per-second   # illustrative name, served by a custom-metrics adapter
      target:
        type: AverageValue
        averageValue: "150"

The first version leaves behavior at its defaults and the app exits as soon as it receives the stop signal.

Rollout and verification

Apply it in staging, replay a doubled load, and watch kubectl get hpa for the target column and replica count. Verify that the count reaches 16 and not more, database connections stay under budget, and no 5xx appear while Pods are removed. Dashboards: requests per Pod, desired versus ready replicas, Pending Pods, database connections.

The 2 a.m. failure

The page reads "checkout errors up, orders missing" an hour after the promotion ends. The HPA waited out the 300-second window, then dropped from 16 to 8 in one evaluation, because the default scale-down policy allows removing 100 percent of replicas per 15 seconds. The containers exited on the stop signal and cut eight Pods' worth of in-flight payments. Checks: kubectl get hpa and the HPA's events for the time of the drop, kubectl get pods for terminations, and the payment ledger for authorised payments with no order. Mitigation: raise minReplicas for the night and reconcile orphaned payments (the idempotency keys in the payments lesson make a retry safe). Lasting fix: a slower scale-down, with values chosen for this scenario (the structure follows the documented defaults block), plus a graceful shutdown that finishes in-flight checkouts, with terminationGracePeriodSeconds above the longest one, tested by deleting a Pod under load.

  behavior:
    scaleDown:
      stabilizationWindowSeconds: 600
      policies:
      - type: Pods
        value: 2
        periodSeconds: 60

Architect's notes

• Load test first. A target is a measured saturation point times a margin, not a guess.
• Set the maximum from the weakest dependency and write down which one it is.
• Set CPU requests on every container the HPA watches.
• Alert on the scaler itself: desired above maximum, Pending Pods, a metric that stopped arriving.
• Check versions. Tolerance and scale-to-zero are recent Kubernetes features.

Common mistakes

• Scaling on a metric that does not fall when instances are added, such as raw queue length.
• A target of 100 percent, which leaves no room for the delay chain.
• Default scale-down for long requests, or no handling of the stop signal.
• A maximum chosen from the traffic, not from the database.
• Equal thresholds for scale-out and scale-in, which invites flapping.

Key takeaways

• An autoscaler is a delayed feedback loop; headroom covers the delay.
• The HPA rule is ceil[current x metric / target], skipped inside a tolerance of 0.1 by default; with several metrics the largest count wins.
• A good metric moves in proportion to the instance count; latency and raw queue length do not.
• Cooldown, warm-up, stabilization window and tolerance are different brakes.
• Scale-in is the risky direction: graceful termination, draining and protection matter.
• Maximums come from the weakest dependency.
• Predictive scaling adds capacity only; dynamic policies still do scale-in.

Practice exercises

Exercise 1 (20 min). Eleven Pods run at 45 percent CPU against a 60 percent target. Compute the desired count, then the count at 55 percent and say why.

Exercise 2 (30 min). Extend the TypeScript model with a scale-down rate limit of 2 Pods per evaluation and print the path from 16 to 8.

Exercise 3 (45 min). For a queue-based photo-thumbnail worker, choose a metric and justify it against the "moves with capacity" test.

Exercise 4 (1 hour). Write the shutdown sequence for a service with 40-second requests: grace period, preStop, what the app does on the stop signal, and the test that proves it.

Related reading

• Scaling Basics: the short version, with vertical versus horizontal and statelessness.
• ShopStream Stage 2: Scale Reads: auto-scaling on the read path.
• Load Balancing: how traffic reaches the instances you add.
• Capacity Planning and Cost: headroom, quotas, load tests.
• Message Queues: queue depth and age as scaling signals.
• Backpressure and Load Shedding and Retry Storms: when scaling out is the wrong answer.

Official documentation

• Kubernetes: Horizontal Pod Autoscaling (algorithm, behavior, tolerance, scale to zero).
• Kubernetes: HPA walkthrough (manifest fields).
• Kubernetes: Pod lifecycle: termination of Pods and Node autoscaling.
• Amazon EC2 Auto Scaling User Guide: target tracking, step and simple scaling, cooldowns.
• Amazon EC2 Auto Scaling User Guide: default instance warmup, scale-in protection, predictive scaling.