QuickCart Stage 10: Secure, Observable and Ready to Run

// SAFE: node-postgres sends the values separately from the SQL text
const result = await client.query(
  "SELECT id, name FROM products WHERE store_id = $1 AND name = $2",
  [storeId, name],
);

Parameters cannot replace identifiers such as column names, so map those from a fixed allow-list (OWASP).

Two relatives matter. Server-side request forgery (SSRF): a feature that imports a store's product photo from a web address makes a server fetch whatever it is given. An attacker supplies an internal address such as the cloud metadata endpoint 169.254.169.254 and gets credentials back. Defend in layers: allow only expected hosts, refuse private and link-local addresses after name resolution and redirects, and block the fetcher's path to internal ranges (OWASP). Insecure deserialisation: in some formats (Java native serialisation, Python pickle) turning outside bytes into objects can run attacker-chosen code, so accept data-only formats such as JSON, validated against a schema. Stage 5 queue messages deserve the same suspicion.

Zero trust and least privilege

The older model trusts the inside of the network. Zero trust (NIST SP 800-207) grants no implicit trust from network location: each request is authenticated and authorised on its own, assuming some part is already compromised. Least privilege gives each identity only what its job needs. For QuickCart:

• Each service has its own identity, mTLS certificate and database user with only the grants it uses. Notifications has no path to the ledger.
• A network policy denies traffic by default and allows named flows. In Kubernetes a NetworkPolicy does this if the network plugin enforces it:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: payments-from-orders-only
  namespace: quickcart
spec:
  podSelector:
    matchLabels:
      app: payments
  policyTypes:
    - Ingress
  ingress:
    - from:
        - podSelector:
            matchLabels:
              app: orders
      ports:
        - protocol: TCP
          port: 8080

Figure 4. Trust boundaries in QuickCart after this stage.

How to read this: solid arrows are allowed, encrypted calls. The red dashed arrow is a call that a network policy and a missing grant both refuse, so a compromised Notification service cannot reach the ledger.

Compliance and privacy

QuickCart holds personal data: names, addresses, order history, courier locations. In the EU the General Data Protection Regulation (GDPR) governs this, and many places have similar laws. The design-relevant ideas:

• Lawful basis (Article 6): each use needs a legal reason, such as a contract (delivering the order), legal obligation (tax records) or consent (marketing).
• Minimisation and storage limitation (Article 5): collect what the purpose needs and keep it no longer, so courier GPS gets short retention.
• Access and erasure (Articles 15 and 17): deleting a person is not one DELETE, because data lives in the orders database, search index, caches, warehouse and backups, and ledger records may have to stay. Keep ledger entries without personal fields, and run a tracked deletion job across every store. Backups age out.
• Residency and transfers (Chapter V): transfers outside the EU need safeguards. The multi-region plan from Stage 9 must state which regions hold personal data, and some national laws are stricter.

Log ids, not names; breaches may need reporting within a fixed time (Article 33). For cards (Stage 6), let the payment provider hold card numbers so QuickCart never stores them, which also narrows its PCI DSS scope.

See what the system is doing

Logs, metrics and traces

You cannot fix what you cannot see. The three signals answer different questions (observability and SLOs goes deeper):

The glue is a correlation id copied through every hop into every log line; a trace id does this job. The W3C Trace Context Recommendation (2021) defines a traceparent header of four dash-separated fields (version, trace id, parent span id, flags), such as 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01. The gateway creates it and each service forwards it.

OpenTelemetry is the vendor-neutral standard: an API and SDK in each service, the Collector, a separate process that receives telemetry (usually over OTLP) and batches, filters and redacts it, and exporters to your chosen backend, so changing backends needs no re-instrumenting. Tail-based sampling keeps every slow or failed trace and a share of healthy ones.

Figure 5. One QuickCart checkout, followed by its trace id.

How to read this: the trace id is made once at the edge and copied downstream, so one search returns the whole story of one checkout. Telemetry goes to the Collector off the request path.

A structured log line from Orders makes that search possible:

{"ts":"2026-11-07T21:14:03.412Z","level":"error","service":"orders","trace_id":"4bf92f3577b34da6a3ce929d0e0e4736","event":"payment_timeout","order_ref":"o_91c2","timeout_ms":2000}

RED (Rate, Errors, Duration) suits request-driven services, such as QuickCart's checkout dashboard. USE (Utilization, Saturation, Errors, from Brendan Gregg) suits resources: the node pool, connection pools and Stage 5 queue lag.

Figure 6. The telemetry pipeline.

How to read this: flow is one-way and off the request path. Alert rules read metrics; humans open dashboards, logs and traces after an alert says where to look.

Alerting and on-call

An alert is a demand on a person's sleep. Page on symptoms users feel, not causes: "checkout fails for 1.5 percent of shoppers" is a symptom, while "CPU on one node is 85 percent" is a cause, often harmless, that belongs on a dashboard.

Symptoms come from SLOs. Suppose checkout's SLO is 99.9 percent success over 30 days (illustrative). At 500,000 orders a day that is 15 million attempts a month, so the error budget is 0.1 percent: 15,000 failures. Burn rate is how fast the budget is spent; 1 uses it exactly over the window. The SRE Workbook chapter Alerting on SLOs pairs a long and a short window:

Check the first row: 30 days is 720 hours, so 14.4 for one hour spends 14.4 / 720 = 2 percent, and for a 99.9 percent SLO it means a 1.44 percent error ratio. As a Prometheus rule (metric name illustrative):

groups:
  - name: quickcart-checkout-slo
    rules:
      - alert: CheckoutFastBurn
        expr: |
          (
            sum(rate(checkout_requests_total{outcome="error"}[1h]))
            / sum(rate(checkout_requests_total[1h]))
          ) > (14.4 * 0.001)
          and
          (
            sum(rate(checkout_requests_total{outcome="error"}[5m]))
            / sum(rate(checkout_requests_total[5m]))
          ) > (14.4 * 0.001)
        labels:
          severity: page
        annotations:
          summary: "Checkout is burning its error budget 14.4 times too fast"
          runbook_url: "https://runbooks.quickcart.example/checkout-fast-burn"

Both windows must exceed the threshold, so a finished spike does not page. Every page links a runbook: what the alert means, the first three checks, how to mitigate (roll back, shed load, fail over) and who to escalate to. Write it for a tired person at 02:00.

On-call is a rota with a primary and a secondary, clear escalation and a cap on pages per shift, plus a blameless postmortem after serious incidents. A page needing no action should be deleted, because alert fatigue is a reliability risk.

Run it

Containers and orchestration

A container image is a read-only, layered package of an application and its dependencies that runs the same everywhere. Build once, promote the same image, and tag by immutable version, never latest. Keep it small, staged and non-root:

FROM node:22-slim AS build
WORKDIR /app
COPY package*.json ./
RUN npm ci
COPY . .
RUN npm run build

FROM node:22-slim
WORKDIR /app
ENV NODE_ENV=production
COPY --from=build /app/package*.json ./
RUN npm ci --omit=dev
COPY --from=build /app/dist ./dist
USER node
EXPOSE 8080
CMD ["node", "dist/server.js"]

The second stage copies only what runs. Pin the base image and scan images.

An orchestrator runs containers across machines and replaces failed ones. In Kubernetes, a Pod is one or more containers scheduled together, a Deployment keeps N copies of a Pod template running and manages updates, and a Service gives Pods one stable address and balances across the ready ones. This Deployment for the Catalogue service carries the settings that matter:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: catalogue
  namespace: quickcart
spec:
  replicas: 6
  selector:
    matchLabels:
      app: catalogue
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 0
  template:
    metadata:
      labels:
        app: catalogue
    spec:
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: topology.kubernetes.io/zone
          whenUnsatisfiable: DoNotSchedule
          labelSelector:
            matchLabels:
              app: catalogue
      containers:
        - name: catalogue
          image: registry.quickcart.example/catalogue:1.42.0
          ports:
            - containerPort: 8080
          resources:
            requests:
              cpu: "1"
              memory: 1Gi
            limits:
              memory: 1Gi
          readinessProbe:
            httpGet:
              path: /ready
              port: 8080
            periodSeconds: 5
            failureThreshold: 3
          livenessProbe:
            httpGet:
              path: /healthz
              port: 8080
            initialDelaySeconds: 10
            periodSeconds: 10

• Probes. Readiness decides whether a Pod receives traffic: while it fails the Pod leaves the Service but is not restarted. Liveness detects a stuck container, and failure restarts it. A startup probe holds both off for slow starters. Keep liveness local: if it tests the database, one blip restarts every Pod.
• Requests and limits. A request is what the scheduler reserves; a limit is a ceiling: over its CPU limit a container is throttled, over its memory limit it is killed.
• Zone spread places replicas evenly across zones, making Stage 9's multi-AZ promise real. Rolling update with maxUnavailable: 0 starts new Pods before removing old ones.

A HorizontalPodAutoscaler (HPA) adds and removes replicas by the documented formula desiredReplicas = ceil[currentReplicas x (currentMetricValue / desiredMetricValue)]: 10 Pods at 90 percent average CPU against a 60 percent target gives ceil(10 x 90 / 60) = 15. Utilisation is measured against CPU requests, so Pods without one give it nothing to act on. It reacts only after metrics arrive and new Pods start, so it follows ramps well and spikes badly, which is why the capacity plan keeps headroom.

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: catalogue
  namespace: quickcart
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: catalogue
  minReplicas: 6
  maxReplicas: 30
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 60

Infrastructure as code

Infrastructure as code (IaC) describes networks, databases, clusters and permissions in version-controlled files instead of console clicks. The tools are declarative: the file states the desired end state and the tool works out the steps. The gains:

• Review. A change is a pull request, and a plan shows what will be created, changed or destroyed first.
• Repeatability. Staging matches production, and a lost region can be recreated, as Stage 9's recovery plan needs.
• Drift detection. Drift is reality diverging from the files, usually after a manual incident fix. A scheduled plan reports it.

The Terraform loop is two commands:

terraform plan -out=tfplan   # compare files to real state, print the proposed changes
terraform apply tfplan       # execute exactly the plan that was reviewed

A plan ends with a summary such as Plan: 1 to add, 1 to change, 0 to destroy.; reviewers look hardest at the destroy count. terraform plan -detailed-exitcode exits 2 when changes are pending, which a scheduled job can use to flag drift. Keep state in a remote, encrypted, locked store, since it can contain secrets. GitOps extends the idea to the cluster: manifests live in Git and a controller reconciles them.

Deployment strategies

A deployment puts new code on servers; a release exposes it to users. Separating them makes change safe.

QuickCart uses rolling updates for low-risk services and canary releases with automatic rollback for Orders and Payments. The Kubernetes Deployment controller does not roll back by itself (past its progress deadline it only reports a condition), so automatic analysis comes from a progressive-delivery controller such as Argo Rollouts or Flagger, or from traffic splitting in a mesh.

Figure 7. The delivery pipeline.

How to read this: application changes flow along the top into the cluster. Infrastructure changes take the lower path with their own plan and review. Nothing reaches production except through Git and the pipeline.

Figure 8. How a canary decides.

How to read this: the percentages and ten-minute hold are illustrative. The shape matters: small first, compare, widen, return automatically on failure.

Database migrations during deploys are where releases hurt, because old and new code share one database mid-rollout. The rule is expand, then contract. To replace orders.legacy_notes with delivery_notes: (1) expand by adding the nullable column, which both versions tolerate; (2) release code that writes both and reads the new one; (3) backfill old rows in small batches, per shard (Stage 4); (4) contract by dropping the old column once no running version reads it.

-- Expand: safe while old and new code run together
ALTER TABLE orders ADD COLUMN delivery_notes text;

-- Backfill in small batches (repeat with the next id range)
UPDATE orders SET delivery_notes = legacy_notes
WHERE id BETWEEN 1 AND 10000 AND delivery_notes IS NULL;

-- Build any new index without blocking writes
CREATE INDEX CONCURRENTLY idx_orders_delivery_notes ON orders (delivery_notes);

-- Contract: only after no running version reads legacy_notes
ALTER TABLE orders DROP COLUMN legacy_notes;

Adding a nullable column is a quick metadata change in PostgreSQL, and since version 11 so is adding one with a non-volatile default; check your engine. Never drop a column in the release that stops using it, or a code rollback meets a schema it no longer matches.

Capacity planning

Capacity planning turns demand into resources: forecast, measured capacity per unit, units needed, headroom (capacity and cost has the method). Demand is from the project: about 2,300 browsing requests and 58 checkouts per second at peak. Unit capacity must come from a load test; these are illustrative assumptions standing in for load-test results: a browsing Pod sustains 150 requests per second and a checkout Pod 20 checkouts per second, each requesting 1 vCPU.

The last row is the sizing rule: capacity must hold after the worst failure you promised to survive. That gives 36 Pods. If an 8 vCPU node offers about 7 vCPU to Pods after system reservations (illustrative), that is 7 Pods per node, so 36 / 7 = 5.1, rounded up to 6 nodes, 2 per zone. Other services add more the same way.

Headroom also covers rolling-update surge and cold caches. Forecast from real weekly and holiday history, and rerun the plan quarterly.

Cost optimization

Cloud cost is an architecture property. Know the drivers, then match levers:

Right-sizing sets requests from observed use plus margin, not guesses. Reserved or committed-use pricing trades a one- or three-year commitment for a lower rate than on-demand: commit to the steady floor (minimum nodes at 04:00) and leave the peak on demand. Interruptible (spot) capacity can be reclaimed by the provider, so it suits stateless workers and batch analytics, not a primary database. Discounts and terms differ by provider, so read each pricing page rather than trusting a figure here.

Track cost per unit of value: monthly cloud spend divided by orders (15 million a month at the project's volume), shown as cost per 1,000 orders. If orders double and the bill doubles, nothing improved. Cutting a replica or retention period trades money for risk; the SLO says how much is acceptable.

Decisions and trade-offs

In practice: QuickCart at this stage

The situation

QuickCart has 5 million monthly active users, heading to 10 million (illustrative). A six-engineer platform team shares an on-call rota of four. A security review found that tokens carry no scopes and three services share one database password, and a key leaked in a ticket took two days to rotate by hand. Releases are weekly and manual.

Starting architecture

The starting point is Figure 1.

Options considered

The decision

Rollout and verification

1. Instrument the gateway and Orders with OpenTelemetry, and find one checkout from a single trace id.
2. Define the checkout SLO, write burn-rate alerts and runbooks, and run a fire drill that reaches the on-call phone.
3. Move secrets into the manager and rotate each one once on purpose.
4. Put OIDC in front of the gateway behind a flag, verifying tokens in shadow mode against the old tokens. Test that one shopper requesting another's order is refused.
5. Move Catalogue to Kubernetes, then the rest, Payments last. Run a game day that removes a zone to test the capacity plan.
6. Enable canaries for Orders and Payments and push a bad build in staging to prove the rollback.

The 2 a.m. failure

At 02:07 the on-call phone pages: "Checkout is burning its error budget 14.4 times too fast." Orders is returning errors on calls to Payments.

Service certificates are short-lived and renewed by an issuer component. Weeks earlier a credential the issuer uses to reach its certificate authority had quietly expired, and no alert watched renewal failures. Certificates expired in waves, and by 02:00 Orders could no longer complete a handshake with Payments.

The runbook's first three checks: search logs by a failing trace id (the error reads "certificate has expired"), read certificate expiry on the Payments Pods, and read the issuer's logs. Mitigation is to restore the issuer's credential and force re-issue. Rolling back the last release would not help. Follow-ups: alert when any certificate is within days of expiry (a cause alert is right here, because the lead time is long), alert on renewal failures, and rehearse rotation quarterly.

This is a realistic scenario, not a case history.

Architecture after this stage

Muted boxes are earlier stages; highlighted boxes are new.

Figure 9. QuickCart after Stage 10.

How to read this: requests flow left to right, authenticated and encrypted at each hop. The top row holds identity and secrets. The bottom row is operations: telemetry flows to alerts, and the pipeline delivers changes into the cluster.

What QuickCart looks like now

Architect's notes

• Order the work: see, then secure, then change. Telemetry makes later steps verifiable.
• Decide identity early. The IdP, token lifetime and who verifies what are costly to change; mesh features can wait.
• Rotate every secret once before you must. Until you have, you do not know it can be.
• Page on burn rate, and size capacity for the failure you promised to survive.

Common mistakes

• Authenticating without authorizing, or trusting the algorithm named in a token header.
• Long-lived JWTs with no revocation plan.
• Secrets in images, repositories or logs; string-built SQL; fetching user-supplied URLs from inside the network.
• A liveness probe that checks the database. One blip restarts every Pod.
• Alerting on causes, and unbounded ids in metric labels.
• Sizing for steady state, and dropping a column in the release that stops using it.

Key takeaways

• Authentication proves who; authorization decides what, per object, in the service that owns the data.
• Use the authorization code flow with PKCE, validate JWT algorithm, issuer, audience and expiry, and keep access tokens short-lived.
• Encrypt in transit (TLS, mTLS) and at rest (envelope encryption); apply zero trust and least privilege.
• Keep secrets out of images and repositories, inject them at runtime, and prefer short-lived ones.
• Parameterise queries, validate at the boundary, and treat server-side fetches and deserialisation as attack surface.
• Privacy law shapes design: lawful basis, minimisation, deletion across every store. Take legal advice.
• Alert on SLO burn rate, give every page a runbook, and tie signals together with trace ids.
• Migrations go expand, then contract; capacity is sized for the worst promised failure.

Practice exercises

Exercise 1 (30 minutes): Redo the browsing capacity table for 3,000 requests per second at peak, 120 per Pod, a 60 percent target, and the loss of one of four zones with survivors at most 80 percent busy.

Exercise 2 (1 hour): Write the authorization rules for a store manager exporting a monthly sales report, and the test proving another store's manager is refused.

Exercise 3 (2 hours): For a 99.5 percent browse-latency SLO (illustrative), compute the error ratio at which burn rate 6 fires and draft the runbook's first three checks.

Exercise 4 (half a day): Plan the migration that splits orders.delivery_address into street, city and postcode: the four releases and the rollback at each point.

Related reading

• QuickCart Stage 9: Surviving Failure Without Losing Orders, the failure model this stage builds on.
• Design Any System: A Repeatable Method With Three New Problems, the next stage.
• System Design: Security and Observability and SLOs for the longer treatments.
• System Design: Capacity and Cost for the full planning method.
• System Design: API Gateway and Service Mesh for mTLS and traffic shifting, and Multi-Region and Disaster Recovery for recovery targets.

Official documentation

• RFC 9700: Best Current Practice for OAuth 2.0 Security
• RFC 8725: JSON Web Token Best Current Practices
• Kubernetes documentation: Deployments
• Google SRE Workbook: Alerting on SLOs
• OpenTelemetry documentation: Signals
• NIST SP 800-207: Zero Trust Architecture