Encryption at Rest and in Transit: TLS, Envelope Keys, Rotation

What you'll learn: By the end of this lesson you will be able to explain what TLS 1.3 gives a connection and where it stops, name the layers of at-rest encryption and the attacker each one defeats, walk through envelope encryption with a key management service (KMS) including the encryption context, state precisely what key rotation does and does not change, and decide where ShopStream's keys live and what happens when one is lost.

You can protect a letter two ways: seal it in an envelope for the journey, or lock it in a filing cabinet once it arrives. The envelope stops the postman reading it and the cabinet stops a burglar who carries off the drawer. Neither helps if the cabinet key is taped to the front. Encryption does not remove the need for trust, it moves trust into key custody.

Security for Distributed Systems lists the layers and ShopStream Stage 10 sketches envelope encryption in four steps. This lesson goes deeper: what TLS 1.3 changed, what each at-rest layer defeats, how a KMS is used, and what rotation, deletion and loss do. TLS and HTTPS basics are a prerequisite lesson (planned, so no link).

The core idea

Encryption protects data from anyone who can see the bytes but does not hold the key. It does nothing against anyone who legitimately holds the key, including your own running application.

That sentence tells you what to ask of every design: who can read the bytes, who holds the keys, and are those the same people?

Three illustrations run through the lesson:

• Minimal example: envelope encryption with AES-GCM in JavaScript, with real output.
• Realistic example: ShopStream's delivery addresses and payment tokens.
• Failure example: a disk-encrypted database that still leaks through an application bug, and a key that disappears.

In transit: what TLS 1.3 gives a connection

Transport Layer Security (TLS) protects a connection between two endpoints. TLS 1.3 was specified in RFC 8446; at the time of writing the IETF has published RFC 9846, which obsoletes RFC 8446 as "a minor update to TLS 1.3 that retains the same version number and is backward compatible". The abstract says TLS is designed to prevent eavesdropping, tampering and message forgery; the server is always authenticated and the client optionally.

Figure 1. The TLS 1.3 handshake in its three phases.

How to read this: one round trip before application data. CertificateVerify is a signature over the whole handshake made with the private key matching the certificate, which is how the server proves it is the named host.

What changed from TLS 1.2, as listed in RFC 8446 section 1.2:

• Static RSA and static Diffie-Hellman suites were removed, so every public-key key exchange provides forward secrecy: stealing a server's long-term key later does not unlock earlier recorded traffic.
• Handshake messages after ServerHello are encrypted.
• Only authenticated-encryption (AEAD) algorithms remain, with legacy algorithms pruned.
• 0-RTT was added: a client resuming with a pre-shared key may send data in its first flight.

Treat 0-RTT carefully. RFC 8446 section 2.3 says 0-RTT data is not forward secret and has no guarantee of non-replay between connections. So anything sent as early data can arrive twice: allow it only for requests safe to repeat (see idempotent and commutative operations).

Two more rules. RFC 8996 says TLS 1.0 and 1.1 "MUST NOT be used". And HTTP Strict Transport Security (RFC 6797) lets a site tell browsers, via the Strict-Transport-Security header with a required max-age, to rewrite http to https and give users no click-through on certificate errors. It is ignored over plain HTTP, so the very first visit stays exposed.

Where TLS stops

TLS protects one hop between two endpoints. The moment a load balancer terminates it, the next hop is whatever you built. A design must name each hop.

Figure 2. TLS terminates at hops, not at the system.

How to read this: every arrow is a separate TLS connection with its own certificate check. Without the re-encrypt step the design has a plaintext hop exactly where an internal attacker would stand.

At rest: which layer stops which attacker

Encryption at rest protects stored bytes. The PostgreSQL 18 manual's "Encryption Options" page lists what each option defeats and what it does not:

That page does not describe built-in transparent data encryption (version 18, at the time of writing); other databases differ, so read your product's page.

Cloud providers encrypt by default: Azure with platform-managed keys, Google Cloud using envelope encryption with its internal key service. Customer-managed keys are the option that, in Azure's words, gives "extra control at the cost of increased management responsibility". AWS documents three key types, and the differences are the decision:

Customer managed keys carry a monthly fee and per-use charges, and you own their lifecycle: choose them when a requirement says you must control or audit the key.

Envelope encryption

A KMS creates keys, keeps them in a protected boundary and performs cryptographic operations on request. Sending all data through it would be slow, and Google's documentation notes that Cloud KMS limits Encrypt and Decrypt input to 64 KiB.

Envelope encryption solves this: a data encryption key (DEK) encrypts the data locally, and a key encryption key (KEK) that never leaves the KMS encrypts ("wraps") the DEK.

Figure 3. Encrypting and decrypting one record.

How to read this: the bulk data never goes to the KMS, and the plaintext key exists in memory only for the moment it is used.

AWS documents this flow: GenerateDataKey returns a plaintext copy and a wrapped copy; you remove the plaintext key from memory as soon as possible; KMS "does not store, manage, or track your data keys". Google's steps match, advising AES-256 in GCM mode and never storing a plaintext DEK. Azure uses the same hierarchy and adds a consequence: because decrypting DEKs needs the KEK, disabling the KEK cryptographically erases the data, and makes every dependent service inaccessible.

The encryption context

AWS KMS accepts an encryption context: non-secret key-value pairs bound to the ciphertext as additional authenticated data, which must be supplied again to decrypt. It is not secret and appears in plaintext in CloudTrail logs, so it must not hold sensitive data. A key policy can allow Decrypt only when a given pair is present. Used well, a ciphertext copied to another row or tenant fails to decrypt.

Illustration 1 (minimal). A teaching sketch with Node's built-in AES-256-GCM. The "KMS" is one in-memory key, which a real system must not do: this shows the data flow and the context check, not key custody. It was run with Node on 2026-11-08.

import { randomBytes, createCipheriv, createDecipheriv } from "node:crypto";

// Teaching sketch: a real system keeps the KEK inside a KMS, never in application memory.
const kek = randomBytes(32);

function seal(key, plaintext, aad) {
  const iv = randomBytes(12);
  const c = createCipheriv("aes-256-gcm", key, iv);
  c.setAAD(Buffer.from(aad));
  const ct = Buffer.concat([c.update(plaintext), c.final()]);
  return { iv, ct, tag: c.getAuthTag() };
}
function open(key, box, aad) {
  const d = createDecipheriv("aes-256-gcm", key, box.iv);
  d.setAAD(Buffer.from(aad));
  d.setAuthTag(box.tag);
  return Buffer.concat([d.update(box.ct), d.final()]);
}

function encryptRecord(plaintext, context) {
  const dek = randomBytes(32);                      // fresh data key per record
  const data = seal(dek, Buffer.from(plaintext), context);
  const wrappedDek = seal(kek, dek, context);       // the "KMS" wraps the DEK under the KEK
  return { data, wrappedDek, context };             // the plaintext DEK is not stored
}
function decryptRecord(rec, context) {
  const dek = open(kek, rec.wrappedDek, context);
  return open(dek, rec.data, context).toString();
}

const plaintext = "12 Example Road, Flat 3";
const rec = encryptRecord(plaintext, "order=1001");
console.log("stored fields:", Object.keys(rec).join(", "));
console.log("plaintext bytes:", plaintext.length, "ciphertext bytes:", rec.data.ct.length);
console.log("wrapped key bytes:", rec.wrappedDek.ct.length, "tag bytes:", rec.data.tag.length);
console.log("decrypt, right context:", decryptRecord(rec, "order=1001"));
try {
  decryptRecord(rec, "order=1002");
} catch (e) {
  console.log("decrypt, wrong context: failed:", e.message);
}

Output:

stored fields: data, wrappedDek, context
plaintext bytes: 23 ciphertext bytes: 23
wrapped key bytes: 32 tag bytes: 16
decrypt, right context: 12 Example Road, Flat 3
decrypt, wrong context: failed: Unsupported state or unable to authenticate data

Each record has its own DEK, only the wrapped DEK is stored, and GCM authenticates both ciphertext and context, so a record presented under the wrong order id is refused. OWASP's guidance is AES with at least a 128-bit key (ideally 256) in an authenticated mode such as GCM, and never a home-made algorithm.

Key rotation: what it does and does not do

Rotation replaces key material with new material. The AWS KMS documentation is precise about the semantics, and they generalise:

The page adds that a wrapping key is used far less often than a data key and is "almost never reused enough to risk key exhaustion", so rotation is mostly hygiene, compliance and event response. OWASP lists four triggers: a compromise (including a key holder leaving), a set cryptoperiod, nearing the mode's usage limits, and a significant new attack on the algorithm.

Figure 4. Which key to replace.

How to read this: the order of damage matters. Exposed data keys need the data re-encrypted, and an exposed KEK needs every key it wrapped replaced, because old key material still decrypts old wrapped keys.

Disabling keys is dangerous. Azure notes that disabling a KEK makes dependent databases, storage and disks inaccessible, and that a key restored from backup into another vault is an independent copy the disable does not touch. Key Vault guidance: enable soft delete (a 7 to 90 day recovery window) and purge protection, and for a suspected compromise rotate to a new key and migrate dependent services before disabling the old one.

Choosing: who holds the keys

In practice: delivery addresses and payment tokens

This is the realistic example, on ShopStream, the course's running system. It is an educational scenario with illustrative numbers.

This is a realistic scenario, not a case history.

The situation

ShopStream stores delivery addresses and the tokens a payment provider returns (not card numbers). The database volume is encrypted by the cloud default, and a reviewer asks what that defends against: a stolen disk or a leaked snapshot, but not a SQL injection bug, a misused replica credential or an over-privileged analyst query. The team wants addresses and tokens protected even from readers of the tables.

Starting architecture

Figure 5. Where the keys live.

How to read this: the database and its backups hold only ciphertext and wrapped keys. The authority to read lives in the KMS policy, where each unwrap is logged.

Options considered

• A. Volume encryption only. Free, but it does not stop table readers or injection.
• B. Envelope-encrypt two fields with a KMS. Table readers see ciphertext and each unwrap is audited; the cost is a KMS call per wrap or unwrap and no SQL search on those fields.
• C. Encrypt every column. Maximum coverage, with cost, complexity and broken queries for columns that are not sensitive.

The decision

Option B, with a separate KEK for each field so access can be granted and revoked separately, and the order id as the encryption context so a ciphertext copied to another order fails. At the course's assumed peak of about 58 checkouts per second (illustrative), one wrap per order is about 58 KMS calls per second, plus unwraps on reads; check the provider's request quotas. Everything on the wire uses TLS, with mutual TLS between services.

Rollout and verification

Deploy writes first with dual-format reads, backfill existing rows in batches, then disable plaintext reads. Verify in a staging copy that no address text appears in raw table pages, that a decrypt with a wrong order id fails, and that KMS audit entries match application reads. Monitor KMS errors, unwrap latency and any Decrypt outside the service role.

The 2 a.m. failure

Illustration 3 (failure). Address reads fail and checkout returns errors. A cleanup job disabled what looked like an unused key: the address KEK. Check the key state and the audit log of who changed it; mitigate by re-enabling it, which works because nothing was deleted. The lasting fixes are soft delete and purge protection where offered, an alert on any key-state change, and retiring keys only after their services have migrated. The same failure with a deleted key and no backup would be permanent data loss.

Architect's notes

• Name the attacker per layer. "Encrypted at rest" alone is a checkbox.
• Put the KEK policy in the design review. Who can call Decrypt is the real access control.
• Know the hot path: how many KMS calls per request, and what if the KMS is slow?
• Plan loss before rotation. Key backups, deletion safeguards and recovery drills matter more than the rotation date.

Common mistakes

• Believing disk encryption protects against the application.
• Terminating TLS at the load balancer and sending plaintext behind it.
• Storing the plaintext data key, or keys beside the data they protect.
• Assuming rotation re-encrypts data or limits a compromised data key.
• Allowing 0-RTT for requests that must not run twice.
• Putting secrets in the encryption context. It is logged in plaintext.
• Deleting or disabling a key without checking what depends on it.

Key takeaways

• Encryption moves trust to key custody: name who reads the bytes and who holds the keys.
• TLS 1.3 gives authentication, integrity and confidentiality per connection, with forward secrecy; TLS 1.0 and 1.1 are prohibited.
• 0-RTT data can be replayed: send only repeatable requests as early data.
• TLS protects hops; list every hop and what protects it.
• Disk encryption stops stolen media; field and client-side encryption stop readers of the stored data.
• Envelope encryption: a local DEK per object, a KEK that never leaves the KMS, the wrapped DEK stored with the data.
• Rotation changes new key material only: it does not re-encrypt data or rescue an exposed data key.
• Losing or disabling a KEK makes data unreadable; protect keys against deletion.

Practice exercises

Exercise 1 (30 min). Run the minimal example, then encrypt a second record under order=1002 and try to decrypt it with the context order=1001. Explain which of the two decrypt steps fails first and why.

Exercise 2 (45 min). List every network hop of ShopStream's checkout, from phone to ledger, with what protects each and what an attacker on it sees.

Exercise 3 (1 hour). A data key was logged by mistake. Using Figure 4, write the steps and say why rotating the KEK alone would not fix it.

Exercise 4 (1.5 hours). Extend the sketch so a record's DEK is re-wrapped under a new KEK without touching the ciphertext, and explain what that saves compared with re-encrypting the data.

Related reading

• Security for Distributed Systems: the layered-defence frame this lesson deepens.
• ShopStream Stage 10: Secure and Operate: envelope encryption in the project.
• Service Mesh: automating mutual TLS and certificate rotation.
• API Gateways and Service Meshes: the edge where TLS usually terminates.
• Idempotent and Commutative Operations: why replayable early data must be safe to repeat.
• Secrets Management: the next lesson, on credentials rather than data keys.

Official documentation

• IETF: RFC 8446: TLS 1.3 and its update RFC 9846; RFC 8996 (TLS 1.0 and 1.1) and RFC 6797 (HSTS).
• AWS KMS developer guide: key types and hierarchy, data keys, key rotation and encryption context.
• Google Cloud, Envelope encryption.
• Microsoft Azure, Data encryption at rest and Secure your Key Vault.
• PostgreSQL 18, Encryption Options.
• OWASP, Cryptographic Storage Cheat Sheet.