Secrets Management: Stores, Rotation and Short-Lived Credentials

What you'll learn: By the end of this lesson you will be able to define what counts as a secret and trace its life from creation to revocation, explain what a secret store adds over environment variables, rotate a credential without downtime using overlapping versions or leases, prefer identity and short-lived credentials over stored passwords, describe what Kubernetes Secrets do and do not protect, and respond to a leaked secret in the right order.

Spare keys under the doormat are easy to reach and impossible to track. A key cabinet that records who signed out which key is better. Best of all is a key that stops working at the end of the shift, so a lost one is a short problem. Secrets management is the move from the doormat to the self-expiring key.

A secret is any credential whose possession is access: a database password, an API key for a payment provider, a signing key, a TLS private key. Security for Distributed Systems lists the principles and ShopStream Stage 10 applies them in four bullets. This lesson works through the mechanics: lifecycle, what stores do, rotation without downtime, the "secret zero" problem, Kubernetes caveats and leak response. Key custody for encrypted data is in encryption at rest and in transit.

The core idea

Treat every secret as a liability with a lifecycle: the safest secret is one that does not exist (identity replaces it), the next safest is one that expires in minutes, and any other must be stored centrally, scoped narrowly, rotated routinely and audited.

Three illustrations run through the lesson:

• Minimal example: a cache that refreshes a secret before its lease ends.
• Realistic example: ShopStream's payment-provider key and database credentials.
• Failure example: a key committed to a repository, and a rotation that breaks callers.

Where secrets leak, and why a store helps

The OWASP Secrets Management Cheat Sheet starts from the pain. Secrets end up hard-coded in source in plaintext; services share the same secrets, so when one leaks it is hard to tell which service leaked it; and manual maintenance invites errors. It recommends centralising how secrets are stored, provisioned, audited and rotated.

Common leak routes, and what the sources say about each:

• Source control and images. Never hard-code secrets with Docker ENV or ARG (OWASP); mount or inject them at runtime instead.
• Environment variables. OWASP advises against them unless other methods are impossible: they are visible to all processes and can appear in logs and system dumps.
• CI/CD. Treat the pipeline as production. Avoid "one big secret", keep pipeline credentials short-lived and narrow, and stop forks and copied job definitions inheriting secrets.
• Logs and error messages. Applications must not log secrets in the clear (Kubernetes documentation).

A secret store replaces the doormat with four capabilities: encrypted central storage, per-secret access control (OWASP: engineers should not have access to all secrets), an audit trail (who requested a secret, when, whether it was approved and when it expired), and versions with rotation hooks.

Figure 1. The lifecycle of one secret.

How to read this: the loop between store, use and rotate is where most programmes stall. Rotation that is manual does not happen, so design for it from the first deployment. Every box writes to the audit trail.

What the main stores give you

The four common stores differ in what they automate. The table summarises the documentation read on 2026-11-08.

Two details are worth keeping. Azure Key Vault secrets have a 25 KB limit and an optional exp attribute that is informational only: it does not stop retrieval. And Azure advises keeping certificates as certificates, not as secrets, so they can be auto-rotated.

Prefer identity, then short-lived credentials

Identity over secrets. If a workload can prove who it is to the cloud or the store, no long-lived password needs to exist. Azure's guidance is to use managed identities for all app and service connections to Key Vault "to eliminate hard-coded credentials". The remaining question is the secret zero problem: to fetch secrets a workload must authenticate to the store, and that first credential cannot itself be in the store. Platform identity (a managed identity or a cluster service account) answers it: the platform vouches for the workload. OWASP adds the guard rail that who may impersonate service accounts must be tightly controlled.

Short-lived credentials. OWASP says dynamic secrets should be used where possible, with short leases and revocation when no longer needed, and notes that restarting an application does not revoke a stolen credential: it stays valid until the backing service expires or revokes it. Vault's database secrets engine shows the model. A role describes the permissions; each request creates a new database user; the lease sets a time to live (TTL) and a maximum TTL, which the page gives as defaults of 1 hour and 24 hours (check the page for your version); and revocation removes the user when the lease ends. Because every instance has its own username, a questionable query can be traced to one instance. The page also describes static roles, a one-to-one mapping to a database user whose password Vault rotates on a schedule (default 24 hours), and warns never to use a static role for the root credential Vault itself uses.

Rotation without downtime

Rotation replaces a secret on a schedule or on suspicion. OWASP lists four triggers: a regular schedule, an expiry set at creation, suspected or confirmed compromise, and incident response. The hard part is changing the secret in two places, the store and the system that checks it, without a moment when callers hold a value the system refuses.

AWS Secrets Manager documents one pattern in four steps, run by a rotation function. It uses staging labels to mark versions: AWSCURRENT, AWSPENDING and AWSPREVIOUS.

Figure 2. A four-step rotation with overlapping versions.

How to read this: the store never serves an untested value as current. Keeping the previous version for a time means callers that cached the old value have something to fall back to.

AWS notes that storing the new value as AWSPENDING first makes a failed rotation retryable, and that the function, "a privileged deputy", should check that the current credential is valid and that the current and pending values name the same resource, to avoid a confused-deputy attack. Azure Key Vault documents tutorials for resources with one set of credentials and with two; with two valid credentials, one can be replaced while the other serves traffic.

Google's model differs. Secret Manager only sends a notification when rotation is due; a failed delivery is retried for up to seven days and then the rotation is cancelled, and scheduled rotations are skipped while an earlier one is in flight. The workflow that changes the secret is yours.

The consumer's side. A rotated secret is useless if callers keep the old one. Applications should read secrets through a small layer that refreshes before expiry and once after an authentication failure.

Illustration 1 (minimal). A cache with an injected clock, refreshing ten seconds before a 60-second lease ends. It was run with Node on 2026-11-08; the fetcher is a stand-in for a real store.

// Teaching sketch: refresh a secret before it expires.
class SecretCache {
  constructor(fetchSecret, now, refreshBeforeMs) {
    Object.assign(this, { fetchSecret, now, refreshBeforeMs, value: null, expiresAt: 0 });
  }
  get() {
    if (this.value === null || this.now() >= this.expiresAt - this.refreshBeforeMs) this.refresh();
    return this.value;
  }
  refresh() {
    const { value, ttlMs } = this.fetchSecret();
    this.value = value;
    this.expiresAt = this.now() + ttlMs;
  }
}

let clock = 0;
let issued = 0;
const store = () => ({ value: `db-password-v${++issued}`, ttlMs: 60_000 });
const cache = new SecretCache(store, () => clock, 10_000);

console.log("t=0s    ", cache.get());
clock = 49_000;
console.log("t=49s   ", cache.get(), "(still inside the lease)");
clock = 50_000;
console.log("t=50s   ", cache.get(), "(refreshed 10 s before expiry)");

Output:

t=0s     db-password-v1
t=49s    db-password-v1 (still inside the lease)
t=50s    db-password-v2 (refreshed 10 s before expiry)

Connection pools complicate this: a pool that opened connections with the old password keeps them until they close, and new connections need the new one. Make the pool read credentials at connect time, not once at startup.

Kubernetes Secrets: what they do and do not protect

The Kubernetes documentation is blunt. Secrets are, by default, stored unencrypted in etcd. Anyone with API access can retrieve or modify a Secret, and so can anyone with access to etcd. Anyone authorised to create a Pod in a namespace can read any Secret there, including indirectly by creating a Deployment. And "Base64 encoding is not an encryption method", so a committed Secret manifest exposes the value to every reader.

Its recommended steps, with the details from the linked pages:

1. Enable encryption at rest for Secrets. An EncryptionConfiguration lists providers; the first provider encrypts new writes and others are tried for reads; if identity is first, "the default identity provider does not provide any confidentiality protection". KMS v2 (envelope encryption: data keys protected by a key held in a remote KMS) has been stable since Kubernetes 1.29, and KMS v1 is deprecated and disabled by default (page for version 1.37, at the time of writing). Objects already stored are not re-encrypted until written again (the page says this for custom resources; plan a rewrite of existing Secrets and verify).
2. Least-privilege RBAC. Granting list on Secrets implicitly lets the subject fetch their contents; restrict watch and list to the most privileged system components.
3. Restrict access to specific containers. Mount a Secret only into the container that needs it.
4. Consider an external store. The Secrets Store CSI Driver lets the kubelet fetch secrets from external stores and mount them as a volume into the Pods you authorise.

Figure 3. Where a cluster workload gets a secret.

How to read this: each step down adds work and risk. A native Secret is the fallback, not the default.

Leaks: the order of operations

OWASP's incident guidance puts rapid containment first: revoke the exposed secret immediately, then rotate it, preferably by an automated process, then remove it from code, logs and other places it spread. It notes that removing a secret from git history has trade-offs, which is why rotation, not cleanup, is the real fix. Detection comes earlier: use secret-scanning tools in pre-commit hooks and CI, run more than one and correlate, because pattern matching alone misses human-chosen passwords.

Figure 4. A leaked secret.

How to read this: revoke before you investigate. An attacker who holds a valid key does not wait for your post-mortem.

Choosing where a secret lives

In practice: ShopStream moves its credentials

This is the realistic example, on ShopStream, the course's running system. It is an educational scenario and every number is illustrative.

This is a realistic scenario, not a case history.

The situation

ShopStream's services read a payment-provider API key and database passwords from environment variables in deployment files. Several services share one payment key, and staging uses the same key as production, so nobody could say which service leaked it. Nobody remembers when the database passwords last changed.

Starting architecture

Figure 5. Credentials delivered from one place.

How to read this: services prove who they are to the store and fetch what they need. The store creates database users on demand and rotates the provider key, and every read lands in the audit log.

Options considered

• A. Environment variables, rotated by hand. Cheapest, but rotation depends on memory and values stay visible to every process and dump.
• B. Native Kubernetes Secrets with encryption at rest. No new service, but Pod creators can read them and rotation stays manual.
• C. A secret store with workload identity, dynamic database users and scheduled rotation. More to operate, but a leaked database credential expires on its own and each read is attributable.

The decision

Option C. Each service gets its own identity and may read only its own secrets, separate stores for staging and production, database credentials minted per instance with a short lease (illustrative: 1 hour, from the Vault page's default), and the payment key rotated on a schedule using two overlapping keys if the provider supports two active keys; if it does not, the rotation window and a fallback are written into the runbook. CI gets a scanner and no production secrets.

Rollout and verification

Move staging first. Rehearse rotation until it causes zero failed requests, including a forced failure at step 2 to prove retry works. Verify that no service can read another's secret, that no deployment file contains a secret value, and that the audit log shows every read. Monitor secret-read errors, lease-renewal failures and secrets past their intended age.

The 2 a.m. failure

Illustration 3 (failure). A scanner alert says a staging payment-provider key was committed to a repository. Following Figure 4: revoke the key at the provider first, then read the provider's and the store's audit logs for use since the commit, issue a replacement through the store, and remove the value from the repository. The team finds staging and production had shared one key, so production traffic fails too: exactly the sharing OWASP warns about. The fix is one secret per purpose and per environment, so a staging leak cannot touch production.

Architect's notes

• Count your secrets. An inventory with owner, purpose, lifetime and rotation method is the first deliverable.
• Aim to delete secrets. Each one replaced by identity is one fewer to leak, rotate or audit.
• One secret per purpose and per environment. Sharing makes a leak undiagnosable and widens its reach.
• Rehearse rotation and revocation before you need them, including the failure path.
• Put the store on the critical path deliberately. Decide what a service does when it cannot reach the store: use the cached value for how long, then fail how?
• Scope the store itself. Separate stores or paths per application and environment, and alert on unusual reads.

Common mistakes

• Secrets in environment files, images or repositories.
• One shared secret across services or environments.
• Rotation by calendar reminder.
• Rotating the store and forgetting the consumer's cache or pool.
• Treating base64 as protection.
• Wide list access on Kubernetes Secrets.
• Investigating a leak before revoking.
• Logging secrets, or using a secret store as a configuration database.

Key takeaways

• A secret is a credential whose possession is access; manage its lifecycle from creation to revocation.
• Replace secrets with platform identity where possible; shorten lifetimes where not.
• A secret store gives central encrypted storage, per-secret access control, audit and versioned rotation hooks.
• Rotate with overlapping versions or leases so callers always hold a valid value; refresh in the consumer.
• Kubernetes Secrets are unencrypted by default and readable by anyone who can create a Pod in the namespace: enable encryption at rest, narrow RBAC, or use an external store.
• On a leak: revoke, check use, replace, clean up, fix the cause, in that order.
• One secret per purpose and per environment.

Practice exercises

Exercise 1 (30 min). Run the sketch. Add a second refresh path that refreshes once when a call fails with an authentication error, and a test with an injected failing fetcher.

Exercise 2 (45 min). Inventory ShopStream's secrets (database users, payment key, signing key, TLS keys, CI tokens). For each, name the owner, the lifetime you would set, and whether identity could replace it.

Exercise 3 (1 hour). Write the four rotation steps of Figure 2 for a database password, and describe what an application experiences if step 2 succeeds but step 3 fails.

Exercise 4 (45 min). A cluster lists identity first in its EncryptionConfiguration. State what an attacker with etcd access can read and the three changes you would make.

Related reading

• Security for Distributed Systems: the principles this lesson implements.
• ShopStream Stage 10: Secure and Operate: secrets in the project.
• Encryption at Rest and in Transit: envelope encryption and key custody.
• Authentication vs Authorization, Sessions vs Tokens: the credentials that secrets are used to issue and verify.
• Service Mesh: automatic certificate issuance and rotation between services.

Official documentation

• OWASP, Secrets Management Cheat Sheet.
• HashiCorp Vault, Lease, renew and revoke and the database secrets engine.
• AWS Secrets Manager, Lambda rotation functions.
• Google Cloud, Secret Manager rotation.
• Microsoft Azure, About Key Vault secrets and Secure your Key Vault.
• Kubernetes, Secrets, good practices, encrypting data at rest and the KMS provider.