System Design: Security for Distributed Systems

OpenID Connect (OIDC) is a thin layer on top of OAuth 2.0 for authentication. It adds an ID token that tells the application who the user is.

A simplified authorisation-code flow, the standard choice for web and mobile apps:

Key points to remember:

• The one-time code travels through the browser, but the tokens are fetched directly between the app (its backend, where it has one) and the identity provider. That keeps tokens out of URLs and browser history.
• Public clients such as single-page and mobile apps must use the PKCE extension (current guidance recommends it for every client), which protects the code from being stolen and replayed.
• The state parameter guards against cross-site request forgery during the redirect dance.
• The client credentials flow is for service-to-service calls with no user involved. A service authenticates as itself and receives a token for its own scopes.

Tokens and JWT pitfalls

A JSON Web Token (JWT) is a compact, signed bundle of claims such as user id, issuer, audience and expiry. Services can verify the signature locally, with no database lookup, which makes JWTs popular for access tokens in distributed systems.

That convenience has sharp edges:

There is a real choice between self-contained tokens (JWTs, fast to verify, hard to revoke) and opaque tokens (random strings looked up on the server, instantly revocable, but each check costs a call or cache hit). The flowchart below is how a design review usually settles it.

Read it top down: a hard revocation requirement pushes you to opaque tokens, many independent verifiers favour short-lived JWTs, and a single web app with one backend is simplest with a plain session. Many systems combine them: short-lived JWTs at the edge, revocable refresh tokens behind it.

Encryption: in transit, at rest, and between services

In transit. TLS encrypts traffic between a client and a server and lets the client verify the server's identity through a certificate. Use it for every external connection, and increasingly for internal ones too, because "the private network is safe" is an assumption attackers love.

Mutual TLS (mTLS) goes further: both sides present certificates, so the server also verifies the client. Inside a cluster this gives each service a cryptographic identity and encrypts service-to-service traffic. Service meshes often automate certificate issuance and rotation, which is the hard part. See API Gateways and Service Meshes for how that works in practice.

At rest. Encryption of disks, volumes and buckets protects against stolen hardware and leaked backups. It does not stop an application bug that reads data it should not, because the application sees plaintext. Treat it as one layer, not a cure.

For especially sensitive fields (payment details, national identifiers) consider field-level or envelope encryption: data is encrypted with a data key, and the data key is itself encrypted by a master key held in a key management service (KMS). Access to the master key can then be tightly controlled and logged.

Secrets management

A secret is any credential: database passwords, API keys, signing keys, certificates. Secrets leak most often through mundane routes: committed to a repository, baked into a container image, printed in logs, pasted in chat.

Principles that prevent most incidents:

• Keep secrets out of source control and images. Inject them at runtime from a dedicated secret manager or the platform's secret store.
• Prefer short-lived, dynamically issued credentials (for example, per-service database users that expire) over a shared password that never changes.
• Rotate on a schedule and on suspicion. If rotation is painful you will not do it, so automate and test it.
• One secret per purpose and per environment. A leaked staging key must not open production.
• Never log secrets, and scan for leaks. Scrub authorisation headers from logs, and let secret scanning in CI catch the accidental commit.

Least privilege, validation, and abuse

Least privilege applies at every layer: a service's database user can only touch its own tables; a worker's cloud role can read one queue and write one bucket; an engineer's production access is temporary and audited. When a credential leaks, the damage equals that credential's permissions, so keep them small.

Input validation treats everything from outside a trust boundary as hostile until checked:

• Validate type, length, range and format on the server. Client-side checks are only a courtesy.
• Use parameterised queries instead of building SQL strings (dynamic identifiers such as column names still need an allow-list).
• Encode output for its destination (HTML, URL, shell) to prevent injection and cross-site scripting.
• Limit request sizes and upload types, and scan or sandbox uploaded files.
• Be careful with server-side requests built from user-supplied URLs; attackers use them to reach internal services (server-side request forgery).

Abuse and rate limiting. Credential stuffing, scraping and brute force simply send many requests. Rate limits per user, IP and API key, plus progressive delays on login endpoints, blunt them. See Rate Limiting.

Audit logs and defence in depth

An audit log is an append-only record of security-relevant events: logins, permission changes, data exports, administrative actions. Good entries record who did what to which resource, when, and from where. Protect the log itself: ship it off the machine that generated it, restrict who can alter it, and set retention to match your legal and operational needs. They answer "what did the attacker touch?" after an incident.

Now combine everything into layers:

An attacker past the firewall still needs a valid token, which only grants its scopes, which still hit per-object checks, behind which a database account can touch one schema. Every step leaves a trace.

In practice: hardening a B2B procurement SaaS before its first enterprise audit

The situation

A procurement and expense platform serves about 400 business customers, each with 20 to 2,000 users who upload invoices and approve spend. Fourteen engineers run it on Kubernetes with an Envoy-based ingress, nine Java and Node services, PostgreSQL, Redis and S3-compatible storage. Login is delegated to an OIDC identity provider. A large prospect has sent a security questionnaire, and an audit is due in six months. Findings from a recent penetration test and an internal review (all illustrative):

• GET /invoices/{id} checks that the caller is logged in, but not that the invoice belongs to the caller's tenant.
• Access tokens live 24 hours. There is no revocation path.
• All nine services share one PostgreSQL password from a Kubernetes secret that has not changed in two years. Rotating it takes a coordinated half-day.
• Bulk exports leave no record of who ran them.

Starting architecture

Trust ends at the ingress, and everything behind it assumes the caller is legitimate.

Options considered

The decision

Rollout and verification

In shadow mode the library logs a would-deny event and still allows the request. The team reviews every would-deny for false positives before flipping a flag per endpoint. Token lifetime falls first for internal users, then a 5 percent canary of customers. Database roles are migrated with both the old and new credentials valid, so a rollback is a config change. Verification: shadow would-deny count trending to zero for legitimate traffic, an automated cross-tenant test in CI that must return 404 for every endpoint, refresh error rate under 0.1 percent, secret age alerts, and exports appearing in the audit stream within a minute.

The 2 a.m. failure

The page reads: "API 401 rate above 20 percent for 5 minutes". The identity provider rotated its signing key overnight, and services cached the old key set for hours.

Detect it with the auth_failures_total{reason="unknown_kid"} counter. First three checks: read one service log for the key id the token carries, fetch the provider's key-set endpoint and confirm that id is present, and compare the age of the service's cached key set. Mitigation: trigger a key-set refresh through the library's refresh hook if it has one, or a rolling restart of the affected services (never all pods at once). Follow-up fix: refetch on an unknown key id with a rate limit, and agree with the identity provider to publish a new key before it signs with it.

Architect's notes

• Measure before deciding. Count endpoints that take a resource id and how many have a tenant check. That number, not a guess, sizes the work.
• Ask in the review: "If this one credential leaks, what exactly does it open, and for how long?" Make every owner answer for their own service.
• The late cost is rotation. Short tokens and rotating secrets create operational events. Budget alerting and a runbook, or people will quietly lengthen lifetimes again.
• Own it. A shared authorisation library needs a named owner and a version policy, otherwise nine services drift across five versions.
• Defer on purpose. mTLS and per-tenant keys are good ideas with a real running cost; write down the trigger instead of arguing about them every quarter.
• Evidence is a feature. Auditors ask for logs of access and change. Design the audit stream so the answer is a query, not a project.

Common mistakes

• Treating the internal network as trusted. Once one machine is compromised, flat networks let the attacker roam. Use mTLS, network segmentation and per-service authorisation.
• Authenticating but not authorising. Confirming the caller is logged in without checking they own the resource produces object-level access bugs. Check ownership on every data access.
• Trusting the client. Hidden form fields, client-side validation and "admin" flags sent from the browser can all be edited. Validate and decide on the server.
• Long-lived, over-scoped tokens and keys. A leaked credential then does maximum damage for maximum time. Use short lifetimes, narrow scopes, and rotation.
• Secrets in code, images or logs. They spread to every clone, cache and backup. Use a secret manager and scan for leaks.
• Rolling your own cryptography or session scheme. Subtle errors are invisible until exploited. Use standard protocols and maintained libraries.
• Logging too little or too much. Without audit logs you cannot investigate; with secrets and personal data in logs you create a new leak. Log events, not credentials.

Key takeaways

• Assume failure. Design so that one broken control is contained by the next one.
• Threat model early. Mark trust boundaries on your diagram, walk the STRIDE categories, and fix the highest-risk items first.
• AuthN is not AuthZ. Verify identity at the edge, but check permissions and ownership in every service that owns data.
• OAuth 2.0 delegates access; OIDC proves identity. Use the authorisation-code flow with PKCE for user-facing apps and follow current guidance.
• Treat JWTs carefully. Validate signature, expiry, issuer and audience; keep them short-lived; remember the payload is readable.
• Encrypt in transit and at rest, and know the limits. TLS and mTLS protect the wire; disk encryption does not stop application bugs.
• Manage secrets as a lifecycle. Inject at runtime, scope narrowly, rotate, and never log or commit them.
• Layer defences and keep evidence. Rate limits, least privilege, validation and audit logs together turn incidents into survivable events.

Practice exercises

Exercise 1 — Starter (30 min): Take the order-service diagram in this lesson. Copy it, add a mobile app and an admin dashboard, and mark every trust boundary. For each boundary crossing, write one threat from the STRIDE table and one defence.

Exercise 2 — Intermediate (2 hours): Design the authentication and authorisation for a photo-sharing API where users have private and shared albums. Specify the login flow, which scopes the access token carries, token lifetimes, where refresh tokens live, and exactly which service checks that a caller may view a given photo. Write the check as pseudocode and list three requests that must be rejected.

Exercise 3 — Advanced (half day): Produce a one-page security design for a system with a gateway, five microservices, a message queue and a database of personal data. Cover mTLS and certificate rotation, secrets injection and rotation, per-service database accounts, encryption choices with the key hierarchy, an audit-log schema, and the five alerts you would configure first. Finish with a table of what each layer would catch if the previous layer failed.

Related reading

• API Gateway and Service Mesh - where authentication at the edge and mTLS between services are implemented.
• Rate Limiting - the main defence against brute force, scraping and noisy clients.
• Observability and SLOs - how to see and alert on the signals that security controls produce.
• Reliability Basics - the same assume-failure mindset applied to availability.