Skip to content

    System Design

    TLS and HTTPS: Certificates, Handshake, Termination and mTLS

    Learn what TLS and HTTPS prove and protect, how certificates, SNI and ALPN fit the handshake, where to terminate TLS, how mTLS works and why certs expire.

    By Published 12 min read

    Share

    Standard lessonintermediateBackend System Design

    Before this: TCP and UDP

    What you'll learn: By the end of this lesson you will be able to say what TLS gives a connection (confidentiality, integrity and authentication of the server) and what HTTPS is, read the handshake as the client and server agree a version, keys and an identity, explain how a certificate ties a name to a key and what a client checks, say what SNI, ALPN and session resumption are for, decide where to terminate TLS and when to use mutual TLS, and plan certificate renewal so that an expiry never becomes an outage.

    A courier arrives with a sealed box. Two things must be true before you trust it: the box cannot be opened or altered on the way (the seal), and the person who sent it is who they say they are (the passport check). A seal without the passport check lets an impostor send you a perfectly sealed box. A passport without a seal lets anyone read the contents on the road. TLS is the seal and the passport check, done together, on a connection you already opened with TCP.

    The existing lesson Encryption at Rest and in Transit puts TLS beside the other encryption layers and lists what changed in TLS 1.3. This lesson is the dedicated treatment of the connection itself: identity, certificates and operations. The browser side of the story is in HTTP/1.1, HTTP/2 and HTTP/3, which depends on the ALPN step below.

    Verified against RFC 8446, 8996, 6066, 7301, 6797, 9525 and 5280, the MDN page on TLS, the nginx ngx_http_ssl_module documentation and the Let's Encrypt FAQ, all read on 2026-11-16. Defaults change between releases, so check the page for the version you run.

    The core idea

    TLS is a protocol that lets a client and server agree on keys over an untrusted network and prove the server's identity, so that what follows is private and tamper-evident; HTTPS is HTTP carried over it. RFC 8446 lists the goals as authentication ("the server side of the channel is always authenticated; the client side is optionally authenticated"), confidentiality ("data sent over the channel after establishment is only visible to the endpoints") and integrity ("cannot be modified by attackers without detection"). MDN says the same of HTTPS: it is the defence against a manipulator-in-the-middle who "can read and modify the traffic exchanged".

    The key idea behind the rest is that encryption without identity is not enough: the client must know the keys belong to the name it asked for. That is the job of the certificate. Three illustrations run through the lesson, each labelled:

    • Minimal example: a handshake read line by line, then a certificate check.
    • Realistic example: ShopStream's edge terminating TLS and re-encrypting to internal services, in "In practice".
    • Failure example: a certificate that expires because its renewal quietly stopped.

    The handshake

    Before TLS starts, TCP has already spent a round trip on its own handshake. TLS then agrees the parameters. MDN lists what it settles: the version ("Client and server agree on which version of TLS to use"), the cipher suite (the algorithms for key agreement, authentication, encryption and message authentication), authentication of the server (optionally the client as well) and a session key. In TLS 1.3 this takes one round trip before application data, and all handshake messages after ServerHello are encrypted (RFC 8446 section 1.2).

    sequenceDiagram accTitle: TCP then a TLS 1.3 handshake accDescr: After the TCP handshake, the client sends a ClientHello with a key share, the server name and supported application protocols. The server replies with a ServerHello and then, encrypted, its extensions, certificate, a signature and Finished. The client sends Finished and then application data. participant C as Client participant S as Server C->>S: TCP handshake (one round trip) C->>S: ClientHello: key share, server name (SNI), protocols (ALPN) S-->>C: ServerHello: chosen key share Note over C,S: everything below is encrypted S-->>C: EncryptedExtensions, Certificate, CertificateVerify, Finished C->>S: Finished, then HTTP request

    Figure 1. Where TLS sits after TCP. How to read this: the client's first TLS message offers a key share, the name it wants (SNI) and the protocols it can speak (ALPN). The server proves itself in the encrypted part with a certificate and a signature over the handshake (CertificateVerify). Application data follows once both Finished messages are checked.

    Two ClientHello fields cause most operational surprises:

    • SNI (server name indication). RFC 6066 says "TLS does not provide a mechanism for a client to tell a server the name of the server it is contacting", so the server_name extension carries the DNS hostname, which lets one address serve many names ("virtual" servers) with the right certificate for each. Without it, a server with many certificates has to guess.
    • ALPN. RFC 7301 lets "the application layer ... negotiate which protocol will be used within the TLS connection" without extra round trips; it was requested "to address the negotiation of HTTP/2 over TLS". It is how a client and server settle on h2 or http/1.1 on port 443.

    Certificates: tying a name to a key

    A key pair on its own proves nothing about who holds it. A certificate fixes that: MDN describes it as "a digital certificate, which contains a digitally signed copy of the public key which is the counterpart of the website's private key", binding "the website's keys to their domain name". A certification authority (CA) signs it; MDN names Let's Encrypt as "a widely used nonprofit Certification Authority which issues free TLS certificates".

    What a client checks, in the order that matters when something breaks:

    1. The name matches. RFC 9525 says to check DNS names only in the subjectAltName extension (dNSName), not to check "strings that look like domain names in the subject's Common Name", and to allow a wildcard only as the complete left-most label. So *.shop.example covers api.shop.example, not a.b.shop.example and not shop.example itself.
    2. The certificate is inside its validity period. RFC 5280 defines it as "the period of time from notBefore through notAfter, inclusive".
    3. A CA the client trusts issued it, and the signature checks out.
    4. The server holds the private key. In TLS 1.3 the receiver of a CertificateVerify message "MUST verify the signature field", a signature made with the key behind the certificate.
    flowchart TD accTitle: What a client checks about a server certificate accDescr: The client checks that the certificate name matches the requested host, that the current time is inside the validity period, that a trusted authority issued it, and that the handshake signature verifies with the certificate key. Any failure ends the connection with an error. S["Server sends certificate and signature"] --> N{"Name in subjectAltName matches the host?"} N -->|"no"| F["Fail: name mismatch"] N -->|"yes"| V{"Now between notBefore and notAfter?"} V -->|"no"| F2["Fail: expired or not yet valid"] V -->|"yes"| T{"Issued by a CA the client trusts?"} T -->|"no"| F3["Fail: untrusted issuer"] T -->|"yes"| G{"Handshake signature verifies?"} G -->|"no"| F4["Fail: key does not match"] G -->|"yes"| OK["Connection proceeds"]

    Figure 2. The checks that make a handshake succeed. How to read this: each branch is a distinct production incident. Name mismatches follow new hostnames, expiry follows a stopped renewal, an untrusted issuer follows an internal CA nobody distributed, and the last one follows a mixed-up key.

    Lifetime and renewal

    A certificate is valid for a bounded period, and the bound is short on purpose. Let's Encrypt's FAQ says its "default certificates are valid for 90 days" and recommends "renewing 90 day certificates every 60 days", which leaves a month of slack for a failed renewal. Short lifetimes force automation: a human-run renewal is a calendar entry waiting to be missed.

    stateDiagram-v2 accTitle: Life of a server certificate accDescr: A certificate is issued, serves traffic, is renewed after about two thirds of its lifetime, and the new one replaces it. If renewal fails the certificate keeps serving until it expires, after which clients refuse the connection. [*] --> Issued Issued --> Serving: deployed to servers Serving --> Renewing: about 60 days in, for a 90 day certificate Renewing --> Issued: new certificate issued Renewing --> Serving: renewal fails, old one still valid Serving --> Expired: notAfter passes Expired --> [*]: clients refuse the connection

    Figure 3. Renewal has slack until it does not. How to read this: a failed renewal is invisible while the old certificate is still valid. The only visible event is expiry, so monitor the days left, not just whether the job ran.

    Resumption, 0-RTT and versions

    • Resumption. RFC 8446 section 2.2 lets the server hand the client a pre-shared key identity after a handshake so a later connection can start from it. In nginx, ssl_session_tickets is on by default, the session timeout is 5 minutes and the shared session cache is none by default; read the page for your version before assuming a cache exists.
    • 0-RTT. TLS 1.3 "allows clients to send data on the first flight", but RFC 8446 section 2.3 says that data "is not forward secret" and "there are no guarantees of non-replay between connections". nginx's ssl_early_data is off by default. Allow early data only for requests safe to repeat (see Idempotent and Commutative Operations).
    • Versions. RFC 8996 says "TLS 1.0 MUST NOT be used" and "TLS 1.1 MUST NOT be used". nginx's ssl_protocols default is TLSv1.2 TLSv1.3. HSTS (RFC 6797) completes the HTTPS story: a Strict-Transport-Security header with max-age makes a browser rewrite http to https and treat errors as fatal, and it is honoured only when received over secure transport, so the very first visit is the exposed one.

    Where TLS terminates

    TLS protects one connection between two endpoints. Wherever a proxy or load balancer ends it, the next hop is whatever you built.

    Placement What protects the internal hop Gain Cost Wrong when
    Terminate at the edge, plain inside Only the network One place to manage certificates, simple inspection Internal traffic readable by anyone on that network The internal network is not trusted or regulated data crosses it
    Terminate at the edge, re-encrypt inside TLS again, often with certificates from an internal CA Encryption end to end of each hop A second certificate set and more CPU Nobody owns the internal CA and its renewals
    Pass TLS through to the service The service's own TLS The edge never sees plaintext The edge cannot route on HTTP content You need layer 7 routing or WAF at the edge
    Mutual TLS between services Both ends prove identity Strong service identity, not only encryption Certificates for every workload and a rotation story You cannot automate issuance and renewal

    Mutual TLS

    In the handshake the server can ask the client for a certificate: RFC 8446 lists "CertificateRequest" for when "certificate-based client authentication is desired". With mutual TLS (mTLS) both ends hold certificates, so a service is identified by what it can prove, not by its IP address. In nginx, ssl_verify_client "enables verification of client certificates" and is off by default. The cost is operational: every workload needs a certificate and a rotation plan, which is the problem Service Mesh and Secrets Management address.

    TLS termination at ShopStream's edge, re-encrypted insideShoppers to Edge proxy (TLS 1.3, SNI); Edge proxy to Cart service (mTLS); Edge proxy to Payment service (mTLS)PUBLIC INTERNETSHOPSTREAM NETWORKShoppersShoppersEdge proxy: public certificateEdge proxypubliccertificateCart service: internal certCart serviceinternal certPayment service: internal certPaymentserviceinternal certTLS 1.3, SNImTLSmTLS

    Figure 4. Two certificate domains. How to read this: shoppers verify the edge's public certificate; the edge and the services verify each other with certificates from an internal CA. Each domain has its own renewal job and its own expiry risk.

    In practice: ShopStream's certificates

    NOTE

    This is a realistic scenario, not a case history. It is a composite on the course's running example, ShopStream. It describes no real company; every number other than the shared ShopStream figures and the documented certificate lifetime is an illustrative assumption.

    The situation

    ShopStream serves shoppers at one public hostname and runs a few internal services. Shoppers must get no browser warnings, payment traffic must be encrypted between hops, and a new hostname, tracking.shop.example, goes live next month. Certificates are renewed by a script one engineer wrote.

    Starting architecture

    Figure 4 is the target: a public certificate at the edge, internal certificates inside. Today the edge has one certificate for shop.example and www.shop.example, and the services talk plain HTTP inside the network.

    Options considered

    Option Benefit Cost Risk
    A. Keep plain HTTP inside No change None Payment data readable on the internal network
    B. TLS inside with certificates from an internal CA, no client verification Encryption on every hop An internal CA and renewal jobs A service can still be impersonated
    C. mTLS inside Encryption and workload identity Certificates for each workload, rotation Expiry in one place breaks a call path

    The decision

    Option B first, then C for the payment service only, with short-lived certificates renewed by automation and an expiry alert at 21 days. The wildcard question for the new name is answered by the rule above: *.shop.example would cover tracking.shop.example, but a wildcard certificate has a wider blast radius if its key leaks, so the team issues a certificate that names each host. Record: context unencrypted internal hop; decision internal CA, TLS inside, mTLS for payment; consequences a renewal pipeline and expiry monitoring; revisit when a mesh can issue workload certificates.

    Rollout and verification

    1. Issue the internal certificates and enable TLS on the payment service in permissive mode, accepting both plain and TLS.
    2. Switch callers one at a time to TLS and watch handshake error counts and latency percentiles.
    3. Enforce TLS, then mTLS, and remove the plain listener.
    4. Verify days to expiry for every certificate on a dashboard fed by a probe that completes a real handshake and reads notAfter, not by a check that the renewal job ran.

    The 2 a.m. failure

    The page reads: checkout fails and the payment service logs certificate errors on every call from the edge. A certificate on the payment service expired. Its renewal job had stopped months earlier after a credential rotation, and since the previous certificate was still valid nothing alerted until notAfter passed (Figure 3).

    The checks: read notBefore and notAfter on the certificate the service actually presents, compare with the clock on both hosts, and check the last successful run of the renewal job. The mitigation is issuing and deploying a new certificate by hand. The lasting fix is the probe from the rollout, an alert well before expiry (21 days here, an illustrative figure), and an owner for the renewal job.

    Architect's notes

    • Check the identity, not just the encryption. An encrypted connection to the wrong server is the failure TLS exists to prevent.
    • Automate issuance and monitor expiry from outside. Probe the live endpoint and alert on days left.
    • Prefer certificates that name each host, and use wildcards deliberately, knowing one leaked key then covers every name.
    • Name the TLS version policy. TLS 1.0 and 1.1 are out; set the minimum explicitly rather than relying on a default.
    • Treat early data as unsafe by default, and allow it only for repeatable requests.
    • Own every hop. For each connection in the request path, write down where TLS ends and what protects the next hop.

    Common mistakes

    • Letting certificates expire. Renewal that "usually works" fails silently until the day it matters.
    • Disabling verification "to make it work". That removes the passport check and keeps only the seal.
    • Matching on the Common Name. Modern clients use subjectAltName.
    • Assuming a wildcard covers the bare name or deeper names. It covers one label.
    • Assuming TLS protects the whole path. It protects one hop; a terminating proxy starts a new one.

    Key takeaways

    • TLS gives confidentiality, integrity and server authentication; HTTPS is HTTP over it.
    • Encryption without identity is not enough: the certificate ties a name to a key.
    • A client checks the name (subjectAltName), the validity period, the issuer and the handshake signature.
    • SNI selects the certificate; ALPN selects the application protocol such as h2.
    • Resumption and 0-RTT save round trips; 0-RTT data can be replayed, so allow it only for safe requests.
    • Terminate where you can manage certificates, then decide what protects the next hop; mTLS gives workloads identity at the price of certificate operations.
    • Certificates expire by design: automate renewal and monitor days left from outside.

    Practice exercises

    Exercise 1 (20 min). For a certificate with *.shop.example and shop.example in its names, say which of api.shop.example, a.b.shop.example and shop.example it is valid for and why.

    Exercise 2 (30 min). A certificate is valid for 90 days and renewed every 60. How many days of slack remain if the renewal fails once, and what would you alert on?

    Exercise 3 (1 hour). Draw the request path of a service you know and mark where TLS terminates at each hop and what protects the traffic after it.

    Exercise 4 (2 hours). Write a rollout plan to add mTLS to one internal call: issuance, permissive mode, enforcement, the metric by outcome and the rollback step.

    Official documentation

    Ask about this article

    Get answers grounded in this post. AI-generated — based on this article, and may be imperfect.

    Scaled AI Weekly

    Enjoyed this? Get more like it every Monday.

    Real architecture decisions, LLMOps patterns that survive production, and engineering leadership advice — from 12+ years of building at enterprise scale. Free. No spam. Unsubscribe anytime.

    Get new articles by email

    Free. We email a confirmation link first, and you can unsubscribe from any email.

    Join engineers building production AI systems

    Comments