Skip to content

    System Design

    DNS: Resolution, Caching, TTLs and Why Changes Take Time

    Learn how a name becomes an address: resolvers, root and authoritative servers, record types, TTL and negative caching, and why DNS changes are never instant.

    By Published 13 min read

    Share

    Standard lessonbeginnerBackend System Design

    Before this: The Request Lifecycle

    What you'll learn: By the end of this lesson you will be able to trace how a hostname becomes an address through a stub resolver, a recursive resolver, the root servers and an authoritative server, say what the common record types are for, explain how the TTL and negative caching decide how long an answer lives and why a DNS change is never instant, describe why DNS answers use UDP but fall back to TCP, read a resolver configuration such as a Kubernetes pod's ndots:5, and plan a cutover that survives the caches you do not control.

    A phone directory is too big for every house to hold, and a single directory office would collapse under the calls. So the system works in layers: you look in the notebook by your phone first, then ask a neighbour who has a copy, and only when nobody knows does anyone ring the office that owns that part of the directory. Every copy is stamped with a date after which it must not be trusted. When a number changes, the office updates its own record at once, but notebooks around town keep the old number until their stamps expire.

    That is DNS. The overview of where it sits in a request is The Request Lifecycle; this lesson is the dedicated treatment of the naming system. It uses the transports from TCP and UDP, and its cache-and-expiry behaviour is the reason Service Discovery and Load Balancing treat DNS as a coarse tool.

    Verified against RFC 1034, 1035, 2181, 2308, 6891 and 9210, the Linux resolv.conf manual page, the Kubernetes documentation on DNS for Services and Pods, the Java networking properties page and the IANA root servers page, all read on 2026-11-16.

    The core idea

    DNS is a distributed, hierarchical database that maps names to records, and almost every answer you see came from a cache, not from the owner. The owner of a zone (an authoritative server) holds the truth; resolvers copy it and keep the copy for the record's TTL, the number of seconds it may be cached. RFC 1034 says "the TTL describes how long a RR can be cached before it should be discarded" and "a zero TTL prohibits caching". So a change is visible to everyone only after every cached copy has expired.

    Three illustrations run through the lesson, each labelled:

    • Minimal example: the steps of one lookup, from a cold cache to an address.
    • Realistic example: moving ShopStream's API to a new load balancer in "In practice".
    • Failure example: an application that keeps using an old address long after the record changed.

    Names, zones and records

    A name such as api.shop.example is read right to left: the root, then example, then shop, then api. Each level can be delegated to different servers. RFC 1034 says a name server "is said to be an AUTHORITY for these parts of the name space", grouped into zones, and RFC 1035 limits a label to 63 octets and a whole name to 255.

    Record Purpose (RFC 1035) Back-end relevance
    A A host address The usual target of an API hostname
    NS An authoritative name server for the zone Delegation: who answers for this zone
    CNAME The canonical name for an alias An alias to another name, for example a CDN hostname
    SOA Marks the start of a zone of authority Carries the timers that decide negative caching
    MX A mail exchange Where mail for the domain goes
    TXT Text strings Domain verification, mail policies

    Two rules cause real incidents. RFC 1034 says "if a CNAME RR is present at a node, no other data should be present", and RFC 2181 section 10.1 says a CNAME "may have no other data" (apart from DNSSEC records), so an alias cannot sit at a name that must also hold other records, such as the SOA and NS records at the top of a zone. And RFC 2181 section 5.2 says "the TTLs of all RRs in an RRSet must be the same": several addresses for one name share one TTL.

    How a lookup works

    Three roles are involved. The stub resolver is the small client in your operating system or runtime. It asks a recursive resolver (your cloud network's or your ISP's). RFC 1034 describes the two styles: "the first server pursues the query for the client at another server" (recursive), or "the server refers the client to another server and lets the client pursue the query" (iterative), adding that "the domain system requires implementation of the iterative approach, but allows the recursive approach as an option". In practice the recursive resolver iterates on the stub's behalf. At the top are the root servers: IANA says the root zone is "configured in the DNS root zone as 13 named authorities", served by a much larger set of machines.

    sequenceDiagram accTitle: A DNS lookup with a cold cache accDescr: A stub resolver asks a recursive resolver for api.shop.example. The recursive resolver asks a root server, which refers it to the servers for example. Those refer it to the authoritative server for shop.example, which answers with the address and a TTL. The recursive resolver caches the answer and returns it. participant A as App (stub resolver) participant R as Recursive resolver participant Root as Root server participant T as Servers for example participant Au as Authoritative for shop.example A->>R: api.shop.example? R->>Root: api.shop.example? Root-->>R: referral: ask the servers for example R->>T: api.shop.example? T-->>R: referral: ask the authoritative servers for shop.example R->>Au: api.shop.example? Au-->>R: address, TTL Note over R: cache the answer for the TTL R-->>A: address

    Figure 1. A cold lookup is a walk down the hierarchy. How to read this: each referral narrows the question until a server that is authoritative answers. The resolver keeps what it learned, so later lookups skip steps: a warm cache answers the app in one hop.

    Who holds what: the truth is at the edge, copies are everywhereYour service to OS or sidecar (lookup); OS or sidecar to Recursive resolver; Recursive resolver to Authoritative server (on a miss)Your service: runtime resolver cacheYour serviceruntime resolvercacheOS or sidecar: stub resolverOS or sidecarstub resolverRecursive resolver: shared cacheRecursiveresolvershared cacheAuthoritative server: the zone's truthAuthoritativeserverthe zone's truthlookupon a miss

    Figure 2. Copies at every layer. How to read this: each box between your service and the authoritative server can hold its own cached copy with its own expiry clock. Lowering the TTL at the authoritative server cannot reach into copies that were stored before the change.

    Caching, TTLs and negative answers

    A resolver stores each record for its TTL. Several points follow from the RFCs:

    • The TTL is yours to choose and everyone else's to honour. It is a 32-bit field counted in seconds; RFC 2181 section 8 sets the maximum at 2147483647.
    • A failure is cached too. RFC 2308 defines negative caching as "the storage of knowledge that something does not exist", for both NXDOMAIN (the name does not exist) and NODATA (the name exists but not that record type). The negative TTL "is taken from the minimum of the SOA.MINIMUM field and SOA's TTL". A name you create five minutes after someone looked it up may stay "missing" for the negative TTL.
    • Every layer caches. Your runtime may cache as well. The Java networking properties page says networkaddress.cache.ttl has a default that is "-1 (forever) if a security manager is installed, and implementation-specific when no security manager is installed", and networkaddress.cache.negative.ttl defaults to 10 seconds. Those are security properties, set in the JDK's security configuration, not with -D.
    flowchart TD accTitle: What happens to a DNS lookup at one cache accDescr: A resolver checks its cache. A fresh positive answer is returned. A fresh negative answer returns the cached failure. Otherwise the resolver asks the next server, then stores the answer or the failure for its TTL. Q["Lookup arrives"] --> C{"In cache and not expired?"} C -->|"yes, address"| P["Return the cached address"] C -->|"yes, NXDOMAIN or NODATA"| N["Return the cached failure"] C -->|"no"| U["Ask the next server"] U --> A{"Answer?"} A -->|"address"| SP["Store for the record TTL, return it"] A -->|"NXDOMAIN or NODATA"| SN["Store for the negative TTL, return the failure"]

    Figure 3. Positive and negative caching share one path. How to read this: both a found address and a failure are stored. That is good for load on the authoritative servers and surprising when a missing record is created: callers keep hearing "no" until the negative entry expires.

    TTL is a trade-off

    TTL Gain Cost Wrong when
    Long (hours) Fewer queries, resilience to a short outage of the authoritative servers A change takes hours to spread You may need to move the target quickly
    Short (tens of seconds) Fast moves and failover More queries, more dependence on the authoritative servers being up The authoritative service is a single point of failure, or the clients ignore your TTL
    Lowered before a change, raised after Fast move with normal load Needs planning one old-TTL ahead Nobody owns the schedule

    UDP, TCP and size

    RFC 1035 restricts messages carried by UDP to 512 bytes; longer ones are truncated with the TC bit set. RFC 6891 (EDNS(0)) lets a requester advertise a larger UDP payload size, and suggests 4096 octets "as a starting point". RFC 9210 settles the other half: "All DNS resolvers and servers MUST support and service both UDP and TCP queries", so a client must be able to "retry truncated UDP responses", and it says filtering DNS over TCP is harmful. A firewall that drops TCP port 53 produces the symptom that small answers work and large ones (long records, DNSSEC) fail.

    sequenceDiagram accTitle: A truncated UDP answer retried over TCP accDescr: A resolver sends a query over UDP and receives a truncated response with the TC bit set. It repeats the query over TCP and receives the full answer. participant R as Resolver participant S as Server R->>S: query over UDP S-->>R: truncated answer, TC bit set R->>S: same query over TCP S-->>R: full answer

    Figure 4. Fall back to TCP when UDP is too small. How to read this: the first answer is incomplete on purpose. If the TCP leg is blocked, the lookup fails even though UDP worked.

    What your resolver configuration does

    On Linux, /etc/resolv.conf tells the stub resolver where to ask and how to expand short names. The manual page says the search list "by default contains one entry, the local domain name", timeout defaults to 5 seconds and attempts to 2, and ndots "default for n is 1, meaning that if there are any dots in a name, the name will be tried first as an absolute name". Kubernetes writes a pod's file with search <namespace>.svc.cluster.local svc.cluster.local cluster.local and options ndots:5. That makes short names work: the documentation says a query for data "may be expanded to data.test.svc.cluster.local", and a Service's name has the form my-svc.my-namespace.svc.cluster-domain.example.

    The cost is extra queries. With ndots:5, an external name with fewer than five dots, such as api.partner.example, is tried with the search suffixes first and only then as an absolute name, so a few lookups fail before the one that works. Using a fully qualified name ending in a dot, or lowering ndots for pods that mostly call outside the cluster, removes them. Service Discovery covers the in-cluster side.

    Three illustrations

    A minimal example (one lookup). An app asks for api.shop.example with every cache cold. The stub asks the recursive resolver; the resolver asks a root server, is referred to the servers for example, is referred to the authoritative servers for shop.example, and gets an address and a TTL (Figure 1). The next caller within the TTL gets the cached answer without any of those steps. The values in this walk are placeholders, not captured output.

    A realistic example (ShopStream's API cutover). ShopStream's read path puts a load balancer in front of the app servers; the next section moves that hostname to a new load balancer.

    A failure or edge case (the stale address). A service resolves a partner's hostname once and keeps the address in a long-lived connection pool and in a runtime cache. The partner changes the record; every resolver in the world serves the new address within the TTL, but this service still dials the old one. Nothing in DNS is wrong. The fault is a cache nobody was watching.

    In practice: moving ShopStream's API to a new load balancer

    NOTE

    This is a realistic scenario, not a case history. It is a composite on the course's running example, ShopStream. It describes no real company; every number other than the shared ShopStream figures is an illustrative assumption.

    The situation

    ShopStream's api hostname points at a load balancer the team wants to replace. At the shared browsing peak of about 2,300 requests per second the move must not drop traffic. Clients are browsers, mobile apps and a few partner services.

    Starting architecture

    ShopStream API hostname: old and new load balancerClients to DNS zone (resolve api); DNS zone to Old load balancer (today); DNS zone to New load balancer (after the change); Old load balancer to App servers; New load balancer to App serversClients: browsers, apps, partnersClientsbrowsers, apps,partnersDNS zone: TTL 300 s, illustrativeDNS zoneTTL 300 s,illustrativeOld load balancerOld loadbalancerNew load balancerNew loadbalancerApp serversApp serversresolve apitodayafter the change

    Figure 5. One record, two targets over time. How to read this: clients only ever resolve the name. The change is an edit to one record, and the time to take effect is set by the TTL cached before the edit.

    Options considered

    Option Benefit Cost Risk
    A. Edit the record and wait Trivial Callers use the old target for up to the old TTL, plus any layer that caches longer Old load balancer must stay up an unknown time
    B. Lower the TTL first, wait one old TTL, then edit Predictable switch time A planned lead time and more queries meanwhile Clients that ignore the TTL still lag
    C. Keep both balancers serving and add the new one as a second address No cliff Traffic splits unevenly while caches differ Both must handle full traffic

    The decision

    Option B with the old balancer left serving until its traffic fades. The record: context replace a balancer; decision lower the TTL, wait out the old TTL, edit, watch both balancers' traffic, decommission only when old-balancer traffic is near zero; consequences a few extra DNS queries; revisit when a client class is found to cache beyond the TTL.

    Rollout and verification

    1. Lower the record's TTL, then wait at least the old TTL so every cache holds the short value.
    2. Edit the record to the new balancer.
    3. Verify by comparing requests per second at both balancers over time, and by asking an authoritative server directly and a public resolver separately: the first shows the truth, the second what callers see.
    4. Raise the TTL again only when the old balancer is idle.

    The 2 a.m. failure

    The page reads: errors from one partner integration and a few app instances, two hours after the cutover and long after the TTL. The partner's client resolves once at start-up, and ShopStream's own JVMs were configured long ago with networkaddress.cache.ttl set to -1, which the Java page documents as "cache forever", to cut lookups; both keep dialling the old balancer, which was decommissioned an hour ago.

    The checks: compare the address a failing process connects to with what a fresh lookup returns; read the JVM's networkaddress.cache.ttl security property. The mitigation is restarting the affected instances and bringing the old balancer back for the stragglers. The lasting fix is setting a finite networkaddress.cache.ttl, bounding connection lifetimes in pools, and putting "who caches beyond the TTL" on the cutover checklist.

    Architect's notes

    • Plan changes one old TTL ahead. The speed of a change is set by what is already cached.
    • Treat DNS as a coarse, slow control. Failover through DNS is bounded by caches you do not own; use load balancers and health checks for fast decisions.
    • Check each layer's cache. Runtime, connection pool, sidecar and resolver all hold answers.
    • Keep TCP 53 open. Large answers fall back to it.
    • Mind negative caching. Create a record before anything asks for it.
    • Fully qualify external names in hot paths when the search list adds failed lookups.

    Common mistakes

    • Assuming a record change is instant. Everyone sees it after their cached copy expires.
    • Lowering the TTL at the moment of the change. It only helps the next lookup; callers holding the old answer wait out the old TTL.
    • Aliasing a bare zone name with a CNAME. A CNAME cannot share a name with other data.
    • Blocking DNS over TCP. It breaks large answers while small ones keep working.
    • Forgetting the runtime cache. A JVM or client library may cache longer than DNS says.

    Key takeaways

    • DNS maps names to records through a hierarchy of delegations; authoritative servers hold the truth and resolvers hold copies.
    • The TTL is how long a record may be cached; changes spread only as old copies expire.
    • Failures are cached as well: NXDOMAIN and NODATA live for the SOA-derived negative TTL.
    • UDP carries most answers but TCP is required for large ones.
    • search and ndots decide how many lookups a short name costs.
    • Plan a cutover around the longest cache between your callers and your zone.

    Practice exercises

    Exercise 1 (20 min). Draw the lookup of checkout.shop.example from a cold cache, naming each server asked and what it answers.

    Exercise 2 (30 min). A record has TTL 3,600 s (illustrative) and you must move it at 10:00. State when to lower the TTL, to what, and when it is safe to decommission the old target.

    Exercise 3 (45 min). A pod has ndots:5 and calls api.partner.example 200 times per second. Explain why extra lookups appear and give two ways to remove them.

    Exercise 4 (1 hour). For a service you know, list every cache between code and the zone (runtime, pool, sidecar, resolver) and the longest time each can serve a stale answer.

    Official documentation

    Ask about this article

    Get answers grounded in this post. AI-generated — based on this article, and may be imperfect.

    Scaled AI Weekly

    Enjoyed this? Get more like it every Monday.

    Real architecture decisions, LLMOps patterns that survive production, and engineering leadership advice — from 12+ years of building at enterprise scale. Free. No spam. Unsubscribe anytime.

    Get new articles by email

    Free. We email a confirmation link first, and you can unsubscribe from any email.

    Join engineers building production AI systems

    Comments