Service Discovery: Registries, DNS and Kubernetes Services

What you'll learn: By the end of this lesson you will be able to describe service discovery as a map from a service name to the healthy instances behind it, compare self-registration with platform registration, explain how Kubernetes Services, EndpointSlices and cluster DNS work together, compare Consul's catalog and Eureka's lease model with Kubernetes, reason about how stale a discovery answer can be, and choose an approach for a given system.

A company has 200 staff who change desks every day. A printed directory, issued monthly, is cheap but wrong the day after it prints. A receptionist who tracks who sits where and forwards your call is always current, but she is one more person in every conversation, and when she is away nobody can reach anyone. A shared live spreadsheet that each caller checks before dialling is current, but every caller must know how to read it. Discovery systems are these three, with the same costs.

The API gateway and service mesh lesson gives a one-table overview. This lesson is the dedicated treatment: registration and health mechanics, what three real systems document, and how stale an answer can be.

Based on the Kubernetes, Consul and Eureka documentation pages listed below, read on 2026-11-08. Defaults and behaviour can differ in the version you run, so check the page for it.

The core idea

Service discovery maintains a map from a logical service name to the addresses of the instances that can serve it right now. Instances come and go (deployments, crashes, autoscaling), so the map cannot be written by hand. Every design answers three questions:

1. Registration: how does an instance get into the map?
2. Lookup: how does a caller read it?
3. Removal: how does a dead or unready instance get out?

Three examples run through the lesson:

• Minimal example: one Kubernetes Service and the DNS names it creates.
• Realistic example: ShopStream's Orders service calling Inventory, in "In practice".
• Failure example: a caller that keeps using an address after the instance behind it has gone.

Registration: who writes the map

There are two directions. In self-registration the instance announces itself, and a heartbeat keeps the entry alive. In platform (third-party) registration something else watches the instances and writes the map, and application code does nothing.

Self-registration works anywhere but every service carries registration code or an agent; platform registration needs no application code but ties discovery to the platform.

Lookup: three ways to read the map

Figure 1. Three lookup styles over the same instances. How to read this: in 1 the caller turns a name into addresses and chooses one. In 2 it asks a registry for the live list and balances itself (client-side discovery). In 3 it calls one stable address and a proxy chooses (server-side discovery).

Eureka's client is the client-side style: its overview page describes a built-in load balancer that does basic round-robin.

Kubernetes: the platform does it

The Kubernetes documentation defines a Service as a method for exposing an application running as one or more Pods, with a logical set of endpoints and a policy for reaching them. The default type is ClusterIP. This is the minimal example (the same shape as in the services stage):

apiVersion: v1
kind: Service
metadata:
  name: inventory
  namespace: shop
spec:
  selector:
    app: inventory
  ports:
    - port: 80
      targetPort: 8080

Figure 2. The pieces behind one Service. How to read this: the controller turns Pod readiness into EndpointSlices; cluster DNS and kube-proxy both watch them. The caller resolves a name to the Service's cluster IP, and kube-proxy rules on its node redirect the traffic to one ready Pod.

What the documentation says about each piece:

• Records. A Service gets a DNS A or AAAA record named my-svc.my-namespace.svc.cluster-domain.example that resolves to the cluster IP, and a named port also gets an SRV record. Pods in the same namespace can use the short name my-svc.
• Endpoints. "The controller for that Service continuously scans for Pods that match its selector" and updates the Service's EndpointSlices. By default the control plane limits a slice to 100 endpoints (configurable up to 1000). Each endpoint carries conditions: ready, serving (for a Pod, its Ready condition) and terminating (set when the Pod is deleted). Proxies normally ignore terminating endpoints.
• Proxying. Each kube-proxy watches Service and EndpointSlice objects and configures its node to redirect traffic for the cluster IP to one of the endpoints. In iptables mode the rules by default select a backend Pod at random.
• Headless Services. With clusterIP: None, no cluster IP is allocated, kube-proxy does not handle the Service, and DNS returns the Pod addresses directly: "a client to connect to whichever Pod it prefers". This is client-side discovery on the platform's data.
• Environment variables. The kubelet adds {SVCNAME}_SERVICE_HOST and {SVCNAME}_SERVICE_PORT for each active Service, but only if the Service existed before the Pod; DNS has no such ordering problem.

Kubernetes could have published several A records and relied on round-robin DNS. Its documentation gives three reasons it does not: DNS implementations that ignore TTLs, applications that look a name up once and cache it forever, and the load that low or zero TTLs put on DNS.

Consul and Eureka: registries you run

Consul. Its documentation describes a service catalog that is the single source of truth, which instances join by registering their address and port. Callers use DNS or the HTTP API. A DNS name has the form <service>.service.consul (an optional tag label comes first), returning A records and, if the client asks, SRV records with the port. Services whose health checks fail are omitted from results, and the nodes in each response are randomised. Checks come in many types (script, HTTP, TCP, UDP, TTL, gRPC and others) with an interval and timeout; a TTL check instead waits for an external process to report in time. A check starts critical by default, and deregister_critical_service_after removes a service that stays critical that long.

Consul's catalog is replicated and can lag. The anti-entropy page says the agent's local state is authoritative and is synchronised with the catalog periodically (every minute for clusters up to 128 nodes), and that reads have three modes: default goes to the leader and may occasionally return a stale result, consistent has the leader confirm leadership with a quorum, and stale lets any server answer, which is faster and works during outages but is more likely stale. That is the consistency and availability choice of the CAP lesson, exposed as a setting.

Eureka. The Netflix Eureka wiki describes registration and renewal as a lease. The client registers (on its first heartbeat, 30 seconds in), renews every 30 seconds, and sends a cancel on shutdown; a server that has not seen a renewal for 90 seconds removes the instance. Clients fetch the registry and cache it locally, refreshing every 30 seconds with deltas, and the page says changes may take up to two minutes to reach all clients. Servers replicate operations to their peers and reconcile on the next heartbeat, so clients can see different views during an outage. If renewals drop below a threshold (the page gives 85 percent of expected renewals within 15 minutes), the server stops expiring instances to protect the registry, which can leave clients holding addresses that no longer exist; the page advises clients to time out quickly and try other servers.

Figure 3. Why a crashed instance stays discoverable for a while. How to read this: the lease expiry and the client refresh are two separate delays. Until both pass, a caller can hold an address that nobody answers.

Discovery is a cache, so plan for stale answers

Every design above has a window in which the map is wrong. With the Eureka numbers, the worst case for a crash is the 90-second expiry plus a 30-second client refresh, which agrees with the page's "up to 2 minutes" (the addition is this lesson's). Kubernetes removes a Pod from endpoints when its readiness fails and marks it terminating when it is deleted, but a client with an open connection or a cached DNS answer still has to notice. Consul's stale-read mode trades freshness for availability.

The defence is the same everywhere, and none of it is discovery's job:

• Timeouts and retries to a different instance, with the retry only for idempotent operations (resilience patterns).
• Health that reflects readiness, not just liveness: an instance that is up but cannot serve yet should not be in the map.
• Graceful shutdown that keeps serving until callers have seen the removal.

Figure 4. A decision, not a ranking. How to read this: start with where the instances run. A platform that already tracks them should be the registry. Add a separate registry only when something the platform does not cover requires it.

Trade-offs

In practice: Orders finds Inventory

This is the realistic example. It is an educational scenario on the course's running example, ShopStream, and nothing in it describes a real company. Figures are illustrative.

This is a realistic scenario, not a case history.

The situation

When ShopStream splits into services (services stage), Orders must call Inventory while Inventory runs three Pods (illustrative) that are replaced on every deployment. The team must pick how Orders finds them.

Starting architecture

Figure 5. One Service, three Pods, one of them not ready. How to read this: the Service sends traffic only to Pods whose readiness passes. Orders uses a single name and never sees Pod addresses.

Options considered

The decision

Option B, because everything runs in one cluster and nothing needs cross-platform discovery. Inventory declares a readiness probe so a Pod joins the Service only when it can serve. Orders calls the name inventory with a per-call timeout and one retry for idempotent reads, never for the stock reservation unless it carries an idempotency key. A registry such as Consul would be reconsidered only if services had to span clusters or non-Kubernetes hosts.

Rollout and verification

Verification checks the Service's endpoints with kubectl get endpointslices -l kubernetes.io/service-name=inventory -n shop (illustrative command, not run here) and watches Orders' error rate during a rolling deployment. Metrics: Inventory call errors by cause (refused, timeout), retries per minute, and ready endpoints.

The 2 a.m. failure

The page reads: Orders reports a burst of connection-refused errors to Inventory during a deployment. Some Orders Pods kept using an address for a Pod that had been deleted. The checks: compare the EndpointSlices with the Pods (is the deleted Pod still listed, and is its endpoint marked terminating?), and check how Orders connects (a long-lived connection pool or an application-level DNS cache pins an address). The mitigation: have Orders retry on a new connection, which resolves again, and limit connection lifetime. The lasting fix is a graceful shutdown for Inventory so a terminating Pod keeps answering for the grace period (the documentation says a Pod's termination grace period defaults to 30 seconds) while its endpoint is removed.

Architect's notes

• Use the platform's registry if you have one. A second registry doubles the places that can disagree.
• Define health as readiness. Liveness restarts a container; readiness decides whether it gets traffic.
• Budget the staleness window in seconds, then set client timeouts and retries inside it.
• Decide what the registry does in a partition: serve stale data or refuse. Eureka's self-preservation and Consul's stale mode are both choices of this kind.

Common mistakes

• Hard-coding addresses or caching a resolved address for the life of the process.
• Using liveness as readiness, so an instance joins the map before it can serve.
• Retrying everything against another instance, including non-idempotent writes.
• Running a registry as a single instance. Its failure would stop every lookup.
• Assuming deregistration is instant. Leases and caches make removal take time.

Key takeaways

• Discovery maps a service name to the healthy instances behind it; every design must answer registration, lookup and removal.
• Eureka and Consul use self-registration with heartbeats or health checks; Kubernetes derives endpoints from Pods and readiness.
• Lookup is by DNS, by a registry client, or by a stable address that a proxy resolves.
• Kubernetes Services give a cluster IP and DNS name; headless Services return Pod addresses.
• Kubernetes avoids round-robin DNS because of ignored TTLs, indefinite client caching and DNS load.
• Discovery data is a cache: expect stale answers, and defend with timeouts, safe retries and graceful shutdown.
• Choose the platform's discovery first; add a registry only for needs it does not meet.

Practice exercises

Exercise 1 (30 min). For a Service cart in namespace shop, write the DNS name, the name a Pod in shop can use, and the SRV query name for a port named http over TCP.

Exercise 2 (45 min). A registry has a 30-second renewal, a 90-second expiry and a 30-second client refresh. Draw the timeline of a crash, compute the worst-case staleness, and choose the caller's timeout and retry.

Exercise 3 (1 hour). Write a decision record choosing between Kubernetes Services and Consul for a company with services on Kubernetes and on virtual machines.

Exercise 4 (2 hours). Design a graceful shutdown for a service that must finish 10-second requests, using the termination grace period and the removal of its endpoint. List what to measure.

Related reading

• API Gateway and Service Mesh: where discovery sits beside gateways and meshes.
• ShopStream Stage 8: services: ShopStream's service split and its Kubernetes Service.
• Load Balancing: how a proxy picks among instances.
• Resilience Patterns: timeouts, retries and circuit breakers for stale answers.
• CAP Theorem and PACELC: the consistency choice behind registry reads.
• Consensus and Leader Election: how registry clusters agree.

Official documentation

• Kubernetes: Service, DNS for Services and Pods, EndpointSlices and Virtual IPs and Service Proxies.
• Kubernetes task page: Configure liveness, readiness and startup probes.
• Consul: Service discovery concepts and DNS static lookups.
• Consul: Health check configuration reference and Anti-entropy.
• Netflix Eureka wiki: Eureka at a glance, client-server communication and peer-to-peer communication.