What you'll learn: By the end of this lesson you will be able to list the jobs an API gateway should (and should not) do, compare the main service discovery approaches, explain what a service mesh adds with sidecar proxies and mTLS, and decide when a simpler setup is the better choice.
Picture a large office building. Visitors do not wander the corridors looking for the right department. They stop at a front desk, show ID, say who they want to see, and get directed. That desk is the API gateway: one controlled entrance for everyone outside the building.
Inside, employees walk between departments all day. In a big building, you might also want badge readers on every door, a log of who visited whom, and a rule that says "if the finance department does not answer in two minutes, stop knocking". That is the service mesh: shared rules for traffic between the departments, enforced at every door instead of being re-implemented by each employee.
Both ideas appear once a system has grown from one application into many services. This lesson explains why they exist, what each one costs, and when you can happily live without them.
The core idea
Once you split an application into services (see architecture styles), two new kinds of traffic appear:
North-south traffic enters from outside: browsers, mobile apps, partner systems calling your public API.
East-west traffic moves between your own services: the order service calling the inventory service.
An API gateway manages north-south traffic. A service mesh manages east-west traffic. They solve the same family of problem (cross-cutting concerns such as security, routing, retries and telemetry) at different places, so that every team does not rebuild those concerns inside every service.
If you only have one or two services, you may need neither. We will return to that at the end.
What an API gateway does
A gateway is a reverse proxy (a server that sits in front of your servers and forwards requests to them) with extra intelligence. Typical responsibilities:
Routing. Map /orders/* to the orders service and /catalog/* to the catalog service. Clients see one hostname; you can move or split services behind it without telling them.
Authentication. Verify the caller's token or API key once at the edge, then pass a trusted identity (for example a header with the user id) inward. Services still need to check authorisation, meaning what that user may do. See security.
Rate limiting and quotas. Reject abusive or over-quota callers before they reach a service. See rate limiting.
TLS termination. Decrypt HTTPS at the edge so backends can speak simpler connections (though many teams re-encrypt internally).
Request and response shaping. Add or strip headers, translate protocols (public REST to internal gRPC), compress responses, and cache safe responses.
Observability. Emit one access log line and one set of metrics per request, plus a trace or correlation id that follows the request inward.
Aggregation. Take one client call, fan it out to several services, and combine the results so a mobile app makes one round trip instead of five.
The sequence below shows an authenticated call: the gateway checks the token once, then passes a trusted identity inward.
sequenceDiagram
participant C as Client
participant G as Gateway
participant K as Identity provider keys
participant S as Orders service
C->>G: request with bearer token
G->>K: fetch signing keys (cached)
G->>G: verify signature, expiry, audience
G->>S: request plus verified identity
S->>S: authorisation check
S-->>G: response
G-->>C: response
How to read it: the gateway authenticates (who are you?), the service authorises (what may you do?), and signing keys are fetched rarely and cached, so the identity provider is not on the hot path.
Backend for Frontend (BFF)
A single generic gateway often ends up serving very different clients. A phone on a slow network wants small payloads and few round trips. An admin web page wants rich detail. The Backend for Frontend pattern gives each client type its own thin gateway layer, owned by the team that builds that client. The mobile BFF aggregates and trims; the web BFF returns more.
The cost is more deployables and some duplicated logic. The benefit is that client needs stop fighting each other inside one shared gateway.
WARNING
A gateway is a tempting place to put business logic ("just compute the discount here"). Resist it. A gateway full of business rules becomes a bottleneck that every team must edit, and a single point of failure with the blast radius of your whole product. Keep it to cross-cutting concerns; keep domain rules in services.
Gateway versus load balancer
A load balancer spreads traffic over identical copies of one service (see load balancing). A gateway routes between different services and applies policy. In practice, products blur the line: a layer-7 load balancer can do path routing, and many gateways load-balance. Think of them as roles, not necessarily separate boxes.
Service discovery: how services find each other
In a fixed setup you could hard-code inventory.internal:8080. In a cloud, instances start, stop and move constantly, so addresses change. Service discovery answers the question: "which addresses are healthy copies of the inventory service right now?"
Approach
How it works
Cost
When it fits
DNS-based
A name such as inventory.internal resolves to one or more addresses; clients re-resolve periodically
Simple, universal. Caching and TTLs mean stale answers, and plain DNS knows little about health
Small or medium systems; Kubernetes also publishes service names through cluster DNS
Client-side discovery
The caller queries a registry (for example Consul or etcd-backed) and picks an instance itself, doing its own load balancing
No extra network hop; but every client needs a discovery library in every language
Fewer languages, performance-sensitive paths
Server-side discovery
The caller talks to a stable load balancer or proxy, which consults the registry
Clients stay simple; the proxy is an extra hop and something to keep highly available
Mixed languages, simple clients
Platform-provided
The orchestrator (for example Kubernetes) maintains the registry and exposes stable service names
Little work for you; tied to the platform
Most modern container deployments
Whatever the mechanism, the registry must learn about health. Instances either register themselves and send heartbeats, or the platform probes them. An instance that stops responding must drop out quickly, otherwise callers keep sending traffic to a dead server.
NOTE
Discovery data is a form of cache. Clients may hold an old list for a few seconds. Good clients therefore also use timeouts and retry on a different instance, because the list can always be slightly wrong.
What a service mesh adds
As services multiply, every team needs the same networking behaviours for east-west calls: timeouts, retries, encryption, metrics, traffic shifting. Writing these into each service, in each language, produces inconsistent and often buggy copies.
A service mesh moves them out of application code into the infrastructure. The classic design uses sidecar proxies: next to every service instance runs a small proxy (Envoy is a widely used one) that intercepts all inbound and outbound traffic. The application talks to what it thinks is the network; the sidecar does the work.
The mesh has two parts:
Data plane: the fleet of proxies that carry real traffic.
Control plane: a central component that pushes configuration, routing rules and certificates to the proxies. It is not in the request path.
What the mesh typically provides:
Mutual TLS (mTLS). Both sides present certificates, so each service proves who it is and traffic is encrypted. The mesh issues and rotates certificates automatically, which is the part teams rarely manage well by hand.
Retries and timeouts. Configured centrally, applied uniformly.
Traffic management. Send 5 percent of requests to a new version (canary release), mirror traffic for testing, or fail over between zones.
Telemetry. Uniform request counts, latencies and trace spans for every service-to-service call without touching application code.
Policy. "Only the orders service may call payments."
Some meshes avoid per-pod sidecars, using a shared proxy per node or kernel-level mechanisms instead. The idea is the same; the trade-off is between isolation, resource overhead and operational complexity.
IMPORTANT
Mesh-level retries do not make an operation safe to repeat. If a retried request is not idempotent (see API design), the mesh may charge a card twice. Retries also multiply load during an outage. Set retry budgets deliberately.
Gateway, mesh and library compared
Concern
Gateway
Service mesh
Library inside each service
Traffic covered
North-south
East-west
Whatever the code calls
Language independence
Yes
Yes
No, one per language
Extra network hop
One, at the edge
One or two local hops per call
None
Operational cost
Moderate
High: control plane, upgrades, debugging
Low initially, high drift over time
Best for
Public API, edge policy
Many services, strict internal security
Few services, one or two languages
A useful mental check: a gateway answers "who may come in and where do they go?" A mesh answers "how do my services talk to each other safely and observably?"
When you do not need them
Added infrastructure is added failure modes. Every proxy is a component that can misconfigure, add latency (typically a small amount per hop, but it adds up across a chain of calls) and make debugging harder because the failure may be in the proxy rather than your code.
One or two languages share a well-maintained client library.
A managed load balancer plus platform-level network policies already meet your security needs.
No team has capacity to operate a control plane.
You can probably skip a dedicated gateway when:
A single application serves the public API and a standard reverse proxy handles TLS and routing.
Your cloud provider's managed gateway or load balancer already covers routing and authentication.
A sensible growth path: begin with a reverse proxy at the edge, add gateway features as public-API needs appear, adopt a mesh only when east-west security or consistency across many services becomes a measured pain. For deeper treatment of the service side, see microservices best practices.
The same reasoning as a decision flowchart you can run in a design review.
flowchart TD
Q1{"Public API, partners or several client types?"} -->|no| N1["Reverse proxy and TLS"]
Q1 -->|yes| G["Add an API gateway"]
G --> Q2{"Must every internal call be authenticated and encrypted?"}
N1 --> Q2
Q2 -->|no| L["Shared client library plus network policies"]
Q2 -->|yes| Q3{"Many services in several languages?"}
Q3 -->|no| L2["Library with mTLS or a platform feature"]
Q3 -->|yes| Q4{"Can a team run a control plane?"}
Q4 -->|no| M2["Managed mesh offering or postpone"]
Q4 -->|yes| M["Adopt a mesh, two services first"]
How to read it: you reach the mesh only by answering "yes" to a security or consistency need and "yes" to being able to operate it. Every other path ends somewhere cheaper.
In practice: a fintech's internal APIs, gateway authentication and mTLS
NOTE
A realistic scenario, not a case history. This is a composite example built from common production patterns. It does not describe any specific company, and every number in it is an illustrative assumption.
The situation
A card-linked savings product has a mobile app, a small partner API and nine services on Kubernetes, built by about 30 engineers in four teams (assumed). Services are written in Java, Node.js and Go. Nginx ingress terminates TLS. Each service validates the caller's JWT itself with a per-language library, and services call each other over plain HTTP inside the cluster. Data lives in PostgreSQL (one database per service) and Kafka carries events.
An external audit is due in 90 days (assumed) and has asked two questions: can you prove that only approved callers reach the ledger service, and is internal traffic encrypted? A code review also found drift. One service accepts expired tokens inside a generous clock-skew window, and another never checks the audience claim. Meanwhile the checkout path has a p99 of 1.8 seconds at about 1,200 requests per second at peak (assumed), and nobody can say which hop is slow.
Starting architecture
How to read it: the edge only encrypts. Authentication logic is copied into each service, and any pod in the cluster can call the ledger.
Options considered
Option
What it fixes
What it costs
Risk
A. Smaller change: one vetted JWT library per language with shared test vectors, plus Kubernetes NetworkPolicies
Auth drift; blocks some unwanted pod-to-pod paths
Library upgrades in three languages; policy upkeep
No encryption or caller identity between services unless every language also implements mTLS and certificate rotation, which is the drift problem again; the audit question stays open
B. Gateway only: an Envoy-based gateway validates tokens, services trust an identity header
One place for north-south authentication
One new component to run
The header is forgeable by any pod, so east-west trust is unchanged
C. Gateway plus mesh: gateway authentication, then Istio-style sidecars with strict mTLS and authorisation policies
Control plane, sidecar resources, upgrade treadmill, harder debugging
Operational complexity for a small platform team
The decision
Field
Entry
Context
Audit in 90 days; three languages; nine services; two engineers who can own platform work
Decision
Option C, staged: gateway first, then mesh for the ledger and payments namespaces, then the rest
Good consequences
Tokens validated once; service identity comes from certificates, not headers; ledger callers listed explicitly; uniform per-hop latency and error metrics
Bad consequences
Extra sidecar CPU and memory on every pod; control plane to upgrade and keep available; one more layer to blame in incidents
Revisit when
Mesh-attributed incidents exceed two per quarter after month six, or the added latency per hop exceeds a 5 ms budget (assumed)
Rollout and verification
Ship in reversible stages, each gated by a metric.
flowchart LR
P1["1. Gateway in shadow mode, log mismatches"] --> P2["2. Gateway enforces, services keep checks"]
P2 --> P3["3. Mesh PERMISSIVE on ledger and payments"]
P3 --> P4["4. STRICT mTLS plus authorisation policies"]
P4 --> P5["5. Remove duplicated JWT checks"]
At stage 1 the gateway records where its verdict differs from the services' own; you proceed only when mismatches are explained. At stage 3 PERMISSIVE mode accepts both plaintext and mTLS, so you can watch the share of requests arriving over mTLS climb toward 100 percent before switching to STRICT. Rollback at any stage is a configuration flip.
Dashboards that prove it worked: 401 and 403 rates per service (a spike after a stage means a policy is wrong), percentage of east-west requests using mTLS, p99 latency per hop compared with the pre-mesh baseline, sidecar CPU and memory, and remaining certificate validity. Run a load test at about twice the peak of 1,200 requests per second (assumed) on a staging copy before stage 4.
The 2 a.m. failure
The page reads: "Ledger 5xx above 20 percent for 5 minutes", followed by a second alert, "Workload certificate validity under 2 hours". Cause: a control plane upgrade that evening left the mesh control plane down. Running proxies kept working on cached configuration, but they could not renew their short-lived certificates (assume the platform team configured a 12-hour lifetime), and the first ones have now expired.
sequenceDiagram
participant P as Payments sidecar
participant L as Ledger sidecar
participant CP as Control plane
L-xCP: renew certificate (no answer)
P->>L: mTLS handshake
L-->>P: handshake fails, certificate expired
P-->>P: reports 503 upstream connect error
First three checks:
kubectl get pods -n istio-system to see whether the control plane is running.
istioctl proxy-status to see which proxies are stale or disconnected.
istioctl proxy-config secret <pod> -n ledger to read the actual certificate expiry.
Mitigation: roll back the control plane upgrade, then confirm the proxies fetch fresh certificates on their own; restart affected pods one at a time only if they do not recover. If that is slow, security can approve a time-boxed fall back to plaintext for the ledger namespace as a documented break-glass step (relaxing only the server-side policy is not enough, because clients keep originating mTLS), reverted after recovery. Follow-ups: alert on control plane availability and on the lowest remaining certificate validity (a leading indicator, as in observability and SLOs), run the control plane with replicas across zones, canary control plane upgrades in a staging cluster, and rehearse this failure in a game day.
Architect's notes
Measure the baseline before adding a hop. Record per-call latency for your deepest call chains first; a mesh adds a small cost per hop, and chains multiply it.
Ask in the review: what if the control plane is down for longer than a certificate lives? The answer shows whether the team understands its failure mode and its break-glass path.
Strip identity headers at the edge. Any identity header the gateway trusts must be removed from inbound external requests, or a client can forge it.
Decide who is paged. The platform team owns the gateway and mesh; service teams own their authorisation policies. Write that split down before the first incident.
Budget the sidecars. Per-pod CPU and memory multiply by pod count, and the mesh upgrade cadence becomes a permanent cost on the platform team.
Defer multi-cluster meshing and mesh-wide retries. Neither is needed for the audit, and both widen the blast radius. Pick one layer for retries when you get there.
Common mistakes
Putting business logic in the gateway. It couples every team to one deployable and makes the gateway a bottleneck. Keep it to routing, security and shaping.
Treating the gateway as a single point of failure. Run several instances across zones behind a highly available entry point, and watch its latency like a product feature.
Adopting a mesh too early. You pay in complexity and latency before you get value. Adopt it when you can name the pain it solves.
Retrying everywhere without idempotency or budgets. Retries at the client, gateway and mesh multiply: three layers with three attempts each can produce 27 calls (3 x 3 x 3) per user request. Limit retries to one layer and cap the total.
Trusting the internal network. "It is behind the gateway, so it is safe" fails after one compromised service. Use mTLS and authorisation checks between services too.
Ignoring discovery staleness. Without timeouts and retry-to-another-instance, a stale registry sends traffic to dead servers.
Key takeaways
Gateway handles north-south, mesh handles east-west. Different traffic, different tools.
Keep the gateway thin. Routing, authentication, rate limiting and shaping belong there; business rules do not.
BFFs fit different clients. Give mobile and web their own aggregation layer when their needs diverge.
Discovery is a cache. Registries and DNS can be stale, so pair them with health checks, timeouts and retries.
A mesh centralises networking policy. Sidecars give you mTLS, uniform retries, telemetry and traffic shifting without per-language libraries.
Mesh features have costs. Extra hops, a control plane to operate, and harder debugging.
Retries need care. Idempotency and a single layer of retries prevent duplicate effects and retry storms.
Start simple. A reverse proxy and shared libraries carry many systems a long way.
Practice exercises
Exercise 1 — Starter (30 min): Draw a diagram of a system with a mobile app, a gateway and four services (users, catalogue, orders, payments). Mark each arrow north-south or east-west, and list which of the gateway responsibilities from this lesson apply to each entry point.
Exercise 2 — Intermediate (2 hours): A checkout request passes through a client (3 attempts), the gateway (3 attempts) and a mesh sidecar (3 attempts) before reaching payments. Calculate the worst-case number of calls reaching payments if it is down for one request. Then redesign the retry policy so the worst case is at most 3 calls, and say which layer you chose to own retries and why.
Exercise 3 — Advanced (half day): Take a system you know (or invent one with 12 services). Write a one-page decision record choosing between: a reverse proxy only, a gateway, or a gateway plus mesh. Include discovery approach, the measured pain that would justify a mesh, the operating cost in team time, a rollout plan that adopts the mesh for two services first, and an exit plan if it does not pay off.
Related reading
Load balancing — the lower-level building block that gateways and meshes build on.
Rate limiting — the algorithm behind the gateway feature mentioned here.
Lesson 9 of 14 · System Design Building Blocks: Data, Messaging and Scale
Learn what an API gateway and a service mesh each do, how services find each other, and how to tell when your system genuinely needs either one.
In the rest of this lesson
Service discovery: how services find each other
What a service mesh adds
Gateway, mesh and library compared
When you do not need them
In practice: a fintech's internal APIs, gateway authentication and mTLS
Architect's notes
Enrolling is free. It unlocks every lesson in System Design Building Blocks: Data, Messaging and Scale, tracks your progress across devices and keeps your quiz scores. The newsletter is optional.
Real architecture decisions, LLMOps patterns that survive production, and engineering leadership advice — from 12+ years of building at enterprise scale. Free. No spam. Unsubscribe anytime.
Get new articles by email
Free. We email a confirmation link first, and you can unsubscribe from any email.