Service Mesh: Sidecars, mTLS, Policy and the Cost of Another Layer

What you'll learn: By the end of this lesson you will be able to follow one service-to-service call through a mesh, explain how workloads get certificates and how mutual TLS and authorization policy use them, say what a mesh does to retries and timeouts (and why that can multiply load), compare sidecar and ambient designs, and decide whether a system should adopt a mesh at all.

Picture a company where every desk has a phone, and every team keeps inventing its own habits for calling other teams: some redial three times, some never hang up, some write nothing down. A switchboard operator who sits beside each desk and handles every call the same way would fix that. A small office would have the operators hear every conversation, charge by the minute for their desks, and need their own supervisor. That supervisor is the part people forget.

A service mesh is that arrangement for network calls between services. The overview lesson, API Gateway and Service Mesh, places it beside the gateway, and ShopStream stage 8 deferred the decision. This lesson goes deeper: certificates, policy, the retry trap, designs without a sidecar, and the cost.

Verified on 2026-11-08 against the Istio documentation (1.31 pages), the Linkerd documentation (2.20 era), the Envoy documentation and the Kubernetes documentation (sidecar containers, stable since v1.33). Behaviour differs between meshes and versions: check the version you run.

The core idea

A mesh moves the repeatable parts of east-west calls (identity, encryption, authorization, retries, timeouts, telemetry) out of each service's code and into proxies that a control plane configures. The application sends an ordinary request; a proxy next to it carries the call across the network.

Three illustrations run through the lesson: a minimal example (one call and a short policy), a realistic example (ShopStream deciding on a mesh), and a failure example (stacked retries, and strict mode rejecting a caller outside the mesh).

What the proxy does for one call

Envoy, the proxy in Istio's data plane, is described in its documentation as a self-contained process that runs alongside every application server; the application talks to localhost and does not need to know the network topology. At its core it is an L3/L4 proxy with a pluggable filter chain, plus an HTTP filter layer for routing, buffering and rate limiting, dynamic configuration APIs for hosts, routes, listeners and certificates, and active health checking with passive outlier detection. Linkerd takes a different route: linkerd2-proxy is described as an ultralight micro-proxy written in Rust, designed only for the mesh.

Because the proxy is a separate process, it works for any language and can be upgraded without touching the application. The price is one or two extra processes on every call.

Figure 1. One call, two proxies. How to read this: the application code at both ends is unchanged. The outbound proxy decides where the call goes and how it is retried. The inbound proxy decides whether the caller is allowed. Linkerd's documentation divides the jobs the same way.

The proxy learns where to send the call from service discovery data that the control plane pushes down. Istio's architecture page says istiod abstracts platform service discovery, converts routing rules into proxy configuration and acts as a certificate authority, and that Envoy proxies are the only Istio components that interact with data plane traffic: the control plane configures proxies but does not carry calls.

Identity: where mutual TLS comes from

Ordinary TLS proves the server to the client. Mutual TLS (mTLS) also proves the client: both present certificates. The hard part is the certificates: one per workload, tied to something a policy can name, replaced before expiry.

Istio's documentation describes the flow. Every workload gets an X.509 certificate. On Kubernetes the identity is a service account. The Istio agent beside each Envoy proxy creates a private key and a signing request and sends it with its credentials to istiod, whose certificate authority validates them before signing. Envoy then asks the agent for the key and certificate through its secret discovery API, and the agent watches expiry and repeats the process. Linkerd's identity service does the equivalent as a certificate authority that accepts requests from proxies; its documentation says the issued certificates expire after 24 hours and are rotated automatically.

Figure 2. Certificates are issued per workload and renewed automatically. How to read this: nobody copies a certificate by hand. The agent proves who the workload is, receives a short-lived certificate and renews it. This removes forgotten manual rotation and adds a control plane and root that someone must watch.

One edge deserves a warning. Linkerd's documentation states that the trust anchor created by its default install command expires after 365 days and then needs manual rotation, and that the issuer certificate has a similar lifetime unless you rotate it or use automation such as cert-manager. Automatic leaf rotation does not rotate the root of trust.

Policy: who may call whom

Once calls carry identities, policy can use them. Two kinds matter.

Peer authentication decides what kind of traffic a workload accepts. Istio defines PERMISSIVE (accept mutual TLS and plain text), STRICT (only mutual TLS) and DISABLE. A mesh-wide policy with the mode unset uses PERMISSIVE, and the narrowest matching policy wins: workload, then namespace, then mesh. Permissive mode makes migration possible: services join while callers that have not joined still work, and STRICT comes last.

Authorization policy decides who may do what. Istio supports ALLOW, DENY and CUSTOM actions, evaluated in the order CUSTOM, DENY, then ALLOW, and a deny always takes precedence. A workload with no authorization policy allows all requests. Deny-by-default begins only once the workload has at least one ALLOW policy. Policies are additive.

This is the minimal example (Istio syntax, illustrative, not run here). It lets only Orders call the Payments charge endpoint:

apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
  name: payments-allow-orders
  namespace: shop
spec:
  selector:
    matchLabels:
      app: payments
  action: ALLOW
  rules:
  - from:
    - source:
        principals: ["cluster.local/ns/shop/sa/orders"]
    to:
    - operation:
        methods: ["POST"]
        paths: ["/v1/charges"]

The identity format is <trust domain>/ns/<namespace>/sa/<service account>, and Istio's reference says the principals field requires mTLS. Identity policy is only as strong as the encryption under it: a plain-text caller has no identity to check. Once this policy exists, every other caller of Payments is denied.

Traffic rules, and the retry trap

A mesh can also steer traffic. Istio's traffic-management page describes weighted routing (a canary sending 5 percent of requests to a new version), retries, timeouts, circuit breaking through connection-pool limits and outlier detection, and fault injection. Its documented defaults are worth knowing: the default retry behaviour for HTTP is two retries, the Envoy timeout for HTTP requests is disabled, and, per the VirtualService reference, the default retried conditions are connect-failure,refused-stream,unavailable,cancelled. With attempts set, a request can be sent up to one plus that many times, and attempts: 0 disables retries.

The same page warns that mesh failure handling and application failure handling work independently and may conflict, so the shorter timeout wins. Linkerd's page says retries are opt-in, belong only on idempotent methods, and can take a struggling service down. This is the failure example, and the arithmetic is simple.

Say Orders tries an Inventory call up to three times in its code, and the mesh sends each of those up to three times: a persistent failure produces up to nine requests per original call. If the real failure is one hop deeper, and each Inventory request calls a Warehouse service with the same two layers, those nine become up to 81. These are worst-case counts for illustration, assuming every layer retries on the same error.

The rule that follows is to retry in one place per hop, give it a budget, and retry only idempotent operations. See resilience patterns for backoff and breakers. If the application already handles it, switch the mesh off for that route:

apiVersion: networking.istio.io/v1
kind: VirtualService
metadata:
  name: inventory
  namespace: shop
spec:
  hosts:
  - inventory.shop.svc.cluster.local
  http:
  - route:
    - destination:
        host: inventory.shop.svc.cluster.local
    timeout: 2s
    retries:
      attempts: 0

Where the proxy lives: sidecar, ambient, or neither

The classic design puts a proxy in every pod. Istio's sidecar injection uses a mutating admission webhook, enabled with the istio-injection=enabled namespace label, and injects at pod creation, so existing pods keep running without a sidecar until they are recreated. Kubernetes models a sidecar as an init container with restartPolicy: Always, which follows init-container ordering and keeps running for the life of the Pod; the feature is stable since v1.33 (first available in v1.28) and lets a Job finish while its sidecar is still running. Linkerd's documentation says its proxy is a native sidecar by default as of 2.20.

Istio's ambient mode is the alternative: a per-node layer 4 proxy (ztunnel) handles mTLS, authentication, layer 4 authorization and telemetry, with an optional per-namespace layer 7 proxy (a waypoint, an Envoy deployment outside application pods) for the full feature set. Pods need no proxy of their own, and Istio's docs say sidecar and ambient pods can coexist in one mesh. The ztunnel does not parse HTTP, so HTTP rules need a waypoint.

Figure 3. Where the proxy runs. How to read this: in sidecar mode, every pod carries its own proxy. In ambient mode, pods hold only the application, traffic is redirected to the node's ztunnel, and an HTTP-aware waypoint is added only for the namespaces that need layer 7 rules. The picture is simplified: in Istio's data-plane documentation the encrypted tunnels run between the source and destination pods.

The proxy costs resources. In Istio's performance page, for Istio 1.24 at 1,000 HTTP requests per second with 1 KB payloads, a sidecar with two worker threads used about 0.20 vCPU and 60 MB of memory, a waypoint about 0.25 vCPU and 60 MB, and a ztunnel about 0.06 vCPU and 12 MB. The page says different hardware gives different values and gives no fixed latency figure. As a shape: a hundred pods (illustrative) at that load would add about 20 vCPU and 6 GB for sidecars.

When a mesh is the wrong tool

Figure 4. A decision, not a trend. How to read this: adopt a mesh for a requirement you can name, with someone to run it, starting with the smallest feature that meets it.

In practice: ShopStream decides on a mesh

This is an educational reference design, not the verified internal architecture of any company. Numbers are illustrative. This is a realistic scenario, not a case history.

The situation

At stage 10, ShopStream has seven services on Kubernetes and a new requirement: a security review wants service-to-service traffic encrypted, and only Orders may call the payment-charging endpoint. A two-person platform team runs the cluster.

Starting architecture

Figure 5. The starting point. How to read this: inside the cluster, nothing proves who is calling Payments, and a nightly reconciliation job calls it from outside any mesh.

Options considered

The decision

Option B. Start in PERMISSIVE, mesh one namespace at a time, add the single ALLOW policy for Payments, and move Payments to STRICT last. Leave retries and timeouts in the existing library code for now; the mesh adds none for Inventory calls (see the attempts: 0 example). The reconciliation job's namespace is meshed before Payments moves to STRICT.

Rollout and verification

Mesh one namespace at a time and roll its Deployments, because injection happens at pod creation. Confirm each pod has a proxy and that proxies are connected to the control plane, send test calls from an allowed and a disallowed workload, and watch error rate and latency per service for a day before the next namespace.

The 2 a.m. failure

The page reads: Payments returns connection errors to the nightly reconciliation job right after STRICT. Checks: is the job's pod meshed (no proxy because its namespace was never labelled)? Does STRICT apply to the Payments workload (it accepts only mutual TLS)? Istio's documentation says plain-text callers are rejected in STRICT. Mitigation: set Payments back to PERMISSIVE for that workload, or mesh the job's namespace and restart its pods. Lasting fix: list every caller of Payments from real traffic before STRICT, and add the job to the rollout checklist. A second sharp edge in the same incident class is health checks. Istio's page says kubelet HTTP probes fail under mutual TLS without a rewrite, which the built-in profiles enable by default.

Architect's notes

• Name the requirement first. "mTLS between services" is a requirement; "we should have a mesh" is not.
• Own the root of trust. Know when the trust anchor expires and who rotates it.
• Budget the control plane: upgrades, scaling and its own outage.

Common mistakes

• Turning on STRICT before listing callers, so a job outside the mesh is rejected.
• Retries in both application and mesh, multiplying load in an outage.
• Assuming an authorization policy exists by default. With none, Istio allows every request.
• Ignoring trust anchor expiry, because leaf certificates rotate by themselves.

Key takeaways

• A mesh is proxies plus a control plane that pushes configuration and certificates.
• mTLS identity comes from per-workload certificates, rotated automatically, tied to a service account in Istio and Linkerd.
• Authorization needs mTLS to have an identity to check; in Istio, a workload without any policy allows all requests.
• Mesh and application retries compound; keep one.
• Sidecars cost per pod; ambient moves layer 4 work to a node proxy.
• Adopt for a named requirement and only with a team to run it.

Practice exercises

Exercise 1 (30 min). Compute the worst-case calls to a failing service when the application, a sidecar and a gateway each try three times.

Exercise 2 (45 min). Write the AuthorizationPolicy that lets only the checkout service account call GET /v1/stock on Inventory.

Exercise 3 (1 hour). Draw a migration order for seven services from PERMISSIVE to STRICT, with what you check before each step.

Exercise 4 (2 hours). Write a decision record on sidecar versus ambient for a team of two, listing the facts you would still need to measure.

Related reading

• API Gateway and Service Mesh: the edge versus the interior.
• Service Discovery: how the proxy learns where to send the call.
• Resilience Patterns: retries, backoff and breakers.
• Idempotent and Commutative Operations: when a retry is safe.
• ShopStream Stage 8: services: where the mesh was deferred.
• Observability and SLOs: what mesh telemetry feeds.

Official documentation

• Istio: Security concepts, Architecture and Traffic management.
• Istio: Ambient mode overview and Performance and scalability.
• Linkerd: Architecture, Automatic mTLS and Retries and timeouts.
• Envoy: What is Envoy.
• Kubernetes: Sidecar containers.