Microservices Done Right: Patterns I've Learned from 12 Years of Enterprise Architecture

Twelve years of building enterprise software will teach you things that no architecture book quite captures. Not because the books are wrong — most of the canonical microservices material is technically sound. But books describe what good looks like. They rarely describe what the path from a legacy monolith to a healthy microservices architecture actually feels like, mid-project, with fourteen stakeholders and a hard deadline.

I've done this migration more than once. I've led teams through it at global companies — healthcare, logistics, financial services, retail. This is what I know about microservices that I wish I'd known earlier.

What Microservices Actually Is (and Isn't)

Let me start with the thing that causes more wasted engineering effort than anything else: people confuse "microservices" with "small services."

Microservices is not about service size. It's about independent deployability. A microservice is a service that can be built, tested, deployed, and scaled independently of every other service in the system. The "micro" refers to the scope of responsibility, not the lines of code.

I've seen teams split a perfectly reasonable service into five "microservices" because they thought smaller was better. Each one shared a database. Every deployment required coordinating all five. They had created distributed coupling, not independent services. It was harder to operate than the original monolith.

The question to ask for every service boundary is: can I deploy this service without touching anything else? If the answer is no, you have a distributed monolith, not microservices.

The Decomposition Decision

How you split a monolith into services matters enormously. The wrong decomposition creates more problems than it solves.

The two decomposition strategies I've used with the most success:

Domain-Driven Design (DDD) Bounded Contexts. Identify the natural language boundaries in your domain. The language the Order Management team uses to talk about "orders" is different from the language the Fulfilment team uses, even when they're talking about the same entity. That boundary — where the meaning of terms shifts — is where a service boundary belongs.

Capability decomposition. Group functions by the business capability they serve. Order placement, order tracking, and order history are all "Orders" — they belong together. Payment processing, refunds, and payment method management are all "Payments" — they belong together. Don't split by technical layer (API service, business service, data service) — that creates services that can never deploy independently.

The left side looks like microservices but it's not. A change to any business logic touches all three layers, which must all be deployed together. The right side is genuinely independent — Orders can deploy without touching Payments.

Communication Patterns: Choose Carefully

Once you've decomposed correctly, the next critical decision is how services communicate.

Synchronous (REST / gRPC)

Use synchronous communication when:

• The caller needs a response before it can continue
• Latency matters and round-trips are bounded
• The operation is simple and transactional

The problem with synchronous chaining: if any service in the chain is slow or down, the entire request fails. At scale, this becomes a resilience problem. I'll cover how to handle that below.

Asynchronous (Event-Driven with Kafka)

Use asynchronous communication when:

• Services need to react to something that happened, not request something
• Decoupling between producer and consumer is important
• You need fan-out (one event, multiple consumers)
• You need durability (events must not be lost if a consumer is temporarily down)

Notice that Order Service doesn't know — or care — that Notification Service, Analytics, Inventory, and Fraud Review all consume the order.placed event. Adding a new consumer is a zero-code change on the producer side. This is genuine decoupling.

I've used this pattern extensively with Confluent Kafka on enterprise programs. The key discipline: design your events around facts that happened, not commands to be executed. order.placed (a fact) is correct. send-order-confirmation-email (a command masquerading as an event) is not — it creates implicit coupling between the producer and a specific consumer's behaviour.

The Hybrid Architecture

In practice, you'll use both. The pattern I've standardised on:

• Synchronous REST via Apigee for external consumer-facing APIs (read and write operations where the consumer needs a response)
• Asynchronous Kafka events for internal cross-domain reactions (notifications, analytics, fulfillment triggers)
• gRPC for high-throughput internal service-to-service calls where latency is critical

The Database Rule You Cannot Violate

Each service owns its data. Shared databases are the fastest way to destroy microservices independence.

If two services share a database, you cannot deploy them independently — a schema change in one breaks the other. You cannot scale them independently — they contend on the same connection pool. You cannot choose the right storage technology for each — you're locked to whatever the shared database is.

One service. One database schema. No sharing.

When services need data from each other, they have two options:

1. API call — synchronous, real-time, strong consistency
2. Event replication — the owning service publishes events; consuming services maintain their own local copy for query purposes (eventual consistency)

I've used both. Option 2 (event-driven data replication) is more complex to implement but more resilient — it removes synchronous dependency on the owning service. For read-heavy cross-service queries, it's often the right choice.

Resilience Patterns: Expect Failures

Distributed systems fail in ways that monoliths don't. A method call never times out. An HTTP call can.

The Three Resilience Controls

I think of resilience in three layers, each handling a different failure mode:

Retry at the API call layer — handles transient errors (a single request that timed out, a momentary 503). Always use exponential backoff with jitter. Never retry on POST requests without idempotency guarantees.

Circuit breaker at the service boundary — handles sustained downstream degradation. If 50% of calls to Payment Service are failing over 10 seconds, stop sending calls and give Payment Service a chance to recover. Without a circuit breaker, a degraded dependency can cascade into your service and then into anything that calls you.

Fallback at the orchestration layer — handles complete downstream unavailability. Return cached data. Return a degraded response. Return a clear error that the caller can handle gracefully.

I implement these with Spring's @Retryable, Resilience4j for circuit breakers, and Redis for cached fallback responses. In production at scale, you need all three — not just retries.

Idempotency

Every write operation in a distributed system should be idempotent — calling it twice should produce the same result as calling it once.

Implement this with idempotency keys. The client generates a unique key per logical operation and sends it in the request header. The server stores the response keyed by the idempotency key. If the same key arrives again (because the first response was lost in transit), return the stored response without re-executing.

This lets clients safely retry on network failure without fear of double-charging or double-processing.

Data Consistency: Learn to Love Eventual

In a distributed system, you give up strong consistency across service boundaries. The CAP theorem is not optional. In most microservices architectures, you choose availability and partition tolerance — which means accepting eventual consistency between services.

The key mental shift: eventual consistency is not eventual incorrectness. Data will converge. The question is how quickly, and what the system does in the window before it converges.

The patterns I use:

Saga pattern for distributed transactions. When an operation spans multiple services (place order → reserve inventory → charge payment → confirm order), each step is a separate local transaction. If any step fails, compensation transactions undo the previous steps. This replaces 2-phase commit with a sequence of event-driven actions and compensations.

The saga pattern requires careful design — compensation logic must be implemented for every step that can fail. It's more complex than a monolithic transaction, but it's the only option that works at scale across service boundaries.

Observability: You'll Wish You'd Done This First

In a monolith, a bug is in one place. In microservices, a slow request might touch eight services. Without observability infrastructure, debugging a production incident is finding a needle in eight haystacks, simultaneously.

I've made the mistake of treating observability as something to add later. Don't.

Distributed tracing with correlation IDs. Every request gets a X-Correlation-ID header assigned at the gateway. Every service propagates this header in outgoing calls and logs it in every log line. When an incident occurs, you filter your centralised logs by correlation ID and see the entire request path across all services in sequence.

Structured logging. Every log line is a JSON object with fields: timestamp, service, correlationId, level, message, and any relevant domain fields. Never use unstructured log strings in production microservices — you cannot query them reliably.

RED metrics per service. For every service, track Rate (requests per second), Errors (error rate), and Duration (latency percentiles). These three metrics tell you the health of any service at a glance. Alert on them, not on CPU and memory, which are vanity metrics for microservices.

Service Mesh or Not?

A question I get asked regularly: should we use a service mesh like Istio?

My answer: probably not yet.

A service mesh provides mutual TLS between services, fine-grained traffic policies, and service-to-service observability without code changes. It's powerful. It's also operationally complex — Istio in particular has a significant learning curve and operational overhead.

I've deployed service meshes in large organisations where the security posture required mTLS between every service-pair and the operations team had the bandwidth to own it. In most projects, the same goals — resilience, observability, auth — are better served by Resilience4j in your Spring services and proper structured logging, with a service mesh deferred until the team is ready.

Don't add a service mesh to reduce complexity. Add it when you have specific requirements — mandatory mTLS, very fine-grained traffic policies — that justify the overhead.

The Anti-Patterns I See Most Often

Shared library coupling. A shared library containing domain models, used by multiple services. Every time the library changes, every service must re-deploy. You've recreated the distributed monolith with extra steps.

Chatty synchronous chains. Service A calls B, B calls C, C calls D. A single user request creates a chain of four synchronous hops. Latency compounds; failures cascade. Redesign using events or consolidate into fewer services.

Microservices for a monolith-sized team. A team of three engineers does not need six services. The operational overhead of microservices — separate deployments, separate databases, distributed tracing, service discovery — is only worth it when the team is large enough that the Conway's Law alignment benefit materialises. Two-pizza teams can ship faster with a well-structured monolith.

No local development story. If running the full system locally requires spinning up twenty Docker containers, new engineers can't contribute productively. Keep the local development story simple: mock the services you don't need, run the services you're changing.

Where to Start

If you're migrating from a monolith, my recommended sequence:

1. Identify one high-value, low-coupling domain. Not the most complex one — the one with the clearest boundary and the least shared data.
2. Extract it into a service behind a stable API. The monolith calls the service through this API, not directly into its database.
3. Run the service and the monolith in parallel. Don't try to replace the whole monolith at once — the strangler fig pattern means you peel away capability incrementally.
4. Add the observability infrastructure before the second service. Distributed tracing and centralised logging are much harder to retrofit.
5. Standardise your service template. Once you've extracted one service, codify the pattern — build config, health checks, metrics, logging, circuit breaker configuration — into a starter template every subsequent service inherits.

Microservices done right is genuinely better than a monolith at scale. Microservices done wrong is worse than anything I've ever inherited. The difference is discipline: clear service boundaries, owned data, proper async patterns, and observability from day one.

Get those right and the rest follows.