Microservices Best Practices

Asynchronous (Kafka/RabbitMQ): Use for event notifications, cross-domain data propagation, and long-running operations. The producer doesn't wait for consumers — far more resilient under load spikes.

A practical rule: if the user is waiting, go synchronous. If the work can happen in the background, go async.

Resilience Patterns You Must Implement

Circuit Breaker. When a downstream service starts failing, stop calling it for a defined period instead of hammering it with retries. Resilience4j's @CircuitBreaker annotation makes this a one-liner in Spring Boot.

Retry with exponential backoff and jitter. Naive retries on a down service create thundering herd problems. Add exponential backoff with randomised jitter:

retry_delay = base_delay × 2^attempt + random(0, base_delay)

Timeout enforcement. Every HTTP client call needs an explicit timeout. The default (often none) means a single slow upstream can exhaust your thread pool. Set connectTimeout and readTimeout explicitly.

Bulkhead. Isolate different types of calls in separate thread pools. A slow payment provider should not block your inventory lookups. Use Resilience4j's @Bulkhead for thread pool isolation.

Data Management

One database per service. Sharing a database between services creates invisible coupling — a schema change in one service breaks another, and you lose the ability to deploy independently.

Eventual consistency is the cost of independence. When OrderService creates an order, it publishes an OrderCreated event. InventoryService consumes that event and reserves stock asynchronously. The inventory is not immediately reserved, but the system remains decoupled and resilient.

Saga pattern for distributed transactions. When an operation spans multiple services (place order → reserve inventory → charge payment), use a saga:

• Choreography: each service publishes events and reacts to events from others — good for simple flows
• Orchestration: a central saga orchestrator drives the steps — better for complex, long-running flows

Observability: Three Pillars

Distributed Tracing. Assign a traceId to every incoming request and propagate it across all service calls. When an order fails, you should be able to see every service call that was part of that request in a single trace view. Use OpenTelemetry + Jaeger or Datadog APM.

Structured Logging. Every log line should be JSON with consistent fields: traceId, spanId, service, level, message, timestamp. This enables log aggregation and correlation across services in Kibana or Grafana Loki.

Metrics. Instrument every service with the four golden signals: latency, traffic, errors, and saturation. Expose them via Micrometer and scrape with Prometheus.

Without all three, debugging a production issue across 10 services is archaeology.

Deployment and Release Practices

Each service has its own CI/CD pipeline. A change to PaymentService should deploy only PaymentService — not trigger a full platform deployment.

Use feature flags for risky changes. Decoupling deployment from release lets you ship code to production, observe it under real traffic with a small percentage of users, and expand gradually. Use LaunchDarkly or a simple database-backed flag system.

Blue-green or canary deployments for zero-downtime releases. Kubernetes rolling deployments are the minimum. Canary (route 5% of traffic to the new version) is better for high-risk changes.

The Hardest Part: Organisational Alignment

Conway's Law states that your architecture will mirror your org structure. If you have 5 cross-functional product teams, you'll have 5 naturally bounded service clusters. If you have a monolithic engineering team, you'll build a distributed monolith that has all the complexity of microservices with none of the independence.

Before decomposing your services, decompose your teams.

Summary

Microservices are an organisational and operational investment as much as a technical one. Get the domain boundaries right, choose communication patterns deliberately, build resilience from the start, and invest in observability before you need it. Done well, the result is an engineering platform where teams can move independently at high velocity — and that's worth every bit of the upfront complexity.