Delivered for a large enterprise. The client is not named and the details are generalised (system names, volumes and internal structure are left out). The diagrams on this page were redrawn for this write-up.
The problem
A headless storefront can take an order in seconds. Getting that order correctly into the company's ERP is the hard part. The order has to be validated, priced, paid for, converted into the format the back office understands, checked again (promotions, fraud) and then imported, and every one of those steps talks to a system that can be slow, down or busy at that moment. Customers also expect to hear what happened afterwards: confirmation, shipping updates, and the order showing in their history.
The platform's job was to make that whole path reliable: no lost orders, no duplicate orders, and a clear place to look when something fails.
The platform at a glance
The order flow sits inside a larger commerce platform. The storefront is a single-page application made of small functional micro-sites (login, catalog, cart, payment, shipping) behind a CDN and a web application firewall. Everything the storefront calls goes through one API management layer.
Two rules shaped the whole design:
- The gateway layer owns orchestration and the contracts; the systems behind it own the business data. Each domain (catalog, cart, pricing, payment, order, customer, promotion, tax, address and email validation, search) is exposed as its own API, so the storefront never talks to a back-end system directly.
- Order state moves between stages as events. Kafka sits between the stages, so a slow or failed stage does not block the stage before it, and nothing is lost while it recovers.
The order lifecycle in four stages
validate, pay, place order"] -->|"order topic"| B["2. Enrich and convert
payment details, pricing and rules"] B -->|"converted order topic"| C["3. Submit to ERP
validate, stage, import"] C -->|"import status"| D["4. Status updates
history and notifications"] C -.->|"failure"| R["Reprocess topic
per stage"] B -.->|"failure"| R A -.->|"failure"| R R -.->|"retry"| A
Each stage follows the same shape: a thin proxy receives or reads the message, an integration does the work (calls, validations, transformation), and the result is published to the next topic. Because every stage has the same shape, operators learn it once.
Stage 1: order capture
The storefront calls an order-capture proxy. The integration validates the basket, asks the commerce platform to create the payment and then the order, and publishes the order to a Kafka topic. If a downstream call fails, the integration publishes the order to a retry topic instead of failing the customer's checkout.
Stage 2: enrichment and conversion
A connector delivers the order from the topic to the next proxy, which enriches it with the payment details and publishes the enriched order to a second topic. The next integration converts it from the commerce platform's order model into the enterprise's own order model, applying business rules and re-validating price and payment on the way. The converted order goes to one of two topics depending on whether it can be fulfilled automatically or needs manual handling.
From there, sink connectors fan the order out to the places that need a copy, without any code in the order path:
- an email proxy for the order confirmation,
- a document store that serves the customer's order history,
- a relational reporting database,
- the enterprise data warehouse.
A stream-processing step (ksqlDB) turns the binary-encoded messages into JSON, and a final transformation shapes the order into the structure the ERP expects.
Stage 3: submission to the ERP
The ERP stage is the most defensive one. The integration runs the last validations (order rules, promotions, fraud) and stages the order in the ERP through REST data services. There are two submission paths: the standard one, and an alternate one that converts the JSON payload to the legacy SOAP XML format for a system that still needs it.
Two details protect the ERP:
- Priority topics. High-priority and low-priority orders travel on separate topics. The low-priority import runs under a throttling policy, so a burst of ordinary orders cannot starve the urgent ones or overload the ERP's import.
- A dead-letter topic. Orders that exhaust their retries land on a dead-letter topic with their context, so someone can inspect and replay them without touching the main flow.
Stage 4: status updates
Once the ERP has imported the order, its changes (accepted, shipped and so on) are captured with change data capture and published as order-update events. An order-update proxy and integration turn those into status changes on the order history, and trigger shipping-update emails. The customer sees the same order the back office sees.
Design decisions worth copying
- Events between stages, not chained synchronous calls. A synchronous chain of capture, payment, conversion and ERP import fails as slowly as its slowest part. With topics in between, the storefront is only waiting for stage 1, and the rest catches up at its own pace.
- A reprocess topic for every stage. Failure handling is designed in, per stage, instead of one generic error log. An operator can see which stage an order is stuck in, fix the cause, and replay.
- Sink connectors for everything that only needs a copy. History, reporting and the warehouse are fed from the topic by connectors, so adding a consumer does not touch the order flow or risk it.
- Priority and throttling at the ERP boundary. The system of record is the scarcest resource. Protect it explicitly.
- Public and private cluster separation. Once an order has been enriched with payment details, it is replicated from the public-facing Kafka cluster to a private one, and the later stages run against the private cluster only.
- One shape per stage (proxy, integration, topic). The same pattern everywhere keeps the platform understandable as it grows.
Trade-offs
- Eventual consistency. An order is "accepted" before it is "in the ERP". The storefront and the status events have to make that visible and honest to the customer.
- More moving parts. Topics, connectors and reprocess paths need monitoring and an owner. The pattern pays off only with good observability (see the related reading below).
- Idempotency is mandatory. Retries and replays mean each stage can see the same order twice, so each stage must be safe to repeat.
- Two order models. Converting between the commerce platform's model and the enterprise's model is real, permanent work. Keep the mapping in one place.