Delivered in the insurance domain for a large enterprise. The client and the data providers are not named, and the details are generalised. The diagrams on this page were redrawn for this write-up.
The problem
Insurance applications constantly need verified data about people, vehicles and policies, and much of it comes from paid third-party providers. Every call costs money and takes time, and many applications ask for the same data again and again. Left alone, each application builds its own integration, its own cache (or none), and its own interpretation of what it is allowed to ask for.
The Data Reuse Service (DRS) is a single, central hub for that traffic. Its purpose is to cut cost, speed up responses and keep the data rules in one place.
How a request is served
An application asks for data. The service answers from the cheapest place that is allowed to answer, and only goes to the provider when it has to.
Three behaviours matter:
- Two cache layers. L2 is a fast in-memory cache (a Redis cluster). L3 is a longer-term, encrypted database (a relational or a document database, depending on the deployment). Data found in L3 is copied back to L2 so the next request is faster.
- Not everything is reusable. The policy engine decides, per type of data, whether a result may be reused at all. Requests that are not eligible go straight through to the provider and are not cached ("write-through").
- Identical concurrent requests are collapsed. If several applications ask for the same thing while the first provider call is still running, they wait for that one call instead of each paying for their own.
Architecture
Design decisions worth copying
- A stateless service in front of stateful stores. The service itself holds no data. State lives in the cache and the database, so the service scales out by adding instances and can be replaced without losing anything.
- Policy first, cache second. Whether data may be reused is a business and privacy decision, so it is made by a policy engine with an admin interface, not by each application. Changing a rule does not mean redeploying callers.
- Consent and purpose-of-use are checked, not assumed. A cached record is still personal data. The service checks that the caller is allowed to use it for the stated purpose before returning it.
- Providers are reached only through connectors behind a gateway. Retries, timeouts and circuit-breaking (the "resilience" component) live in one place, and the provider-facing gateway is separate from the internal one.
- A mock layer for non-production. Outside production the connectors call a mock service instead of the paid providers, so testing never costs money or sends real personal data to a provider.
- Secrets and encryption by default. Provider credentials come from a secrets manager, and the L3 store is encrypted.
- Audit and observability from day one. Every decision is logged to an append-only audit trail (write once, read many), traces and metrics go to the observability stack, and two views are built on top: a technical dashboard for operations and a financial view that shows what the reuse is saving on paid calls.
Trade-offs
- Freshness versus cost. A cached answer can be stale. Each type of data needs its own time-to-live and, where the provider supports it, a way to force a refresh.
- Personal data in a cache. Reuse is only acceptable with retention limits, encryption, access control and an audit trail. The privacy design is as much of the product as the cache is.
- A new shared dependency. Every application now relies on one service. It has to be highly available, and its failure modes (provider down, cache down, database down) have to be defined, not discovered.
- Policy complexity. A central rule engine is powerful and easy to over-grow. Keep rules declarative and versioned.