A worked answer to an AI gateway and integration design question for a fictional distributor: identity, deadlines, retries, idempotency and a scoring rubric.
What you'll learn: how to answer a design question about adding AI to a Java and Node.js estate: where the AI service and model gateway sit, how deadlines, retries and idempotency keys fit together, and how to score the answer.
At a glance
Figure 1. The worked answer: AI platform inside an existing estate
How to read this: existing services reach AI only through the AI service, only the model gateway holds keys and retries, and long jobs use the event bus.
Identity travels with the request, and the AI service enforces tenancy again.
Each layer's timeout is shorter than its caller's, and exactly one layer retries.
A POST that changes state carries an idempotency key.
Scenario
Composite scenario, not a real company. Lumen Industrial Supply, a fictional distributor, runs a Node.js customer portal and several Spring Boot services for orders and CRM. Three teams want AI features: summarising customer emails, answering product questions, and drafting quotes. Each team calls a model provider directly with its own key. The interviewer asks: "Design the integration so this stays safe and affordable." All numbers are illustrative.
What you will be able to do
Place an AI service and a model gateway in a Java and Node.js estate.
Set a deadline budget across layers and pick the retrying layer.
Use an idempotency key to make a retried POST safe.
A Python AI service owns prompts, retrieval and tools, so Java and Node teams call it like any internal API. The model gateway alone holds provider keys and sets rate limits, retries and fallback. Bulk work goes through the event bus.
Figure 2. A retried POST with an idempotency key
sequenceDiagram
accTitle: A retried POST with an idempotency key
accDescr: The client sends a create-draft request with an idempotency key and a 30 second deadline. The gateway forwards it with a shorter remaining deadline. The AI service records the key as in progress and calls the model gateway, which receives a rate-limit error from the provider, waits within the deadline and retries once. The result is stored under the key. If the client timed out and retries with the same key, the AI service returns the stored result and creates no second draft.
participant C as Client app
participant G as API gateway
participant A as AI service
participant M as Model gateway
participant L as LLM provider
C->>G: POST create draft, key K, deadline 30 s
G->>A: forward K and user, 28 s left
A->>A: first time K seen, mark in progress
A->>M: prompt, 25 s left
M->>L: attempt 1
L-->>M: rate-limit error
M->>M: wait, stay inside the deadline
M->>L: attempt 2
L-->>M: response
M-->>A: result
A-->>G: result stored under K
C->>G: retry after a timeout, same key K
G->>A: forward K
A-->>G: stored result, no second draft
How to read this: the key makes the second request return the first result, and the only retry against the provider happens in the model gateway.
Under RFC 9110 section 9.2.2, POST is not among the idempotent methods, and a client should not automatically retry a non-idempotent request unless it knows the request is effectively idempotent or can detect that the first one was never applied. So a retried POST needs a key. The IETF Idempotency-Key header is an Internet-Draft, not an RFC, as of 2026-11-02.
Steps 4 to 6: trade-offs, failures, measurement
Decision
Default
Switch when
Separate service or library
Separate Python service
One team, one JVM: a library may be enough
Failure
Response
Measured by
Caller times out and retries
Idempotency key returns the stored result
Duplicate-draft count, target zero
Wrong tenant's data returned
Tenancy checked in the AI service
Cross-tenant test cases
At the customer: a platform team will ask "who holds the provider keys and who gets the bill". Answer with the gateway and the usage tags.
Worked example
Assumptions (illustrative): client timeout 30 seconds; each inner layer gets less time than its caller; every layer would retry up to 3 attempts if allowed.
Layer
Timeout
Retries allowed
Client app
30 s
no
API gateway
28 s
no
AI service
25 s
no
Model gateway
20 s
yes, up to 3 attempts
If the client, gateway and model gateway each tried 3 times, one request could cause 3 x 3 x 3 = 27 provider calls. With one retrying layer it is at most 3. Shorter inner timeouts let the caller return a clear error.
Rubric scores (weak 0, acceptable 1, strong 2):
Criterion
Strong looks like
Score
Boundary and placement
Reasons and a switch condition
2
Identity and tenancy
Passed on, checked again inside
2
Timeouts and retries
Deadline budget, one retrying layer
2
Idempotency
Key, stored result, retry behaviour
2
Measurement
Provider calls per request, duplicates, thresholds
1
Total 9 of 10.
Practice task
Answer a variant in 35 minutes.
Prompt: "The Node.js portal and a Spring Boot order service both need AI summaries of customer emails. Design the integration."
Draw the components, mark where identity is checked, and write a four-layer deadline budget with one retrying layer.
Say what happens when the portal retries a request that is still running.
Exercise
Spot the flaw. A candidate says: "The portal, the order service and the AI service each retry three times on a timeout. Every team stores its own provider key. The create-draft endpoint is a POST, and retries are rare, so we need nothing extra."
Answer
Three defects. Stacked retries multiply: 27 provider calls for one request. Keys in every team spread secrets and hide usage. A retried POST can create two drafts. Fix: one retrying layer, keys only in the model gateway, an idempotency key on the POST.
Task: rewrite the description to score strong. Done when it has a table of timeouts that shrink inward, exactly one retrying layer, a stated behaviour for a repeated key, and you can compute the worst-case provider calls per request.
Hint 1: who should give up first?
The innermost layer, so the caller can still answer the user.
Hint 2: what does the second request return?
Stored result, "in progress", or a refusal because the payload differs?
Common mistakes
Letting every team call the provider directly.
Equal timeouts at every layer.
Retrying in several layers.
Forgetting that a retried POST can run twice.
Knowledge check
1. Why should only one layer retry?
Retries multiply: three layers of three attempts can produce 27 provider calls for one request. One retrying layer, closest to the provider, bounds the total.
2. Why must inner timeouts be shorter than outer ones?
An inner layer that outlasts its caller works for nobody. A shorter timeout lets the caller return a clear error.
3. What does an idempotency key change about a retried POST?
RFC 9110 says POST is not idempotent, so a retry could run the action twice. With a key, the server returns the stored result, so the user gets one draft.
4. Why a separate AI service rather than code in each Spring Boot service?
One service owns prompts, retrieval, tools and tracing, so every team shares one contract.
5. Who should hold the provider keys?
Only the model gateway. That narrows the secret to one place, sets rate limits and fallback once, and tags usage by team.
You can now
Place an AI service and model gateway in a Java and Node.js estate.
Write a deadline budget and name the one retrying layer.
Explain what an idempotency key does and its standards status.
Lesson 4 of 6 · FDE Interview: Answer Framework and Design Exercises
A worked answer to an AI gateway and integration design question for a fictional distributor: identity, deadlines, retries, idempotency and a scoring rubric.
In the rest of this lesson
Worked example
Practice task
Exercise
Common mistakes
Knowledge check
You can now
Enrolling is free. It unlocks every lesson in FDE Interview: Answer Framework and Design Exercises, tracks your progress across devices and keeps your quiz scores. The newsletter is optional.
Real architecture decisions, LLMOps patterns that survive production, and engineering leadership advice — from 12+ years of building at enterprise scale. Free. No spam. Unsubscribe anytime.
Get new articles by email
Free. We email a confirmation link first, and you can unsubscribe from any email.