What you'll learn: how to ask a customer for recovery targets, choose one of four recovery strategies per component, and plan what is specific to an AI assistant.
At a glance
Figure 1. A failover from decision to verified recovery
How to read this: the clock starts at the alarm; a person decides, because a false failover also costs time and data.
- Ask for a recovery time and a recovery point per component.
- Pick the cheapest of four strategies that meets each target.
- Rebuild what is derived; copy what is state, if a copy is allowed.
Scenario
Composite scenario, not a real company. Lumen Industrial Supply, a fictional distributor, runs the Order and Policy Assistant for 24 customer-service agents. The Head of Customer Operations says: "If the data centre burns down, I want it back." Agents still have the old process. All numbers are illustrative.
What you will be able to do
- Ask for a recovery time and a recovery point objective and record both.
- Choose a recovery strategy per component and justify its cost.
- Decide whether to rebuild or restore the search index.
Prerequisites
Multi-Tenant AI Design and Latency, Capacity and Availability Targets; theory: AI Architect: reliability and scaling.
Core ideas
Recovery targets: two numbers per component
The recovery time objective (RTO) is, in the AWS whitepaper's words, "the maximum acceptable delay between the interruption of service and restoration of service". The recovery point objective (RPO) is "the maximum acceptable amount of time since the last data recovery point". Google's guide defines the same pair.
Figure 2. Questions that produce the numbers
| Ask the customer | It sets | Lumen answer (illustrative) |
|---|---|---|
| What do agents do while it is down? | RTO | Use the old process |
| What does an hour cost, and when is it worst? | RTO | Month-end is worst |
| What could you not recreate? | RPO | Approval proposals |
| Which regions may hold a copy? | Where | One named region (open) |
How to read this: both sources start from a business impact analysis.
AWS says not to buy recovery that costs more than the loss it prevents, unless something else, such as a regulation, requires it.
Four strategies, one cost ladder
Figure 3. Recovery strategies, cheapest and slowest first
| Strategy | What waits in the recovery region | Cost and recovery time |
|---|---|---|
| Backup and restore | Backups; you redeploy infrastructure and code | Lowest cost; longest recovery |
| Pilot light | Replicated data, servers switched off | Low cost; faster than restore |
| Warm standby | A scaled-down working copy that takes traffic at once | Higher cost; faster again |
| Active/active | The workload serving traffic in several regions | Most complex and costly; near-zero recovery time for most disasters |
How to read this: the order and descriptions are AWS's; Google calls the idea cold, warm or hot and says smaller targets cost more.
AWS also says automatic failover on an alarm "should be used with caution"; manual initiation with automated steps is common.
What is different for an AI assistant
These are design choices, not source claims.
- Model provider. Put a second provider or region behind the model gateway, with the targets lesson's degraded mode (cited passages, no generated answer). Rerun the golden set on it.
- Versions in Git. Prompts, policy exports and model identifiers deploy to both regions together, as AWS advises for pilot light.
- Secrets and quotas. Keys must already exist in the recovery region; AWS says to check its service quotas.
- Data residency. Google notes that compliance regulations also apply to the recovered environment; see the data-residency case. Legal decides.
Predict first. The newest index snapshot predates this quarter's change, when P-RET-01 v3 (30 days) replaced v2 (14 days). The primary region is lost and you restore that snapshot. What will the assistant quote?
Answer
The superseded 14-day rule. A snapshot carries the old state, so rebuild the index from the current policy export.