Architect
The recovery drill
Twice a year, a drill rebuilds the shop from scratch in another region by running recovery/restore.sh on a brand-new cluster, with nobody at the keyboard. The runbook was written for the old single-server Compose setup and never tested since. Make the drill pass: within 15 minutes of starting, the new cluster serves the shop with every order of last night's backup.
Your terminal is on the manager of staging, an empty swarm identical to what the drill gets (ops/dr-policy.md): rehearse there as much as you need. Your shell exports DB_PASSWORD, like the drill's. The drill copies stack.yaml, recovery/restore.sh and backups/ to a new cluster, runs every line of the script in order, then checks http://<cluster>/products every 5 seconds and counts the orders in the database.
What your server must hold
- stack.newServerServes = 1
- data.restoredRecords ≥ 48210
- stack.recoveryMinutes ≤ 15