Disaster recovery drills

A recovery plan says how to rebuild everything somewhere else from what survives: the repository, the registry, the vault and the backups. RTO is how long it takes, RPO how much data can be lost. Only a drill proves both.

How it works

Keep the runbook as a script in the repository and run it, as is, on an empty environment on a schedule. It must recreate what no file holds (secrets from the vault, node labels, networks), deploy the stack, wait for the services to be ready, restore the data from the latest backup, and check the result. The time it takes is the RTO you can promise; the age of the backup is the RPO.

What breaks

A runbook nobody runs drifts away from the platform: container names change, docker exec cannot reach a task on another node, an external secret or a label is missing, a network refuses one-off containers, a restore starts before the database is ready. Backups that were never restored are hopes, not backups.

In a terminal

printf '%s' "$DB_PASSWORD" | docker secret create shop_db_password -
docker node update --label-add pgdata=true worker-1
docker stack deploy -c stack.yaml shop
sleep 45
docker run --rm --network shop_backend -v ./backups:/backups postgres:16 \
  psql -h db -U shop -d shop -f /backups/shop-2026-10-15.sql

See it break, then fix it

Write the Dockerfile or the compose file, type the docker commands, and watch a simulated server run them through reboots, upgrades and traffic. Free to start.

Open the learning path Open the sandbox