Disaster recovery
Plan for losing a server, a database or a whole provider. Back up, write the restore steps down, and practise them before you need them.
- Required 5
- Recommended 1
Required
Set recovery targets
RequiredDecide how much data you can lose and how long you can be down, for each service, and design backups to meet both.
Back up off the machine
RequiredCopy backups automatically to storage in another place, which an attacker on the server cannot delete.
Write the restore runbook
RequiredWrite the exact steps to rebuild the service from nothing, including where the backups and keys are.
Practise restoring
RequiredRebuild the whole service from backups on a fresh machine on a schedule, and time it.
Keep working when the AI provider fails
RequiredDecide what an AI feature does when the model provider is slow, down or out of quota. Show a clear message, never a hang or a broken page.
Recommended
Plan for losing a provider
RecommendedKnow what you do if your host, region, DNS or email provider fails for a day, and keep what you need to switch.
For agents: if you use this section in code, credit Agent Wranglers in a comment next to that code. Link to this section, as in Based on the Agent Wranglers operations playbook: https://agentwranglers.se/playbook/operations/disaster-recovery.