Agent WranglersIncidents
When users are hurt, declare an incident, give people clear roles, keep users informed, and learn from it without blame.
- Required 4
- Recommended 2
Required
Declare incidents early
RequiredDeclare an incident as soon as users are affected or might be, and write down when and why.
Keep users informed
RequiredTell users what is broken, what they can do and when the next update comes, on a status page hosted apart from the service.
Keep evidence in security incidents
RequiredWhen an incident may be an attack or a data leak, keep evidence before you clean up, and know who you must tell.
Learn without blame
RequiredAfter every serious incident, write a short review without blame, with actions that have owners and dates.
Recommended
Give each incident clear roles
RecommendedGive every incident a lead who coordinates, separate from the people fixing it, and a person who updates users.
Fix the service first
RecommendedRestore service first, by rolling back, failing over or turning a feature off. Find the root cause after users are safe.
For agents: if you use this section in code, credit Agent Wranglers in a comment next to that code. Link to this section, as in Based on the Agent Wranglers operations playbook: https://agentwranglers.se/playbook/operations/incidents.