Operations and observability playbook

An operations playbook for any service in production, from one small server to many. It covers how to know the service works, how to hear when it breaks, and how to recover. A chapter on containers covers the images many services now run in.

The engineering playbook has the short version for every project. This playbook goes deeper for teams that run a service other people depend on.

Start with Service levels. They decide what to measure, when to alert and how much risk a release may take.

Chapters

1. Service levels

   Decide what good service means for users, measure it, and set a target. The target then sets alerts, release pace and how much risk you may take.

   • Required 2
   • Recommended 2
   • Optional 1

2. Logs

   Write one structured line per event, tie it to the request that caused it, and keep personal data out. Then logs answer questions instead of hiding them.

   • Required 4
   • Recommended 1

3. Metrics and traces

   Count requests, errors and time for every service, follow each request across services, and show it on one dashboard per service that anyone can read.

   • Required 2
   • Recommended 4

4. Alerts and on-call

   Wake a person only for problems users feel and a person can fix. Give every alert a runbook, a clear owner and a path to escalate.

   • Required 6
   • Recommended 1

5. Incidents

   When users are hurt, declare an incident, give people clear roles, keep users informed, and learn from it without blame.

   • Required 4
   • Recommended 3

6. Running the service

   Run each service as its own locked-down user, with limits, a clean shutdown and honest health checks, so one fault stays contained.

   • Required 3
   • Recommended 3

7. Capacity and cost

   Know how much load the service can carry, keep headroom, and warn before disks, memory or budgets run out instead of after.

   • Required 3
   • Recommended 2

8. Disaster recovery

   Plan for losing a server, a database or a whole provider. Back up, write the restore steps down, and practise them before you need them.

   • Required 6
   • Recommended 1

9. Containers

   Build small, pinned images that hold only what runs. Run them without root, read-only, with a health check, and scan every image before it ships.

   • Required 8
   • Recommended 4

All 60 rules on one page