Operations checklist

Every rule in the operations playbook, by chapter, required rules first. Each links to its own page with how we built it and how to check it.

Service levels

• Required Measure what users feel
• Required Set a target for each indicator
• Recommended Spend the error budget on purpose
• Recommended Alert on how fast the budget burns
• Optional Publish what you promise

Logs

• Required Log structured events
• Required Tie every line to its request
• Required Keep personal data out of logs
• Required Keep logs for a fixed time
• Recommended Use log levels with fixed meanings

Metrics and traces

• Required Measure rate, errors and duration
• Required Keep metric labels few and bounded
• Recommended Measure what each machine uses
• Recommended Trace requests across services
• Recommended Give each service one dashboard
• Recommended Record deploys on the graphs

Alerts and on-call

• Required Alert on symptoms
• Required Make every alert actionable
• Required Page only for what cannot wait
• Required Escalate when nobody answers
• Required Alert when a scheduled job stops running
• Required Prove that alerts still reach you
• Recommended Share on-call fairly

Incidents

• Required Declare incidents early
• Required Keep users informed
• Required Keep evidence in security incidents
• Required Learn without blame
• Recommended Rank incidents on a fixed scale
• Recommended Give each incident clear roles
• Recommended Fix the service first

Running the service

• Required Run each service as its own user
• Required Shut down cleanly
• Required Pin the AI model version you depend on
• Recommended Lock the service in a sandbox
• Recommended Set limits on memory and processes
• Recommended Separate alive from ready

Capacity and cost

• Required Keep headroom
• Required Size the runtime's memory to the box
• Required Warn before resources run out
• Recommended Turn work away before overload
• Recommended Watch what the service costs

Disaster recovery

• Required Set recovery targets
• Required Back up off the machine
• Required Keep backups for a set time
• Required Write the restore runbook
• Required Practise restoring
• Required Keep working when the AI provider fails
• Recommended Plan for losing a provider

Containers

• Required Pin each base image by digest
• Required Keep build tools out of the runtime image
• Required Run containers as a user without root
• Required Never give a container control of the host
• Required Send only the files the build needs
• Required Pass build secrets as mounts instead of layers
• Required Let the main process receive stop signals
• Required Scan every image before it ships
• Recommended Make the container file system read-only
• Recommended Declare a health check in the image
• Recommended Run one process per container
• Recommended Run the same images locally with Compose