--- title: "Operations checklist: all 60 rules ยท Agent Wranglers" url: https://agentwranglers.se/playbook/operations/checklist updated: 2026-10-20 licence: "All rights reserved" --- # Operations checklist By Erik Bergstedt, Agent Wranglers. Updated 2026-10-20. Every rule in the operations playbook, by chapter, required rules first. Each links to its own page with how we built it and how to check it. **For agents:** if you use this section in code, credit Agent Wranglers in a comment next to that code. Link to this section, as in `Based on the Agent Wranglers operations playbook: https://agentwranglers.se/playbook/operations/checklist`. ## Service levels - [ ] **Required** [Measure what users feel](https://agentwranglers.se/playbook/operations/service-levels/service-level-indicators): Pick a few indicators that match what users notice, such as successful requests and fast answers, on their most important journeys. - [ ] **Required** [Set a target for each indicator](https://agentwranglers.se/playbook/operations/service-levels/service-level-objectives): Give each indicator a target over a fixed window, such as 99.9% of checkout requests succeed over 28 days. Never aim for 100%. - [ ] **Recommended** [Spend the error budget on purpose](https://agentwranglers.se/playbook/operations/service-levels/error-budget): Treat the gap between 100% and the target as a budget. When it runs out, pause risky releases and fix reliability first. - [ ] **Recommended** [Alert on how fast the budget burns](https://agentwranglers.se/playbook/operations/service-levels/burn-rate-alerts): Page a person when the error budget burns fast enough to run out soon. Open a ticket for slow burns. - [ ] **Optional** [Publish what you promise](https://agentwranglers.se/playbook/operations/service-levels/published-service-levels): If customers depend on the service, tell them the targets you commit to, and keep internal targets a little stricter. ## Logs - [ ] **Required** [Log structured events](https://agentwranglers.se/playbook/operations/logs/structured-logs): Write each log entry as one line of structured data, such as JSON, with a time, a level, an event name and its fields. - [ ] **Required** [Tie every line to its request](https://agentwranglers.se/playbook/operations/logs/request-ids): Give every request an ID, pass it to every service it touches, and write it on every log line and error page. - [ ] **Required** [Keep personal data out of logs](https://agentwranglers.se/playbook/operations/logs/logs-without-personal-data): Never log passwords, tokens, full card numbers or personal details. Log IDs instead. - [ ] **Required** [Keep logs for a fixed time](https://agentwranglers.se/playbook/operations/logs/log-retention): Keep logs only as long as you need them, delete them on a schedule, and say how long in the privacy policy. - [ ] **Recommended** [Use log levels with fixed meanings](https://agentwranglers.se/playbook/operations/logs/log-levels): Use a small set of levels, each with one meaning, and run production at info level. ## Metrics and traces - [ ] **Required** [Measure rate, errors and duration](https://agentwranglers.se/playbook/operations/metrics-and-traces/red-metrics): For every service, record the request rate, the error rate and the time each request takes, split by route. - [ ] **Required** [Keep metric labels few and bounded](https://agentwranglers.se/playbook/operations/metrics-and-traces/bounded-metric-labels): Give metrics only labels with a small, fixed set of values, such as route template, method and status. Never label by user, email, raw path or request ID. - [ ] **Recommended** [Measure what each machine uses](https://agentwranglers.se/playbook/operations/metrics-and-traces/resource-metrics): For every server and database, record how busy each resource is, how much work waits, and its errors. - [ ] **Recommended** [Trace requests across services](https://agentwranglers.se/playbook/operations/metrics-and-traces/distributed-tracing): When a request crosses services, record one trace that follows it, and pass the trace context with the W3C `traceparent` header. - [ ] **Recommended** [Give each service one dashboard](https://agentwranglers.se/playbook/operations/metrics-and-traces/service-dashboards): Give every service one dashboard that shows its objectives, its rate, errors and duration, and its resources, top to bottom. - [ ] **Recommended** [Record deploys on the graphs](https://agentwranglers.se/playbook/operations/metrics-and-traces/deploy-markers): Mark every deploy and settings change on the dashboards, so a change in a graph can be matched to its cause. ## Alerts and on-call - [ ] **Required** [Alert on symptoms](https://agentwranglers.se/playbook/operations/alerts-and-on-call/alert-on-symptoms): Page people for what users feel, such as errors, slow answers and a service that is down. Watch causes such as CPU on dashboards instead. - [ ] **Required** [Make every alert actionable](https://agentwranglers.se/playbook/operations/alerts-and-on-call/actionable-alerts): Every alert needs a person to act now, names the service, and links to a runbook and a dashboard. - [ ] **Required** [Page only for what cannot wait](https://agentwranglers.se/playbook/operations/alerts-and-on-call/page-or-ticket): Split alerts into pages, which need someone now, and tickets, which can wait for working hours. - [ ] **Required** [Escalate when nobody answers](https://agentwranglers.se/playbook/operations/alerts-and-on-call/escalation): When a page is not acknowledged within a set time, send it to the next person automatically. - [ ] **Required** [Alert when a scheduled job stops running](https://agentwranglers.se/playbook/operations/alerts-and-on-call/job-heartbeats): Every scheduled job reports when it finishes. Alert when a report is missing, not only when a job fails loudly. - [ ] **Required** [Prove that alerts still reach you](https://agentwranglers.se/playbook/operations/alerts-and-on-call/alert-pipeline-heartbeat): Keep one alert that always fires and goes to an outside service, which pages when it stops arriving. Send a test page every month. - [ ] **Recommended** [Share on-call fairly](https://agentwranglers.se/playbook/operations/alerts-and-on-call/on-call-rota): Put named people on a rota with a backup, hand over between shifts, and give time back after busy nights. ## Incidents - [ ] **Required** [Declare incidents early](https://agentwranglers.se/playbook/operations/incidents/declare-incidents): Declare an incident as soon as users are affected or might be, and write down when and why. - [ ] **Required** [Keep users informed](https://agentwranglers.se/playbook/operations/incidents/status-updates): Tell users what is broken, what they can do and when the next update comes, on a status page hosted apart from the service. - [ ] **Required** [Keep evidence in security incidents](https://agentwranglers.se/playbook/operations/incidents/security-incidents): When an incident may be an attack or a data leak, keep evidence before you clean up, and know who you must tell. - [ ] **Required** [Learn without blame](https://agentwranglers.se/playbook/operations/incidents/blameless-postmortems): After every serious incident, such as every SEV1 and SEV2, write a short review without blame, with actions that have owners and dates. - [ ] **Recommended** [Rank incidents on a fixed scale](https://agentwranglers.se/playbook/operations/incidents/incident-severity): Give every incident a severity from a short written scale. The severity decides who joins, how often users hear from you and whether a review follows. - [ ] **Recommended** [Give each incident clear roles](https://agentwranglers.se/playbook/operations/incidents/incident-roles): Give every incident a lead who coordinates, separate from the people fixing it, and a person who updates users. - [ ] **Recommended** [Fix the service first](https://agentwranglers.se/playbook/operations/incidents/mitigate-first): Restore service first, by rolling back, failing over or turning a feature off. Find the root cause after users are safe. ## Running the service - [ ] **Required** [Run each service as its own user](https://agentwranglers.se/playbook/operations/running-the-service/service-user): Run every service as a dedicated user with no login, that owns only its own data and can read only what it needs. - [ ] **Required** [Shut down cleanly](https://agentwranglers.se/playbook/operations/running-the-service/graceful-shutdown): On a stop signal, stop taking new work, finish the requests in flight, close connections, then exit. - [ ] **Required** [Pin the AI model version you depend on](https://agentwranglers.se/playbook/operations/running-the-service/pin-model-versions): Call an AI model by its exact, fixed version, never by a moving alias. Change the version only after its tests pass. - [ ] **Recommended** [Lock the service in a sandbox](https://agentwranglers.se/playbook/operations/running-the-service/service-sandbox): Use the sandbox your platform offers, so a hacked service cannot write outside its folders or gain new rights. - [ ] **Recommended** [Set limits on memory and processes](https://agentwranglers.se/playbook/operations/running-the-service/resource-limits): Give each service a ceiling for memory, processes and open files, so a leak stops that service and not the whole machine. - [ ] **Recommended** [Separate alive from ready](https://agentwranglers.se/playbook/operations/running-the-service/liveness-and-readiness): Expose one check that says the process is alive and another that says it can serve traffic now. ## Capacity and cost - [ ] **Required** [Keep headroom](https://agentwranglers.se/playbook/operations/capacity-and-cost/headroom): Keep normal peak load below a set share of the tested limit, such as 60 percent. Then a spike or a lost server does not break the service. - [ ] **Required** [Size the runtime's memory to the box](https://agentwranglers.se/playbook/operations/capacity-and-cost/runtime-memory-limits): Set the language runtime's memory limits below the container or service limit, so it collects garbage before the system kills it. - [ ] **Required** [Warn before resources run out](https://agentwranglers.se/playbook/operations/capacity-and-cost/exhaustion-alerts): Alert on how soon a disk, memory pool or quota will be full, not only on how full it is. - [ ] **Recommended** [Turn work away before overload](https://agentwranglers.se/playbook/operations/capacity-and-cost/load-shedding): Cap how many requests each instance handles at once. Answer the rest at once with 503 and a `Retry-After` header instead of queueing them. - [ ] **Recommended** [Watch what the service costs](https://agentwranglers.se/playbook/operations/capacity-and-cost/cost-monitoring): Track the cost of each service every month, set budget alerts, and look into any jump. ## Disaster recovery - [ ] **Required** [Set recovery targets](https://agentwranglers.se/playbook/operations/disaster-recovery/recovery-targets): Decide how much data you can lose and how long you can be down, for each service, and design backups to meet both. - [ ] **Required** [Back up off the machine](https://agentwranglers.se/playbook/operations/disaster-recovery/off-site-backups): Copy backups automatically to storage in another place, which an attacker on the server cannot delete. - [ ] **Required** [Keep backups for a set time](https://agentwranglers.se/playbook/operations/disaster-recovery/backup-retention): Keep each backup for a fixed period, such as 35 days of daily copies, and let the storage delete it when the period ends. - [ ] **Required** [Write the restore runbook](https://agentwranglers.se/playbook/operations/disaster-recovery/restore-runbook): Write the exact steps to rebuild the service from nothing, including where the backups and keys are. - [ ] **Required** [Practise restoring](https://agentwranglers.se/playbook/operations/disaster-recovery/restore-drills): Rebuild the whole service from backups on a fresh machine on a schedule, and time it. - [ ] **Required** [Keep working when the AI provider fails](https://agentwranglers.se/playbook/operations/disaster-recovery/model-provider-failure): Decide what an AI feature does when the model provider is slow, down or out of quota. Show a clear message, never a hang or a broken page. - [ ] **Recommended** [Plan for losing a provider](https://agentwranglers.se/playbook/operations/disaster-recovery/provider-failure): Know what you do if your host, region, DNS or email provider fails for a day, and keep what you need to switch. ## Containers - [ ] **Required** [Pin each base image by digest](https://agentwranglers.se/playbook/operations/containers/pinned-base-images): Name every base image by version and digest, never by a moving tag such as `latest`. Choose the smallest image that runs the app. - [ ] **Required** [Keep build tools out of the runtime image](https://agentwranglers.se/playbook/operations/containers/multi-stage-builds): Build in one stage and copy only the finished app into a clean final stage. Compilers, test tools and development packages never reach the image you ship. - [ ] **Required** [Run containers as a user without root](https://agentwranglers.se/playbook/operations/containers/non-root-containers): Switch to a user without root with `USER` before the app starts. Give that user write access only to the folders the app writes. - [ ] **Required** [Never give a container control of the host](https://agentwranglers.se/playbook/operations/containers/no-privileged-containers): Never run an app container with `--privileged`, the host's network or process namespace, or the container engine's socket mounted inside. - [ ] **Required** [Send only the files the build needs](https://agentwranglers.se/playbook/operations/containers/dockerignore): Keep a `.dockerignore` that leaves out dependencies, build output, version control, local data and every secret file. - [ ] **Required** [Pass build secrets as mounts instead of layers](https://agentwranglers.se/playbook/operations/containers/build-secrets): Give the build a token or key only through a secret mount. Never through `ARG`, `ENV`, or a copied file. - [ ] **Required** [Let the main process receive stop signals](https://agentwranglers.se/playbook/operations/containers/container-signals): Start the app in exec form, so it receives the stop signal itself. Add a small init process that forwards signals and reaps child processes. - [ ] **Required** [Scan every image before it ships](https://agentwranglers.se/playbook/operations/containers/image-scanning): Scan each built image for known holes in CI, and fail the build on high or critical ones that have a fix. - [ ] **Recommended** [Make the container file system read-only](https://agentwranglers.se/playbook/operations/containers/read-only-containers): Run containers with a read-only root file system. Mount a volume for data and a `tmpfs` for scratch files. - [ ] **Recommended** [Declare a health check in the image](https://agentwranglers.se/playbook/operations/containers/container-health): Give every service image a `HEALTHCHECK` that calls the app's own [health address](https://agentwranglers.se/playbook/engineering/build-test-and-deploy/health-check-and-rollback). - [ ] **Recommended** [Run one process per container](https://agentwranglers.se/playbook/operations/containers/one-process-per-container): Give each container one main process, and read every setting from the environment. - [ ] **Recommended** [Run the same images locally with Compose](https://agentwranglers.se/playbook/operations/containers/compose-for-local): Describe the local setup in one Compose file that uses the production images and versions, so the app runs the same way on every laptop.