{"playbook":"Operations","url":"https://agentwranglers.se/playbook/operations","updated":"2026-10-21","license":"https://creativecommons.org/licenses/by/4.0/","credit":"Required by the licence: credit Agent Wranglers with a link to the rule or playbook you used.","how":"Each rule's How text is on its own page: add .md to its url.","rules":[{"chapter":"Service levels","name":"Measure what users feel","level":"Recommended","rule":"Pick a few indicators that match what users notice, such as successful requests and fast answers, on their most important journeys.","check":"Each important journey has at least one indicator written as good events divided by valid events, and a dashboard shows its last 28 days.","url":"https://agentwranglers.se/playbook/operations/service-levels/service-level-indicators","related":["/playbook/website/performance/field-measurement","/playbook/ai-features/evals-and-quality/production-quality-metrics"],"sources":[]},{"chapter":"Service levels","name":"Set a target for each indicator","level":"Recommended","rule":"Give each indicator a target over a fixed window, such as 99.9% of checkout requests succeed over 28 days. Never aim for 100%.","check":"A written document lists every objective with its indicator, target, window and owner, and its last review date is within the past three months.","url":"https://agentwranglers.se/playbook/operations/service-levels/service-level-objectives","related":["/playbook/operations/service-levels/error-budget","/playbook/operations/service-levels/published-service-levels"],"sources":[]},{"chapter":"Service levels","name":"Spend the error budget on purpose","level":"Recommended","rule":"Treat the gap between 100% and the target as a budget. When it runs out, pause risky releases and fix reliability first.","check":"A written policy says what changes when the budget is spent. The last time it was spent, the release log shows that the policy was followed.","url":"https://agentwranglers.se/playbook/operations/service-levels/error-budget","related":["/playbook/operations/service-levels/service-level-objectives"],"sources":[]},{"chapter":"Service levels","name":"Alert on how fast the budget burns","level":"Recommended","rule":"Page a person when the error budget burns fast enough to run out soon. Open a ticket for slow burns.","check":"Inject 100% errors on staging. The page fires within 5 minutes. Twice the budgeted error rate for an hour opens no page. A steady 1.5 times the budgeted rate opens a ticket within 3 days.","url":"https://agentwranglers.se/playbook/operations/service-levels/burn-rate-alerts","related":["/playbook/payments/payment-events-and-reconciliation/payment-health-alerts"],"sources":[]},{"chapter":"Service levels","name":"Publish what you promise","level":"Optional","rule":"If customers depend on the service, tell them the targets you commit to, and keep internal targets a little stricter.","check":"Every published promise has a matching internal objective that is equal or stricter, measured on the same indicator.","url":"https://agentwranglers.se/playbook/operations/service-levels/published-service-levels","related":["/playbook/operations/service-levels/service-level-objectives"],"sources":[]},{"chapter":"Logs","name":"Log structured events","level":"Recommended","rule":"Write each log entry as one line of structured data, such as JSON, with a time, a level, an event name and its fields.","check":"Pipe one hour of logs through a JSON parser, such as `jq`. Every line parses, and every line has a time, a level and an event.","url":"https://agentwranglers.se/playbook/operations/logs/structured-logs","related":["/playbook/engineering/running-in-production/logs-and-alerts","/playbook/adversarial-testing/rules-of-engagement/log-every-request","/playbook/infrastructure/servers/time-sync","/playbook/ai-features/prompts-as-code/prompt-version-in-logs","/playbook/stack/operations-tools/json-logs-and-loki"],"sources":[{"label":"OpenTelemetry Logs Data Model","url":"https://opentelemetry.io/docs/specs/otel/logs/data-model/"}]},{"chapter":"Logs","name":"Tie every line to its request","level":"Recommended","rule":"Give every request an ID, pass it to every service it touches, and write it on every log line and error page.","check":"Trigger an error on staging. The error page shows an ID, and searching the logs for it returns every line of that request, across services.","url":"https://agentwranglers.se/playbook/operations/logs/request-ids","related":["/playbook/api-design/errors/no-internal-details","/playbook/ai-features/logs-privacy-and-feedback/log-model-calls"],"sources":[{"label":"W3C Trace Context traceparent header","url":"https://www.w3.org/TR/trace-context/#traceparent-header"}]},{"chapter":"Logs","name":"Use log levels with fixed meanings","level":"Recommended","rule":"Use a small set of levels, each with one meaning, and run production at info level.","check":"In one day of production logs, every `error` line matches an error the team tracks. No `debug` lines appear.","url":"https://agentwranglers.se/playbook/operations/logs/log-levels","related":["/playbook/engineering/running-in-production/logs-and-alerts"],"sources":[{"label":"RFC 5424 section 6.2.1 syslog severity levels","url":"https://www.rfc-editor.org/rfc/rfc5424#section-6.2.1"}]},{"chapter":"Logs","name":"Keep personal data out of logs","level":"Required","risk":"data","rule":"Never log passwords, tokens, full card numbers or personal details. Log IDs instead.","check":"Sign in, reset a password and place a test order on staging. Search the logs for the test password, token and email address. None of them appear.","url":"https://agentwranglers.se/playbook/operations/logs/logs-without-personal-data","related":["/playbook/data/product-analytics/events-without-personal-data","/playbook/email/email-privacy/lean-email-logs","/playbook/adversarial-testing/data-and-secrets/private-data-out-of-logs","/playbook/ai-features/logs-privacy-and-feedback/prompt-text-retention","/playbook/stack/operations-tools/glitchtip"],"sources":[{"label":"OWASP Logging Cheat Sheet","url":"https://cheatsheetseries.owasp.org/cheatsheets/Logging_Cheat_Sheet.html"}]},{"chapter":"Logs","name":"Keep logs for a fixed time","level":"Required","risk":"law","applies":["eu-users"],"rule":"Keep logs only as long as you need them, delete them on a schedule, and say how long in the privacy policy.","check":"The oldest file or entry in each log store is younger than its stated limit, and the privacy policy names the same limit.","url":"https://agentwranglers.se/playbook/operations/logs/log-retention","related":["/playbook/ai-assisted-development/safety-and-permissions/agent-activity-log","/playbook/email/email-privacy/lean-email-logs","/playbook/compliance/audit-logs-and-access/audit-log-retention","/playbook/operations/logs/off-host-logs","/playbook/compliance/records-and-data-map/retention-schedule"],"sources":[{"label":"GDPR Article 5 on principles of processing","url":"https://eur-lex.europa.eu/eli/reg/2016/679/oj"},{"label":"OWASP Logging Cheat Sheet","url":"https://cheatsheetseries.owasp.org/cheatsheets/Logging_Cheat_Sheet.html"}]},{"chapter":"Logs","name":"Send logs off the machine as they are written","level":"Recommended","rule":"Ship every log line to a store on another machine within 60 seconds of writing it. The server's own credentials must not be able to edit or delete it there.","check":"Write a test line on the server. It appears in the central store within 60 seconds. A delete request for that line, sent with the server's own credentials, is refused.","url":"https://agentwranglers.se/playbook/operations/logs/off-host-logs","related":["/playbook/ai-assisted-development/safety-and-permissions/agent-activity-log","/playbook/compliance/audit-logs-and-access/audit-log-immutable","/playbook/operations/logs/log-retention"],"sources":[{"label":"OWASP Logging Cheat Sheet","url":"https://cheatsheetseries.owasp.org/cheatsheets/Logging_Cheat_Sheet.html"}]},{"chapter":"Metrics and traces","name":"Measure rate, errors and duration","level":"Recommended","rule":"For every service, record the request rate, the error rate and the time each request takes, split by route.","check":"For each service, a query returns requests per second, errors per second and p95 duration for the last hour, per route. For a single service, a daily script over the access log may produce the same three numbers.","url":"https://agentwranglers.se/playbook/operations/metrics-and-traces/red-metrics","related":["/playbook/api-design/resources-and-methods/status-codes","/playbook/data/operations/slow-query-log","/playbook/website/performance/server-timing","/playbook/stack/operations-tools/prometheus"],"sources":[{"label":"OpenTelemetry semantic conventions for HTTP metrics","url":"https://opentelemetry.io/docs/specs/semconv/http/http-metrics/"}]},{"chapter":"Metrics and traces","name":"Keep metric labels few and bounded","level":"Recommended","rule":"Give metrics only labels with a small, fixed set of values, such as route template, method and status. Never label by user, email, raw path or request ID.","check":"A series count per metric, such as `topk(10, count by (__name__)({__name__=~\".+\"}))` in Prometheus, shows none above the limit. No label is named `user_id`, `email`, `path` or `request_id`.","url":"https://agentwranglers.se/playbook/operations/metrics-and-traces/bounded-metric-labels","related":["/playbook/adversarial-testing/data-and-secrets/private-data-out-of-logs"],"sources":[{"label":"OpenTelemetry semantic conventions for HTTP metrics","url":"https://opentelemetry.io/docs/specs/semconv/http/http-metrics/"}]},{"chapter":"Metrics and traces","name":"Measure what each machine uses","level":"Recommended","rule":"For every server and database, record how busy each resource is, how much work waits, and its errors.","check":"A dashboard shows CPU, memory, disk and queue length for every host and the database, with no gaps in the last day.","url":"https://agentwranglers.se/playbook/operations/metrics-and-traces/resource-metrics","related":["/playbook/infrastructure/cost-and-ownership/rightsize-from-measurements"],"sources":[]},{"chapter":"Metrics and traces","name":"Trace requests across services","level":"Recommended","rule":"When a request crosses services, record one trace that follows it, and pass the trace context with the W3C `traceparent` header.","check":"Make one request that crosses two services on staging. The tracing tool shows one trace with spans from both, and the logs carry the same trace ID. A request that fails is always in the tracing tool.","url":"https://agentwranglers.se/playbook/operations/metrics-and-traces/distributed-tracing","related":["/playbook/website/performance/server-timing","/playbook/data/queries-and-indexes/no-n-plus-one","/playbook/stack/operations-tools/opentelemetry"],"sources":[{"label":"W3C Trace Context traceparent header","url":"https://www.w3.org/TR/trace-context/#traceparent-header"}]},{"chapter":"Metrics and traces","name":"Give each service one dashboard","level":"Recommended","rule":"Give every service one dashboard that shows its objectives, its rate, errors and duration, and its resources, top to bottom.","check":"Every service has exactly one dashboard, and every alert links to it. Every dashboard exists in version control.","url":"https://agentwranglers.se/playbook/operations/metrics-and-traces/service-dashboards","related":["/playbook/operations/metrics-and-traces/deploy-markers","/playbook/stack/operations-tools/grafana"],"sources":[]},{"chapter":"Metrics and traces","name":"Record deploys on the graphs","level":"Recommended","rule":"Mark every deploy and settings change on the dashboards, so a change in a graph can be matched to its cause.","check":"After a deploy, the dashboards show a marker at that time with the new version.","url":"https://agentwranglers.se/playbook/operations/metrics-and-traces/deploy-markers","related":["/playbook/engineering/build-test-and-deploy/version-every-release","/playbook/operations/metrics-and-traces/service-dashboards"],"sources":[]},{"chapter":"Alerts and on-call","name":"Alert on symptoms","level":"Recommended","rule":"Page people for what users feel, such as errors, slow answers and a service that is down. Watch causes such as CPU on dashboards instead.","check":"List every paging alert. Each one is tied to an indicator, an outside check or a resource that will run out within hours.","url":"https://agentwranglers.se/playbook/operations/alerts-and-on-call/alert-on-symptoms","related":["/playbook/engineering/running-in-production/logs-and-alerts","/playbook/website/when-things-break/browser-error-reports","/playbook/engineering/running-in-production/monitoring-uptime","/playbook/payments/payment-events-and-reconciliation/payment-health-alerts","/playbook/stack/operations-tools/prometheus"],"sources":[]},{"chapter":"Alerts and on-call","name":"Make every alert actionable","level":"Recommended","rule":"Every alert needs a person to act now, names the service, and links to a runbook and a dashboard.","check":"Every alert rule has a runbook link and a dashboard link. In the last month, no alert fired more than three times without anyone taking an action.","url":"https://agentwranglers.se/playbook/operations/alerts-and-on-call/actionable-alerts","related":["/playbook/engineering/documentation/runbooks-in-repo"],"sources":[]},{"chapter":"Alerts and on-call","name":"Page only for what cannot wait","level":"Recommended","rule":"Split alerts into pages, which need someone now, and tickets, which can wait for working hours.","check":"Every alert rule has a severity of page or ticket, and every page in the last month matches a moment users were affected.","url":"https://agentwranglers.se/playbook/operations/alerts-and-on-call/page-or-ticket","related":["/playbook/infrastructure/environments/threat-detection"],"sources":[]},{"chapter":"Alerts and on-call","name":"Share on-call fairly","level":"Recommended","rule":"Put named people on a rota with a backup, hand over between shifts, and give time back after busy nights.","check":"The paging tool shows a primary and a backup for every hour of the next month. The last handover note is less than a week old. With one person, the backup channel and the reply time are written down.","url":"https://agentwranglers.se/playbook/operations/alerts-and-on-call/on-call-rota","related":["/playbook/operations/alerts-and-on-call/escalation","/playbook/stack/operations-tools/pagerduty"],"sources":[]},{"chapter":"Alerts and on-call","name":"Escalate when nobody answers","level":"Recommended","rule":"When a page is not acknowledged within a set time, send it to the next person automatically.","check":"Send a test page and do not answer it. The backup receives it within the set time. With one person on call, the status page states the reply time and the backup channel.","url":"https://agentwranglers.se/playbook/operations/alerts-and-on-call/escalation","related":["/playbook/operations/alerts-and-on-call/on-call-rota"],"sources":[]},{"chapter":"Alerts and on-call","name":"Alert when a scheduled job stops running","level":"Recommended","rule":"Every scheduled job reports when it finishes. Alert when a report is missing, not only when a job fails loudly.","check":"Stop a job's timer on staging. An alert fires within its interval plus the margin. Make one step fail, and the alert fires the same way.","url":"https://agentwranglers.se/playbook/operations/alerts-and-on-call/job-heartbeats","related":["/playbook/payments/payment-events-and-reconciliation/missed-event-sweep"],"sources":[]},{"chapter":"Alerts and on-call","name":"Prove that alerts still reach you","level":"Recommended","rule":"Keep one alert that always fires and goes to an outside service, which pages when it stops arriving. Send a test page every month.","check":"Stop the alert manager on staging. The outside monitor pages within 15 minutes. The paging tool shows a test page from the last 31 days.","url":"https://agentwranglers.se/playbook/operations/alerts-and-on-call/alert-pipeline-heartbeat","related":["/playbook/engineering/running-in-production/monitoring-uptime"],"sources":[]},{"chapter":"Incidents","name":"Declare incidents early","level":"Recommended","rule":"Declare an incident as soon as users are affected or might be, and write down when and why.","check":"For every outage in the last quarter, an incident record exists, and it was opened within 15 minutes of the first alert or user report.","url":"https://agentwranglers.se/playbook/operations/incidents/declare-incidents","related":["/playbook/compliance/breaches-and-reporting/reporting-clocks"],"sources":[]},{"chapter":"Incidents","name":"Rank incidents on a fixed scale","level":"Recommended","rule":"Give every incident a severity from a short written scale. The severity decides who joins, how often users hear from you and whether a review follows.","check":"Every incident record has a severity from the scale, and every SEV1 and SEV2 incident has a review.","url":"https://agentwranglers.se/playbook/operations/incidents/incident-severity","related":["/playbook/compliance/breaches-and-reporting/breach-register","/playbook/compliance/breaches-and-reporting/breach-risk-method"],"sources":[]},{"chapter":"Incidents","name":"Give each incident clear roles","level":"Recommended","rule":"Give every incident a lead who coordinates, separate from the people fixing it, and a person who updates users.","check":"Every incident record names a lead and has a timeline with entries at least every 30 minutes until it closed.","url":"https://agentwranglers.se/playbook/operations/incidents/incident-roles","related":["/playbook/operations/incidents/status-updates"],"sources":[]},{"chapter":"Incidents","name":"Keep users informed","level":"Recommended","rule":"Tell users what is broken, what they can do and when the next update comes, on a status page hosted apart from the service.","check":"Stop the main site on staging. The status page still loads. During the last incident, every update arrived by the promised time.","url":"https://agentwranglers.se/playbook/operations/incidents/status-updates","related":["/playbook/website/when-things-break/maintenance-pages","/playbook/operations/incidents/incident-roles","/playbook/compliance/breaches-and-reporting/processor-breach-notice"],"sources":[]},{"chapter":"Incidents","name":"Fix the service first","level":"Recommended","rule":"Restore service first, by rolling back, failing over or turning a feature off. Find the root cause after users are safe.","check":"For each incident in the last quarter, the timeline shows when service was restored, before the time the root cause was found.","url":"https://agentwranglers.se/playbook/operations/incidents/mitigate-first","related":["/playbook/engineering/build-test-and-deploy/atomic-deploys-with-rollback","/playbook/operations/incidents/security-incidents"],"sources":[]},{"chapter":"Incidents","name":"Keep evidence in security incidents","level":"Required","risk":"law","applies":["eu-users"],"rule":"When an incident may be an attack or a data leak, keep evidence before you clean up, and know who you must tell.","check":"The security plan names who decides, who to notify and their deadlines. The last security drill or incident record shows that logs and a disk snapshot were saved before cleanup.","url":"https://agentwranglers.se/playbook/operations/incidents/security-incidents","related":["/playbook/website/privacy/breach-plan","/playbook/operations/incidents/mitigate-first","/playbook/adversarial-testing/supply-chain-and-setup/compromised-package-response","/playbook/compliance/breaches-and-reporting/reporting-clocks"],"sources":[{"label":"NIST SP 800-61 Rev. 3 Incident Response Recommendations","url":"https://csrc.nist.gov/pubs/sp/800/61/r3/final"},{"label":"GDPR Article 33 on breach notification within 72 hours","url":"https://eur-lex.europa.eu/eli/reg/2016/679/oj"}]},{"chapter":"Incidents","name":"Learn without blame","level":"Recommended","rule":"After every serious incident, such as every SEV1 and SEV2, write a short review without blame, with actions that have owners and dates.","check":"Every serious incident has a review within five working days. Every action in it has an owner and a due date, and overdue actions are listed in the team review.","url":"https://agentwranglers.se/playbook/operations/incidents/blameless-postmortems","related":["/playbook/engineering/rules-as-code/check-every-rule-in-code"],"sources":[]},{"chapter":"Running the service","name":"Run each service as its own user","level":"Required","risk":"takeover","rule":"Run every service as a dedicated user with no login, that owns only its own data and can read only what it needs.","check":"`ps -o user= -p \"$(systemctl show -p MainPID --value shop)\"` prints the service user, not root. `sudo -u nobody ls /var/lib/shop` fails with permission denied.","url":"https://agentwranglers.se/playbook/operations/running-the-service/service-user","related":["/playbook/adversarial-testing/data-and-secrets/least-privilege-keys"],"sources":[]},{"chapter":"Running the service","name":"Lock the service in a sandbox","level":"Recommended","rule":"Use the sandbox your platform offers, so a hacked service cannot write outside its folders or gain new rights.","check":"`systemd-analyze security --threshold=50 shop.service` exits 0, which means an exposure of 5.0 or lower. Otherwise the container runs with a read-only root and a non-root user.","url":"https://agentwranglers.se/playbook/operations/running-the-service/service-sandbox","related":["/playbook/operations/containers/non-root-containers","/playbook/operations/containers/read-only-containers","/playbook/adversarial-testing/data-and-secrets/least-privilege-keys"],"sources":[]},{"chapter":"Running the service","name":"Set limits on memory and processes","level":"Recommended","rule":"Give each service a ceiling for memory, processes and open files, so a leak stops that service and not the whole machine.","check":"The running unit or pod shows the limits, for example in `systemctl show shop -p MemoryMax`. Kill the process with `kill -9`, and it restarts within seconds.","url":"https://agentwranglers.se/playbook/operations/running-the-service/resource-limits","related":["/playbook/architecture/errors-and-nulls/bounded-collections","/playbook/adversarial-testing/abuse-and-limits/resource-limits","/playbook/operations/capacity-and-cost/runtime-memory-limits","/playbook/operations/containers/one-process-per-container"],"sources":[]},{"chapter":"Running the service","name":"Shut down cleanly","level":"Recommended","rule":"On a stop signal, stop taking new work, finish the requests in flight, close connections, then exit.","check":"Start a slow request, then stop the service. The request finishes, and the logs show a clean exit within the time limit.","url":"https://agentwranglers.se/playbook/operations/running-the-service/graceful-shutdown","related":["/playbook/engineering/running-in-production/stateless-processes","/playbook/engineering/build-test-and-deploy/atomic-deploys-with-rollback","/playbook/operations/containers/container-signals"],"sources":[]},{"chapter":"Running the service","name":"Separate alive from ready","level":"Recommended","rule":"Expose one check that says the process is alive and another that says it can serve traffic now.","check":"Stop the database on staging. Readiness answers 503 and traffic stops, while liveness still answers 200 and the process is not restarted.","url":"https://agentwranglers.se/playbook/operations/running-the-service/liveness-and-readiness","related":["/playbook/engineering/build-test-and-deploy/health-check-and-rollback","/playbook/operations/containers/container-health","/playbook/stack/operations-tools/uptime-kuma"],"sources":[]},{"chapter":"Running the service","name":"Pin the AI model version you depend on","level":"Recommended","applies":["ai-features"],"rule":"Call an AI model by its exact, fixed version, never by a moving alias. Change the version only after its tests pass.","check":"Search the settings and code for model names. None ends in `latest` or is an alias the provider moves, and the last version change has a passing eval run attached.","url":"https://agentwranglers.se/playbook/operations/running-the-service/pin-model-versions","related":["/playbook/testing/non-functional-tests/llm-evals","/playbook/ai-features/deciding-to-use-a-model/smallest-passing-model","/playbook/adversarial-testing/ai-agents-and-models/trusted-model-sources","/playbook/ai-features/prompts-as-code/settings-with-prompt","/playbook/ai-features/retrieval/embedding-model-version"],"sources":[]},{"chapter":"Running the service","name":"Check the service is ready before launch","level":"Recommended","rule":"Before a new service or a large feature takes users, walk through one written checklist. Fix every open item, or give it an owner and a date.","check":"Every launch in the last 12 months has a filled checklist in the repository, dated before launch day. Every item is done, or has an owner and a due date.","url":"https://agentwranglers.se/playbook/operations/running-the-service/launch-readiness","related":["/playbook/engineering/build-test-and-deploy/staged-rollouts","/playbook/testing/non-functional-tests/load-tests"],"sources":[{"label":"Google SRE book, Reliable Product Launches at Scale","url":"https://sre.google/sre-book/reliable-product-launches/"}]},{"chapter":"Capacity and cost","name":"Keep headroom","level":"Recommended","rule":"Keep normal peak load below a set share of the tested limit, such as 60 percent. Then a spike or a lost server does not break the service.","check":"Last month's peak load is below the agreed share of the tested limit. The autoscaling maximum is set.","url":"https://agentwranglers.se/playbook/operations/capacity-and-cost/headroom","related":["/playbook/testing/non-functional-tests/load-tests","/playbook/infrastructure/cost-and-ownership/rightsize-from-measurements"],"sources":[]},{"chapter":"Capacity and cost","name":"Turn work away before overload","level":"Recommended","rule":"Cap how many requests each instance handles at once. Answer the rest at once with 503 and a `Retry-After` header instead of queueing them.","check":"Load staging with twice its tested limit. Extra requests get 503 within 100 ms, accepted requests stay within their speed objective, and the service recovers without a restart.","url":"https://agentwranglers.se/playbook/operations/capacity-and-cost/load-shedding","related":["/playbook/website/security/rate-limits","/playbook/api-design/reliability/retry-backoff"],"sources":[{"label":"RFC 9110 section 15.6.4 503 Service Unavailable","url":"https://www.rfc-editor.org/rfc/rfc9110#section-15.6.4"},{"label":"RFC 9110 section 10.2.3 Retry-After","url":"https://www.rfc-editor.org/rfc/rfc9110#section-10.2.3"}]},{"chapter":"Capacity and cost","name":"Size the runtime's memory to the box","level":"Required","risk":"outage","rule":"Set the language runtime's memory limits below the container or service limit, so it collects garbage before the system kills it.","check":"The load test reports peak memory below 80 percent of the service limit, and its 95th percentile response time is unchanged by the caps.","measured":"On a reference site with a 260 MB service limit, shrinking the young generation took peak memory under load from 190 to 139 MB. Response times stayed at 42 ms.","url":"https://agentwranglers.se/playbook/operations/capacity-and-cost/runtime-memory-limits","related":["/playbook/testing/non-functional-tests/load-tests","/playbook/operations/running-the-service/resource-limits"],"sources":[]},{"chapter":"Capacity and cost","name":"Warn before resources run out","level":"Required","risk":"outage","rule":"Alert on how soon a disk, memory pool or quota will be full, not only on how full it is.","check":"Fill a test disk on staging at a steady rate. The page fires at least 1 hour before it is full.","url":"https://agentwranglers.se/playbook/operations/capacity-and-cost/exhaustion-alerts","related":["/playbook/data/operations/connection-pooling","/playbook/data/operations/capacity","/playbook/website/hosting-and-network/certificate-renewal","/playbook/operations/containers/container-log-rotation"],"sources":[]},{"chapter":"Capacity and cost","name":"Watch what the service costs","level":"Recommended","rule":"Track the cost of each service every month, set budget alerts, and look into any jump.","check":"A budget alert exists for every account, and a monthly report shows cost per service.","url":"https://agentwranglers.se/playbook/operations/capacity-and-cost/cost-monitoring","related":["/playbook/infrastructure/cost-and-ownership/required-tags","/playbook/infrastructure/cost-and-ownership/budgets-for-every-account","/playbook/infrastructure/cost-and-ownership/cost-in-review","/playbook/infrastructure/cost-and-ownership/commitment-discounts","/playbook/ai-features/evals-and-quality/eval-cost-and-latency","/playbook/ai-features/latency-cost-and-fallbacks/cost-per-call"],"sources":[]},{"chapter":"Disaster recovery","name":"Set recovery targets","level":"Recommended","rule":"Decide how much data you can lose and how long you can be down, for each service, and design backups to meet both.","check":"Each service has written recovery targets. The backup interval is no longer than the recovery point, and the last timed restore was faster than the recovery time.","url":"https://agentwranglers.se/playbook/operations/disaster-recovery/recovery-targets","related":["/playbook/data/backups-and-recovery/point-in-time-recovery","/playbook/data/backups-and-recovery/failover","/playbook/stack/operations-tools/postgres-backups"],"sources":[{"label":"NIST SP 800-34 Rev. 1 Contingency Planning Guide","url":"https://csrc.nist.gov/pubs/sp/800/34/r1/upd1/final"}]},{"chapter":"Disaster recovery","name":"Back up off the machine","level":"Required","risk":"data","rule":"Copy backups automatically to storage in another place, which an attacker on the server cannot delete.","check":"The newest backup in the off-site store is younger than the recovery point objective, and the server's own keys cannot delete it.","url":"https://agentwranglers.se/playbook/operations/disaster-recovery/off-site-backups","related":["/playbook/operations/disaster-recovery/backup-retention","/playbook/data/backups-and-recovery/encrypted-backups","/playbook/data/backups-and-recovery/restore-tests","/playbook/adversarial-testing/data-and-secrets/least-privilege-keys","/playbook/infrastructure/infrastructure-as-code/protect-stateful-resources","/playbook/stack/operations-tools/postgres-backups"],"sources":[{"label":"NIST SP 800-34 Rev. 1 Contingency Planning Guide","url":"https://csrc.nist.gov/pubs/sp/800/34/r1/upd1/final"}]},{"chapter":"Disaster recovery","name":"Keep backups for a set time","level":"Recommended","rule":"Keep each backup for a fixed period, such as 35 days of daily copies, and let the storage delete it when the period ends.","check":"The oldest backup in the off-site store is no older than the period plus one day. The retention schedule and the storage's expiry rule name the same period.","url":"https://agentwranglers.se/playbook/operations/disaster-recovery/backup-retention","related":["/playbook/website/privacy/data-requests","/playbook/data/personal-data/retention-jobs","/playbook/data/personal-data/retired-data-stores","/playbook/operations/disaster-recovery/off-site-backups","/playbook/compliance/records-and-data-map/retention-schedule"],"sources":[]},{"chapter":"Disaster recovery","name":"Write the restore runbook","level":"Recommended","rule":"Write the exact steps to rebuild the service from nothing, including where the backups and keys are.","check":"A person who did not write the runbook restores staging from it, without asking anyone for help.","url":"https://agentwranglers.se/playbook/operations/disaster-recovery/restore-runbook","related":["/playbook/engineering/documentation/runbooks-in-repo","/playbook/engineering/documentation/doc-ownership","/playbook/infrastructure/servers/rebuild-from-scratch"],"sources":[]},{"chapter":"Disaster recovery","name":"Practise restoring","level":"Required","risk":"data","rule":"Rebuild the whole service from backups on a fresh machine on a schedule, and time it.","check":"The drill log shows a restore within the last three months, its duration, and that the data matched.","url":"https://agentwranglers.se/playbook/operations/disaster-recovery/restore-drills","related":["/playbook/data/backups-and-recovery/restore-tests","/playbook/data/backups-and-recovery/failover","/playbook/infrastructure/servers/rebuild-from-scratch"],"sources":[{"label":"NIST SP 800-34 Rev. 1 Contingency Planning Guide","url":"https://csrc.nist.gov/pubs/sp/800/34/r1/upd1/final"}]},{"chapter":"Disaster recovery","name":"Plan for losing a provider","level":"Recommended","rule":"Know what you do if your host, region, DNS or email provider fails for a day, and keep what you need to switch.","check":"The plan names a second place to run, and the last game day is less than a year old.","url":"https://agentwranglers.se/playbook/operations/disaster-recovery/provider-failure","related":["/playbook/ai-features/latency-cost-and-fallbacks/provider-fallback","/playbook/compliance/processors-and-transfers/vendor-tiers"],"sources":[]},{"chapter":"Disaster recovery","name":"Keep working when the AI provider fails","level":"Required","risk":"outage","applies":["ai-features"],"rule":"Decide what an AI feature does when the model provider is slow, down or out of quota. Show a clear message, never a hang or a broken page.","check":"Block the provider's address on staging. The feature answers within its timeout with the fallback or the message, and the rest of the site works.","url":"https://agentwranglers.se/playbook/operations/disaster-recovery/model-provider-failure","related":["/playbook/ai-features/latency-cost-and-fallbacks/model-timeouts","/playbook/ai-features/latency-cost-and-fallbacks/provider-fallback"],"sources":[]},{"chapter":"Containers","name":"Pin each base image by digest","level":"Recommended","rule":"Name every base image by version and digest, never by a moving tag such as `latest`. Choose the smallest image that runs the app.","check":"Every `FROM` line, or the `ARG` it reads, holds `@sha256:` and 64 hex characters, as the grep below confirms in CI. `docker image inspect` shows the same image ID in CI and production.","url":"https://agentwranglers.se/playbook/operations/containers/pinned-base-images","related":["/playbook/engineering/build-test-and-deploy/container-build-gated-by-tests","/playbook/engineering/running-in-production/reproducible-builds","/playbook/operations/containers/image-scanning","/playbook/stack/delivery-and-infrastructure/docker-and-compose"],"sources":[]},{"chapter":"Containers","name":"Keep build tools out of the runtime image","level":"Recommended","rule":"Build in one stage and copy only the finished app into a clean final stage. Compilers, test tools and development packages never reach the image you ship.","check":"`docker run --rm --entrypoint sh shop -c 'command -v npm pip apk apt-get gcc'` prints nothing, or the image has no shell at all. A scan of the final image lists no test or build packages.","url":"https://agentwranglers.se/playbook/operations/containers/multi-stage-builds","related":["/playbook/engineering/build-test-and-deploy/container-build-gated-by-tests","/playbook/infrastructure/servers/minimal-services"],"sources":[]},{"chapter":"Containers","name":"Run containers as a user without root","level":"Required","risk":"takeover","rule":"Switch to a user without root with `USER` before the app starts. Give that user write access only to the folders the app writes.","check":"`docker run --rm shop id -u` prints a number other than 0.","url":"https://agentwranglers.se/playbook/operations/containers/non-root-containers","related":["/playbook/operations/running-the-service/service-sandbox","/playbook/operations/containers/no-privileged-containers"],"sources":[{"label":"NIST SP 800-190 Application Container Security Guide","url":"https://csrc.nist.gov/pubs/sp/800/190/final"},{"label":"OWASP Docker Security Cheat Sheet","url":"https://cheatsheetseries.owasp.org/cheatsheets/Docker_Security_Cheat_Sheet.html"}]},{"chapter":"Containers","name":"Never give a container control of the host","level":"Required","risk":"takeover","rule":"Never run an app container with `--privileged`, the host's network or process namespace, or the container engine's socket mounted inside.","check":"For every running container, `docker inspect --format '{{.HostConfig.Privileged}} {{.HostConfig.NetworkMode}} {{.HostConfig.PidMode}} {{.HostConfig.SecurityOpt}} {{json .Mounts}}'` shows `false`, no `host`, no `seccomp=unconfined` and no `docker.sock`.","url":"https://agentwranglers.se/playbook/operations/containers/no-privileged-containers","related":["/playbook/operations/containers/non-root-containers"],"sources":[{"label":"NIST SP 800-190 Application Container Security Guide","url":"https://csrc.nist.gov/pubs/sp/800/190/final"},{"label":"OWASP Docker Security Cheat Sheet","url":"https://cheatsheetseries.owasp.org/cheatsheets/Docker_Security_Cheat_Sheet.html"},{"label":"Docker seccomp security profiles","url":"https://docs.docker.com/engine/security/seccomp/"}]},{"chapter":"Containers","name":"Send only the files the build needs","level":"Required","risk":"data","rule":"Keep a `.dockerignore` that leaves out dependencies, build output, version control, local data and every secret file.","check":"Build with a test file named `.env.leak-test` in the folder. Then `docker run --rm --entrypoint sh shop -c 'find / -name \".env*\" -not -path \"/proc/*\"'` prints nothing.","url":"https://agentwranglers.se/playbook/operations/containers/dockerignore","related":["/playbook/adversarial-testing/data-and-secrets/secrets-out-of-code","/playbook/adversarial-testing/data-and-secrets/no-hidden-files"],"sources":[]},{"chapter":"Containers","name":"Pass build secrets as mounts instead of layers","level":"Required","risk":"takeover","rule":"Give the build a token or key only through a secret mount. Never through `ARG`, `ENV`, or a copied file.","check":"`docker history --no-trunc shop` and a secret scanner run over the saved image find no token.","url":"https://agentwranglers.se/playbook/operations/containers/build-secrets","related":["/playbook/engineering/running-in-production/secrets-store"],"sources":[]},{"chapter":"Containers","name":"Let the main process receive stop signals","level":"Recommended","rule":"Start the app in exec form, so it receives the stop signal itself. Add a small init process that forwards signals and reaps child processes.","check":"`docker stop` on a running container finishes before its kill timeout, which is 10 seconds by default. `docker inspect --format '{{.State.ExitCode}}'` then shows 0, not 137, and the app logs its shutdown message.","url":"https://agentwranglers.se/playbook/operations/containers/container-signals","related":["/playbook/operations/running-the-service/graceful-shutdown"],"sources":[]},{"chapter":"Containers","name":"Make the container file system read-only","level":"Recommended","rule":"Run containers with a read-only root file system. Mount a volume for data and a `tmpfs` for scratch files.","check":"`docker exec web touch /app/write-test` fails with a read-only error, and the app still serves pages and writes its data.","url":"https://agentwranglers.se/playbook/operations/containers/read-only-containers","related":["/playbook/operations/running-the-service/service-sandbox"],"sources":[{"label":"OWASP Docker Security Cheat Sheet","url":"https://cheatsheetseries.owasp.org/cheatsheets/Docker_Security_Cheat_Sheet.html"}]},{"chapter":"Containers","name":"Declare a health check in the image","level":"Recommended","rule":"Give every service image a `HEALTHCHECK` that calls the app's own [health address](https://agentwranglers.se/playbook/engineering/build-test-and-deploy/health-check-and-rollback).","check":"Once the start period has passed, `docker inspect --format '{{.State.Health.Status}}' web` prints `healthy`. With a readiness check, stopping the database turns it `unhealthy`. With a liveness check, it stays `healthy`.","url":"https://agentwranglers.se/playbook/operations/containers/container-health","related":["/playbook/engineering/build-test-and-deploy/health-check-and-rollback","/playbook/operations/running-the-service/liveness-and-readiness"],"sources":[]},{"chapter":"Containers","name":"Scan every image before it ships","level":"Recommended","rule":"Scan each built image for known holes in CI, and fail the build on high or critical ones that have a fix.","check":"Build an image from an old base with a known fixed hole on a branch. The scan step fails and names it.","url":"https://agentwranglers.se/playbook/operations/containers/image-scanning","related":["/playbook/operations/containers/pinned-base-images","/playbook/adversarial-testing/supply-chain-and-setup/sbom-and-provenance","/playbook/stack/delivery-and-infrastructure/trivy"],"sources":[{"label":"NIST SP 800-190 Application Container Security Guide","url":"https://csrc.nist.gov/pubs/sp/800/190/final"}]},{"chapter":"Containers","name":"Run one process per container","level":"Recommended","rule":"Give each container one main process, and read every setting from the environment.","check":"`docker top web` lists one main process plus the init process. The same image starts in staging and production with only different variables.","url":"https://agentwranglers.se/playbook/operations/containers/one-process-per-container","related":["/playbook/engineering/running-in-production/settings-outside-the-code","/playbook/operations/running-the-service/resource-limits"],"sources":[]},{"chapter":"Containers","name":"Run the same images locally with Compose","level":"Recommended","rule":"Describe the local setup in one Compose file that uses the production images and versions, so the app runs the same way on every laptop.","check":"On a clean machine, `docker compose up --wait` exits 0 once every service is healthy, and the database reports the same major version as production.","url":"https://agentwranglers.se/playbook/operations/containers/compose-for-local","related":["/playbook/engineering/documentation/readme-quick-start","/playbook/stack/delivery-and-infrastructure/docker-and-compose"],"sources":[]},{"chapter":"Containers","name":"Publish only the ports visitors use","level":"Required","risk":"data","rule":"Publish a container port on all host addresses only when visitors must reach it. Bind every other published port to 127.0.0.1.","check":"`docker ps --format '{{.Names}} {{.Ports}}'` shows `0.0.0.0` or `[::]` only on the proxy's ports 80 and 443. A port scan from outside the host finds no other open port.","url":"https://agentwranglers.se/playbook/operations/containers/published-ports","related":["/playbook/infrastructure/networking-and-dns/deny-by-default-firewall"],"sources":[{"label":"Docker port publishing and mapping","url":"https://docs.docker.com/engine/network/port-publishing/"},{"label":"Docker packet filtering and firewalls","url":"https://docs.docker.com/engine/network/packet-filtering-firewalls/"}]},{"chapter":"Containers","name":"Put each container only on the networks it needs","level":"Recommended","rule":"Split containers into networks by who must talk to whom. Put the database on an internal network that only the app joins.","check":"A connection test from the proxy container to the database port fails, and the same test from the app container succeeds. `docker network inspect` shows `\"Internal\": true` for the back network.","url":"https://agentwranglers.se/playbook/operations/containers/container-networks","related":["/playbook/infrastructure/networking-and-dns/egress-control"],"sources":[{"label":"Compose file networks reference","url":"https://docs.docker.com/reference/compose-file/networks/"},{"label":"Docker bridge network driver","url":"https://docs.docker.com/engine/network/drivers/bridge/"}]},{"chapter":"Containers","name":"Cap the size of container logs","level":"Required","risk":"outage","rule":"Give every container's logs a size limit and a file count, so one chatty container cannot fill the disk.","check":"`docker info --format '{{.LoggingDriver}}'` prints `local`, or every running container's `docker inspect --format '{{.HostConfig.LogConfig}}'` shows a `max-size`.","url":"https://agentwranglers.se/playbook/operations/containers/container-log-rotation","related":["/playbook/operations/capacity-and-cost/exhaustion-alerts"],"sources":[{"label":"Docker logging driver configuration","url":"https://docs.docker.com/engine/logging/configure/"},{"label":"Docker local logging driver","url":"https://docs.docker.com/engine/logging/drivers/local/"}]}]}