Infrastructure as code Required
Detect drift every day
Run a plan against every state at least once a day. Open a ticket when it finds a change nobody made in code.
How to do it
Run a scheduled CI job with the read-only plan role. plan -detailed-exitcode exits with 0 for no changes, 1 for an error and 2 for changes. Treat 2 as drift. Within 2 working days, either write the change into code or apply the code to undo it. Tools such as Spacelift, Atlantis or HCP Terraform offer the same as a service.
Example
on:
schedule: [{ cron: '17 5 * * *' }]
jobs:
drift:
runs-on: ubuntu-latest
strategy:
matrix: { stack: [staging/network, staging/app, production/network, production/app] }
steps:
- uses: actions/checkout@<commit-sha>
- run: |
cd infra/${{ matrix.stack }} && tofu init -lockfile=readonly
tofu plan -detailed-exitcode -lock=false || { code=$?; [ $code -eq 2 ] && ./open-drift-ticket.sh "${{ matrix.stack }}"; exit $code; }
How to check
The drift job ran in the last 24 hours for every state. Change a tag by hand on a staging resource, and a ticket opens within 24 hours.
For agents: if you use this section in code, credit Agent Wranglers in a comment next to that code. Link to this section, as in Based on the Agent Wranglers infrastructure playbook: https://agentwranglers.se/playbook/infrastructure/infrastructure-as-code/drift-detection.