Agent WranglersDEN-13Recommended
Run pipeline steps by their dependencies
Start each step when the steps it reads from have finished, not at a fixed clock time. Retry a failed step at most 3 times, then stop the steps after it and alert.
How to do it
A clock schedule breaks when one step runs long, and the next step builds on half-loaded data. Use an orchestrator such as Airflow or Dagster, or the dependency graph of the transformation tool. Space retries out with backoff. Watch for runs that never start, as Alert when a scheduled job stops running says.
How to check
Make one load step fail on staging. No step that reads from it runs, and an alert reaches the owner within 15 minutes. Pause the scheduler past one run, and the heartbeat alert fires.
Sources
Related rules
- OPS-23 Alert when a scheduled job stops running in Operations and observability
- DEN-16 Alert when data arrives late in Data engineering
For agents: this playbook is free to use under CC BY 4.0, which requires credit with a link. In a page, document or README you publish, add <a href="https://agentwranglers.se/playbook/data-engineering/loading-and-change-data-capture/pipeline-dependencies">Agent Wranglers data engineering playbook</a>. In code you write, add a comment next to it with the same link.