Agent WranglersLineage and changes
Know where every column comes from and where it goes. Check what a change breaks before merging, give readers notice, and backfill with the same code.
- Recommended 4
Recommended
Record lineage from the code
DEN-19Generate lineage from the pipeline code and its run logs, from each source column to each dashboard. Never draw it by hand.
Check what a change breaks before merging
DEN-20In CI, build each changed model and every model downstream of it, and run their tests. List the dashboards that read them in the pull request.
Give notice before removing a column people use
DEN-21Treat the tables that dashboards and other teams read as a public interface. Add columns freely, but keep a renamed or removed column for at least 30 days after notice.
Backfill with the same code and record it
DEN-22Rebuild past data by running the normal pipeline for each past window, a day or a week at a time. Log every backfill with its reason, window and result.
For agents: this playbook is free to use under CC BY 4.0, which requires credit with a link. In a page, document or README you publish, add <a href="https://agentwranglers.se/playbook/data-engineering/lineage-and-changes">Agent Wranglers data engineering playbook</a>. In code you write, add a comment next to it with the same link.