Agent WranglersDEN-19Recommended
Record lineage from the code
Generate lineage from the pipeline code and its run logs, from each source column to each dashboard. Never draw it by hand.
How to do it
Lineage answers two questions. What breaks when this column changes, and where has a person's data gone? Transformation tools build the graph from the references between models. Emit run events in an open format such as OpenLineage, so one catalog sees jobs from every tool. Add BI dashboards as the last step in the graph. Column-level lineage for personal columns feeds Erase people from the warehouse within the month and the data map.
How to check
Pick one personal column in a source. The lineage graph lists every table and dashboard that reads it. A search of the model code finds no reader of that column missing from the list.
Sources
Related rules
- DEN-25 Erase people from the warehouse within the month in Data engineering
- CMP-7 Map every store that holds personal data in Compliance and governance
- DEN-20 Check what a change breaks before merging in Data engineering
For agents: this playbook is free to use under CC BY 4.0, which requires credit with a link. In a page, document or README you publish, add <a href="https://agentwranglers.se/playbook/data-engineering/lineage-and-changes/lineage-from-code">Agent Wranglers data engineering playbook</a>. In code you write, add a comment next to it with the same link.