Agent WranglersDEN-9Required
Carry deletes into the warehouse
Copy deletes from every source into the warehouse, along with inserts and updates. Within 24 hours of a delete in the source, the row and every row built from it are gone.
How to do it
A load that reads changed rows never sees a deleted one, so an erased customer lives on in the warehouse. Use change data capture, which records deletes, or a soft-delete column that the load reads. Otherwise compare the source's keys with the warehouse's keys every day. Then delete in raw and rebuild or delete in every later layer, which Erase people from the warehouse within the month relies on. Deletes for retention follow Delete old data automatically.
How to check
Delete a test customer in the source. Within 24 hours, no raw, staging or mart table holds its ID.
Sources
Related rules
- DEN-8 Load only what changed since the last run in Data engineering
- DEN-25 Erase people from the warehouse within the month in Data engineering
- DAT-31 Delete old data automatically in Data and databases
For agents: this playbook is free to use under CC BY 4.0, which requires credit with a link. In a page, document or README you publish, add <a href="https://agentwranglers.se/playbook/data-engineering/loading-and-change-data-capture/propagate-deletes">Agent Wranglers data engineering playbook</a>. In code you write, add a comment next to it with the same link.