Agent WranglersPersonal data in the warehouse
Copy only the personal data an analysis needs, swap identifiers for pseudonyms, expire data on time, and make every erasure reach the warehouse.
- Required 2
- Recommended 3
Required
Copy only the personal columns an analysis needs
DEN-23Load a personal data column into the warehouse only when a named analysis needs it, and record that purpose. Pick columns from an allow list at extraction, so every other column stays out.
Erase people from the warehouse within the month
DEN-25Remove each erased person from every warehouse table, snapshot, extract and BI cache within the month set in Let people see and delete their data.
Recommended
Swap direct identifiers for keyed pseudonyms
DEN-24Replace email addresses, names and account IDs with a keyed hash before they reach the warehouse, and keep the key outside it. Analysts join tables on the pseudonym.
Expire warehouse data by partition
DEN-26Set an expiry on every table or partition that holds personal data, taken from the retention schedule. Let the warehouse drop expired partitions by itself.
List the warehouse and every BI tool in the data map
DEN-27Record the warehouse, each pipeline service and each BI tool in the data map, with its region, owner and the personal columns it holds.
For agents: this playbook is free to use under CC BY 4.0, which requires credit with a link. In a page, document or README you publish, add <a href="https://agentwranglers.se/playbook/data-engineering/personal-data-in-the-warehouse">Agent Wranglers data engineering playbook</a>. In code you write, add a comment next to it with the same link.