Agent WranglersDEN-26Recommended
Expire warehouse data by partition
Set an expiry on every table or partition that holds personal data, taken from the retention schedule. Let the warehouse drop expired partitions by itself.
How to do it
Partition by event date and set the expiry in the table definition, such as 400 days for raw orders. Dropping a whole partition is faster and cheaper than deleting rows one by one. Counts without IDs can stay longer. Each period comes from Keep one retention schedule, and raw analytics events follow Delete raw events on a schedule.
How to check
A script lists every warehouse table with personal columns and fails on any without an expiry that matches the schedule. No partition is older than its expiry plus one day.
Sources
Related rules
- DEN-2 Land raw data unchanged and build tables in layers in Data engineering
- CMP-11 Keep one retention schedule in Compliance and governance
- DAT-57 Delete raw events on a schedule in Data and databases
For agents: this playbook is free to use under CC BY 4.0, which requires credit with a link. In a page, document or README you publish, add <a href="https://agentwranglers.se/playbook/data-engineering/personal-data-in-the-warehouse/warehouse-retention">Agent Wranglers data engineering playbook</a>. In code you write, add a comment next to it with the same link.