Data engineering checklist

Every rule in the data engineering playbook, by chapter, required rules first. Each links to its own page with how we built it and how to check it. This is a lot to check. An Expert can check it for you.

Warehouse and models

• Recommended DEN-1 Keep analytics queries off the production database
• Recommended DEN-2 Land raw data unchanged and build tables in layers
• Recommended DEN-3 Build every warehouse table from reviewed code
• Recommended DEN-4 Give every table an owner and a description
• Recommended DEN-5 Build and test models in separate schemas
• Recommended DEN-6 Cap what each query and each month can cost

Loading and change data capture

• Required DEN-9 Carry deletes into the warehouse
• Required DEN-11 Cap and watch replication slots
• Recommended DEN-7 Make every load safe to rerun
• Recommended DEN-8 Load only what changed since the last run
• Recommended DEN-10 Read changes from the database log
• Recommended DEN-12 Treat each source schema as a contract
• Recommended DEN-13 Run pipeline steps by their dependencies

Data quality tests

• Recommended DEN-14 Test the keys and values of every model
• Recommended DEN-15 Publish only tables that pass their tests
• Recommended DEN-16 Alert when data arrives late
• Recommended DEN-17 Reconcile totals with the source
• Recommended DEN-18 Alert on sudden changes in volume

Lineage and changes

• Recommended DEN-19 Record lineage from the code
• Recommended DEN-20 Check what a change breaks before merging
• Recommended DEN-21 Give notice before removing a column people use
• Recommended DEN-22 Backfill with the same code and record it

Personal data in the warehouse

• Required DEN-23 Copy only the personal columns an analysis needs
• Required DEN-25 Erase people from the warehouse within the month
• Recommended DEN-24 Swap direct identifiers for keyed pseudonyms
• Recommended DEN-26 Expire warehouse data by partition
• Recommended DEN-27 List the warehouse and every BI tool in the data map

Access and BI tools

• Required DEN-29 Mask personal columns for everyone outside a named role
• Required DEN-30 Give each pipeline and BI tool its own narrow account
• Required DEN-31 Filter shared dashboards by customer inside the warehouse
• Recommended DEN-28 Grant warehouse access by role from the directory
• Recommended DEN-32 Limit exports and public links from BI tools

Show the badge on your project

When your project passes every Required rule, you can show this badge on your site or in your README. It links to this checklist, so anyone can run the same checks.

<a href="https://agentwranglers.se/playbook/data-engineering/checklist"><img src="https://agentwranglers.se/badge.svg" alt="Checked against the Agent Wranglers data engineering playbook" width="214" height="20"></a>

[![Checked against the Agent Wranglers data engineering playbook](https://agentwranglers.se/badge.svg)](https://agentwranglers.se/playbook/data-engineering/checklist)