Data engineering playbook

A data engineering playbook for teams that copy app data into a warehouse for reports and dashboards. It starts with the warehouse and its models, kept as reviewed code. Then it covers loading and change data capture, data quality tests, and lineage with safe changes. It ends with personal data in the warehouse and access to BI tools. Events and metrics from the app live in the product analytics chapter of the databases playbook, so this one links to it.

Chapters

1. Warehouse and models

   Keep analytics off the production database. Land raw data unchanged, build every table in reviewed code, give each an owner, and cap query costs.

   • Recommended 6

2. Loading and change data capture

   Load data so a rerun changes nothing, carry deletes as well as updates, read changes from the log, and stop when a source changes shape.

   • Required 2
   • Recommended 5

3. Data quality tests

   Test keys, links and allowed values in every model, publish only tables that pass, alert on late data, and reconcile totals with the source.

   • Recommended 5

4. Lineage and changes

   Know where every column comes from and where it goes. Check what a change breaks before merging, give readers notice, and backfill with the same code.

   • Recommended 4

5. Personal data in the warehouse

   Copy only the personal data an analysis needs, swap identifiers for pseudonyms, expire data on time, and make every erasure reach the warehouse.

   • Required 2
   • Recommended 3

6. Access and BI tools

   Grant warehouse access by role from the directory, mask personal columns, give each tool its own narrow account, and stop data leaving by link.

   • Required 3
   • Recommended 2

All 32 rules on one page

This is a lot to check. An Expert can check it for you.

Related playbooks

• Product analytics

  In the Data and databases playbook. Events and metrics start in the app. Plan them, validate them and keep people out of them before they reach the warehouse.

• Transactions and concurrency

  In the Data and databases playbook. Publish facts from the app through an outbox, so the warehouse never counts an order that was rolled back.

• Records and data map

  In the Compliance and governance playbook. Every warehouse and BI tool is a store of personal data. Map it, record it and give it a retention period.

• Identity and access

  In the Infrastructure and environments playbook. Warehouse and BI accounts follow the same single sign-on and least privilege as the cloud account.

• Alerts and on-call

  In the Operations and observability playbook. A pipeline that stops in silence leaves dashboards showing yesterday. Alert on missed runs and old queues.