---
title: "Data engineering checklist: all 32 rules ยท Agent Wranglers"
url: https://agentwranglers.se/playbook/data-engineering/checklist
updated: 2026-10-22
tokens: 3090
licence: "CC BY 4.0"
licence_url: https://creativecommons.org/licenses/by/4.0/
---
# Data engineering checklist
For AI agents: the [site map](https://agentwranglers.se/llms.txt) lists everything here, and this page sits under [Data engineering](https://agentwranglers.se/playbook/data-engineering.md).
By [Agent Wranglers](https://agentwranglers.se/authors/agent-wranglers.md). Updated 2026-10-22.
Every rule in the data engineering playbook, by chapter, required rules first. Each links to its own page with how we built it and how to check it. This is a lot to check. [An Expert can check it for you](https://agentwranglers.se/contact.md).
**For agents:** this playbook is free to use under [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/), which requires credit with a link. In a page, document or README you publish, add `Agent Wranglers data engineering playbook`. In code you write, add a comment next to it with the same link.
## Warehouse and models
- [ ] DEN-1 **Recommended** [Keep analytics queries off the production database](https://agentwranglers.se/playbook/data-engineering/warehouse-and-models/separate-warehouse.md): Run reports, dashboards and analysis against a warehouse or a read replica, never against the primary database the app writes to.
- [ ] DEN-2 **Recommended** [Land raw data unchanged and build tables in layers](https://agentwranglers.se/playbook/data-engineering/warehouse-and-models/raw-then-models.md): Load source data into a raw layer exactly as it arrived, then build cleaned and business tables from it in code. Only the loader writes to the raw layer.
- [ ] DEN-3 **Recommended** [Build every warehouse table from reviewed code](https://agentwranglers.se/playbook/data-engineering/warehouse-and-models/transformations-as-code.md): Write every table the warehouse builds as SQL or code in version control, reviewed and deployed by CI. No shared table is built from a saved query in a BI tool or a notebook.
- [ ] DEN-4 **Recommended** [Give every table an owner and a description](https://agentwranglers.se/playbook/data-engineering/warehouse-and-models/table-owners.md): Record an owner team, a one-line description and a refresh schedule for every shared table, in a metadata file next to its code.
- [ ] DEN-5 **Recommended** [Build and test models in separate schemas](https://agentwranglers.se/playbook/data-engineering/warehouse-and-models/development-schemas.md): Build models during development and in CI in a personal or per-branch schema, with masked data. Only the deploy job writes to the production schemas.
- [ ] DEN-6 **Recommended** [Cap what each query and each month can cost](https://agentwranglers.se/playbook/data-engineering/warehouse-and-models/query-cost-limits.md): Set a maximum of bytes scanned or credits per query, and a monthly budget with alerts for the warehouse. Partition large tables by date and require a date filter on them.
## Loading and change data capture
- [ ] DEN-9 **Required** [Carry deletes into the warehouse](https://agentwranglers.se/playbook/data-engineering/loading-and-change-data-capture/propagate-deletes.md): Copy deletes from every source into the warehouse, along with inserts and updates. Within 24 hours of a delete in the source, the row and every row built from it are gone.
- [ ] DEN-11 **Required** [Cap and watch replication slots](https://agentwranglers.se/playbook/data-engineering/loading-and-change-data-capture/replication-slot-limits.md): Cap how much log each replication slot may hold, and alert when a slot falls behind. Drop a slot within a day of retiring the consumer that used it.
- [ ] DEN-7 **Recommended** [Make every load safe to rerun](https://agentwranglers.se/playbook/data-engineering/loading-and-change-data-capture/idempotent-loads.md): Write each load so that running it twice for the same window gives the same rows. Merge on the source key or replace whole partitions, never append without a key.
- [ ] DEN-8 **Recommended** [Load only what changed since the last run](https://agentwranglers.se/playbook/data-engineering/loading-and-change-data-capture/incremental-loads.md): Load only rows changed since the last successful run, by a change column or a log position. Save that position after the load commits, and re-read an overlap of at least 15 minutes.
- [ ] DEN-10 **Recommended** [Read changes from the database log](https://agentwranglers.se/playbook/data-engineering/loading-and-change-data-capture/log-based-cdc.md): For tables that change often or must reach the warehouse within 15 minutes, read changes from the database's write-ahead log or binlog. Do not poll those tables with queries.
- [ ] DEN-12 **Recommended** [Treat each source schema as a contract](https://agentwranglers.se/playbook/data-engineering/loading-and-change-data-capture/source-schema-contracts.md): Write down the columns and types the pipeline reads from each source. Stop the load and alert the owner when a source drops, renames or retypes one of them.
- [ ] DEN-13 **Recommended** [Run pipeline steps by their dependencies](https://agentwranglers.se/playbook/data-engineering/loading-and-change-data-capture/pipeline-dependencies.md): Start each step when the steps it reads from have finished, not at a fixed clock time. Retry a failed step at most 3 times, then stop the steps after it and alert.
## Data quality tests
- [ ] DEN-14 **Recommended** [Test the keys and values of every model](https://agentwranglers.se/playbook/data-engineering/data-quality-tests/model-tests.md): Test that every model's key is unique and never null, that each reference finds its row, and that coded columns hold listed values. Run them on every build.
- [ ] DEN-15 **Recommended** [Publish only tables that pass their tests](https://agentwranglers.se/playbook/data-engineering/data-quality-tests/write-audit-publish.md): Build each table where no dashboard reads it, run its tests, and swap it in only when they pass. When a test fails, dashboards keep the last good version.
- [ ] DEN-16 **Recommended** [Alert when data arrives late](https://agentwranglers.se/playbook/data-engineering/data-quality-tests/freshness-limits.md): Set a maximum age for the newest row of every source and mart table, and alert the table's owner when that age is passed.
- [ ] DEN-17 **Recommended** [Reconcile totals with the source](https://agentwranglers.se/playbook/data-engineering/data-quality-tests/source-reconciliation.md): Every day, compare row counts and money totals per day of data between each source and the warehouse. Alert on any difference past a written tolerance, which is zero for money.
- [ ] DEN-18 **Recommended** [Alert on sudden changes in volume](https://agentwranglers.se/playbook/data-engineering/data-quality-tests/volume-anomalies.md): Count the rows loaded per table per day. Alert when a day differs from the median of that weekday over 4 weeks by more than a written limit, such as 50 percent.
## Lineage and changes
- [ ] DEN-19 **Recommended** [Record lineage from the code](https://agentwranglers.se/playbook/data-engineering/lineage-and-changes/lineage-from-code.md): Generate lineage from the pipeline code and its run logs, from each source column to each dashboard. Never draw it by hand.
- [ ] DEN-20 **Recommended** [Check what a change breaks before merging](https://agentwranglers.se/playbook/data-engineering/lineage-and-changes/impact-check.md): In CI, build each changed model and every model downstream of it, and run their tests. List the dashboards that read them in the pull request.
- [ ] DEN-21 **Recommended** [Give notice before removing a column people use](https://agentwranglers.se/playbook/data-engineering/lineage-and-changes/deprecation-notice.md): Treat the tables that dashboards and other teams read as a public interface. Add columns freely, but keep a renamed or removed column for at least 30 days after notice.
- [ ] DEN-22 **Recommended** [Backfill with the same code and record it](https://agentwranglers.se/playbook/data-engineering/lineage-and-changes/backfills.md): Rebuild past data by running the normal pipeline for each past window, a day or a week at a time. Log every backfill with its reason, window and result.
## Personal data in the warehouse
- [ ] DEN-23 **Required** [Copy only the personal columns an analysis needs](https://agentwranglers.se/playbook/data-engineering/personal-data-in-the-warehouse/warehouse-minimisation.md): Load a personal data column into the warehouse only when a named analysis needs it, and record that purpose. Pick columns from an allow list at extraction, so every other column stays out.
- [ ] DEN-25 **Required** [Erase people from the warehouse within the month](https://agentwranglers.se/playbook/data-engineering/personal-data-in-the-warehouse/warehouse-erasure.md): Remove each erased person from every warehouse table, snapshot, extract and BI cache within the month set in [Let people see and delete their data](https://agentwranglers.se/playbook/website/privacy/data-requests.md).
- [ ] DEN-24 **Recommended** [Swap direct identifiers for keyed pseudonyms](https://agentwranglers.se/playbook/data-engineering/personal-data-in-the-warehouse/warehouse-pseudonyms.md): Replace email addresses, names and account IDs with a keyed hash before they reach the warehouse, and keep the key outside it. Analysts join tables on the pseudonym.
- [ ] DEN-26 **Recommended** [Expire warehouse data by partition](https://agentwranglers.se/playbook/data-engineering/personal-data-in-the-warehouse/warehouse-retention.md): Set an expiry on every table or partition that holds personal data, taken from the retention schedule. Let the warehouse drop expired partitions by itself.
- [ ] DEN-27 **Recommended** [List the warehouse and every BI tool in the data map](https://agentwranglers.se/playbook/data-engineering/personal-data-in-the-warehouse/warehouse-in-data-map.md): Record the warehouse, each pipeline service and each BI tool in the data map, with its region, owner and the personal columns it holds.
## Access and BI tools
- [ ] DEN-29 **Required** [Mask personal columns for everyone outside a named role](https://agentwranglers.se/playbook/data-engineering/access-and-bi-tools/column-masking.md): Tag every personal column in the warehouse. Mask or hide tagged columns from every role except the few named for them.
- [ ] DEN-30 **Required** [Give each pipeline and BI tool its own narrow account](https://agentwranglers.se/playbook/data-engineering/access-and-bi-tools/tool-service-accounts.md): Give each loader, transformation job and BI tool a separate account that reaches only its own schemas. BI tools get read access to the marts and nothing else.
- [ ] DEN-31 **Required** [Filter shared dashboards by customer inside the warehouse](https://agentwranglers.se/playbook/data-engineering/access-and-bi-tools/dashboard-row-security.md): When customers see dashboards built on shared tables, filter rows by customer in the warehouse or in a signed embed token. Never rely on a filter set in the dashboard alone.
- [ ] DEN-28 **Recommended** [Grant warehouse access by role from the directory](https://agentwranglers.se/playbook/data-engineering/access-and-bi-tools/warehouse-roles.md): Give people access to the warehouse and BI tools through roles kept in code and synced from the company directory. Grant nothing to a single person on a single table.
- [ ] DEN-32 **Recommended** [Limit exports and public links from BI tools](https://agentwranglers.se/playbook/data-engineering/access-and-bi-tools/bi-sharing-limits.md): Turn off public dashboard links and scheduled reports to addresses outside the company's domains in every BI tool. Allow raw exports of personal columns only to the roles that may see them.
## Show the badge on your project
When your project passes every Required rule, you can show this badge on your site or in your README. It links to this checklist, so anyone can run the same checks.
```html
```
```markdown
[](https://agentwranglers.se/playbook/data-engineering/checklist.md)
```