Agent WranglersDEN-16Recommended
Alert when data arrives late
Set a maximum age for the newest row of every source and mart table, and alert the table's owner when that age is passed.
How to do it
A stopped load leaves dashboards that look normal but show yesterday. Store the limit with the table's metadata, such as 2 hours for orders and 26 hours for a daily load. Measure it from the load time column. Show the time of the last refresh on every dashboard. The owner comes from Give every table an owner and a description. Failed and missed runs are covered by Run pipeline steps by their dependencies.
How to check
Stop a load on staging. The freshness alert fires within the table's limit plus 15 minutes. Every dashboard shows the time its data was last refreshed.
Sources
Related rules
- DEN-4 Give every table an owner and a description in Data engineering
- DEN-13 Run pipeline steps by their dependencies in Data engineering
For agents: this playbook is free to use under CC BY 4.0, which requires credit with a link. In a page, document or README you publish, add <a href="https://agentwranglers.se/playbook/data-engineering/data-quality-tests/freshness-limits">Agent Wranglers data engineering playbook</a>. In code you write, add a comment next to it with the same link.