Publish a service level for every shared table

How to do it

An alert says that one load is late. A service level says how often the table met its promise, so its readers can trust it or plan around it. Pick one target per kind, such as fresh within its limit in 99 percent of hours and blocking tests passed on 99 percent of days. Take the freshness limit from Alert when data arrives late. Store the targets with the table's owner from Give every table an owner and a description. Targets work as Set a target for each indicator describes for services. When a table misses its target, its owner pauses changes to it and fixes the cause first, as Spend the error budget on purpose does.

Examples

Python example

def freshness_share(hourly_ages_minutes: list[int], limit_minutes: int) -> float:
    on_time = sum(1 for age in hourly_ages_minutes if age <= limit_minutes)
    return on_time / len(hourly_ages_minutes)

def meets_target(hourly_ages_minutes: list[int], limit_minutes: int, target: float = 0.99) -> bool:
    return freshness_share(hourly_ages_minutes, limit_minutes) >= target

JavaScript example

export const freshnessShare = (hourlyAgesMinutes, limitMinutes) =>
  hourlyAgesMinutes.filter(age => age <= limitMinutes).length / hourlyAgesMinutes.length;

export const meetsTarget = (hourlyAgesMinutes, limitMinutes, target = 0.99) =>
  freshnessShare(hourlyAgesMinutes, limitMinutes) >= target;

How to check

Every table with a reader outside its team has freshness and test targets in its metadata. A monthly report lists each such table with its measured share for the last 28 days, and the report from last month exists.

Sources

• dbt freshness

Related rules

• DEN-16 Alert when data arrives late in Data engineering
• DEN-4 Give every table an owner and a description in Data engineering
• OPS-2 Set a target for each indicator in Operations and observability
• OPS-3 Spend the error budget on purpose in Operations and observability