Backfill with the same code and record it

How to do it

A one-off script for history gives numbers that differ from the daily load. Give the pipeline a start and an end date, and run it per window. Then each batch is safe to rerun, as Make every load safe to rerun requires. Mark the affected dashboards as changing until the backfill ends. Fixes to the source data itself go through Fix production data with reviewed scripts.

Examples

Python example

from datetime import date, timedelta

def backfill(start: date, end: date, reason: str):
    day = start
    while day <= end:
        run_daily_pipeline(day)
        day += timedelta(days=1)
    backfill_log.insert(start=start, end=end, reason=reason, result="done")

JavaScript example

const DAY_MS = 24 * 60 * 60 * 1000;

export async function backfill(start, end, reason) {
  for (let day = start; day <= end; day = new Date(day.getTime() + DAY_MS)) {
    await runDailyPipeline(day);
  }
  await backfillLog.insert({ start, end, reason, result: 'done' });
}

How to check

A backfill of 3 days on staging gives the same rows as the daily runs gave for those days. Every production backfill has a log entry with a reason, a window and a result.

Sources

• Apache Airflow Backfill

Related rules

• DEN-7 Make every load safe to rerun in Data engineering
• DAT-25 Fix production data with reviewed scripts in Data and databases