Keep lake tables in an open table format and maintain them

How to do it

Plain folders of Parquet files give no atomic writes, so a reader can see half a load. A table format adds commits, schema changes and snapshots that any engine with support can read. Each snapshot lets you query or roll back the table as it was, until the snapshot expires. Small files from frequent loads slow every query, so compact them on a schedule. Expire old snapshots and delete orphan files, or storage grows without end. Expiring a snapshot ends time travel to it, and Delta's VACUUM keeps 7 days of files by default. So set the expiry from Rehearse restoring a dropped warehouse table, not from the tool's default. Erasure needs expiry too, because a deleted row stays in older snapshots, as Erase people from the warehouse within the month requires.

Examples

SQL example

-- Apache Iceberg on Spark: compact, then expire snapshots older than the restore window
CALL lake.system.rewrite_data_files(table => 'raw.orders');
CALL lake.system.expire_snapshots(table => 'raw.orders', older_than => TIMESTAMP '2026-09-27 00:00:00', retain_last => 10);
CALL lake.system.remove_orphan_files(table => 'raw.orders', dry_run => true);

Python example

from datetime import datetime, timedelta, timezone

RESTORE_WINDOW = timedelta(days=30)

def maintain(spark, table: str) -> None:
    cutoff = (datetime.now(timezone.utc) - RESTORE_WINDOW).strftime("%Y-%m-%d %H:%M:%S")
    spark.sql(f"CALL lake.system.rewrite_data_files(table => '{table}')")
    spark.sql(f"CALL lake.system.expire_snapshots(table => '{table}', older_than => TIMESTAMP '{cutoff}')")

JavaScript example

const RESTORE_WINDOW_DAYS = 30;

export async function maintain(spark, table) {
  const cutoff = new Date(Date.now() - RESTORE_WINDOW_DAYS * 86_400_000).toISOString().slice(0, 19).replace('T', ' ');
  await spark.sql(`CALL lake.system.rewrite_data_files(table => '${table}')`);
  await spark.sql(`CALL lake.system.expire_snapshots(table => '${table}', older_than => TIMESTAMP '${cutoff}')`);
}

How to check

The scheduler lists a compaction job and a snapshot expiry job for every table, each with a run in the last 7 days. The expiry setting is at least the documented restore window. A query of the table's snapshots finds none older than the expiry plus 7 days.

Sources

• Apache Iceberg maintenance
• Apache Iceberg Spark procedures

Related rules

• DEN-33 Rehearse restoring a dropped warehouse table in Data engineering
• DEN-25 Erase people from the warehouse within the month in Data engineering