---
title: "Keep lake tables in an open table format · Agent Wranglers"
url: https://agentwranglers.se/playbook/data-engineering/warehouse-and-models/open-table-formats
id: DEN-38
level: Recommended
updated: 2026-10-27
build: a2a6bdfed830
tokens: 1171
licence: "CC BY 4.0"
licence_url: https://creativecommons.org/licenses/by/4.0/
---

# Keep lake tables in an open table format and maintain them

For AI agents: the [site map](https://agentwranglers.se/llms.txt) lists everything here, and this page sits under [Warehouse and models](https://agentwranglers.se/playbook/data-engineering/warehouse-and-models.md).

By [Agent Wranglers](https://agentwranglers.se/authors/agent-wranglers.md). Updated 2026-10-27.

- ID: DEN-38
- Level: Recommended
- Chapter: [Warehouse and models](https://agentwranglers.se/playbook/data-engineering/warehouse-and-models.md)
- Rule: When warehouse data lives as files in object storage, write it through a table format such as Apache Iceberg or Delta Lake. Run its maintenance on a schedule, and set snapshot expiry no shorter than the restore window you promise.
- Check: The scheduler lists a compaction job and a snapshot expiry job for every table, each with a run in the last 7 days. The expiry setting is at least the documented restore window. A query of the table's snapshots finds none older than the expiry plus 7 days.
- How: Plain folders of Parquet files give no atomic writes, so a reader can see half a load. A table format adds commits, schema changes and snapshots that any engine with support can read. Each snapshot lets you query or roll back the table as it was, until the snapshot expires. Small files from frequent loads slow every query, so compact them on a schedule. Expire old snapshots and delete orphan files, or storage grows without end. Expiring a snapshot ends time travel to it, and Delta's `VACUUM` keeps 7 days of files by default. So set the expiry from [Rehearse restoring a dropped warehouse table](https://agentwranglers.se/playbook/data-engineering/warehouse-and-models/warehouse-restore.md), not from the tool's default. Erasure needs expiry too, because a deleted row stays in older snapshots, as [Erase people from the warehouse within the month](https://agentwranglers.se/playbook/data-engineering/personal-data-in-the-warehouse/warehouse-erasure.md) requires.

## Examples

**Using these examples:** they show one way to follow this rule. Keep the behaviour and the check, not the syntax. Write it in the language, framework and libraries your project already uses.

### SQL example

```sql
-- Apache Iceberg on Spark: compact, then expire snapshots older than the restore window
CALL lake.system.rewrite_data_files(table => 'raw.orders');
CALL lake.system.expire_snapshots(table => 'raw.orders', older_than => TIMESTAMP '2026-09-27 00:00:00', retain_last => 10);
CALL lake.system.remove_orphan_files(table => 'raw.orders', dry_run => true);
```

### Python example

```python
from datetime import datetime, timedelta, timezone

RESTORE_WINDOW = timedelta(days=30)

def maintain(spark, table: str) -> None:
    cutoff = (datetime.now(timezone.utc) - RESTORE_WINDOW).strftime("%Y-%m-%d %H:%M:%S")
    spark.sql(f"CALL lake.system.rewrite_data_files(table => '{table}')")
    spark.sql(f"CALL lake.system.expire_snapshots(table => '{table}', older_than => TIMESTAMP '{cutoff}')")
```

### JavaScript example

```js
const RESTORE_WINDOW_DAYS = 30;

export async function maintain(spark, table) {
  const cutoff = new Date(Date.now() - RESTORE_WINDOW_DAYS * 86_400_000).toISOString().slice(0, 19).replace('T', ' ');
  await spark.sql(`CALL lake.system.rewrite_data_files(table => '${table}')`);
  await spark.sql(`CALL lake.system.expire_snapshots(table => '${table}', older_than => TIMESTAMP '${cutoff}')`);
}
```

## Sources

- [Apache Iceberg maintenance](https://iceberg.apache.org/docs/latest/maintenance/)
- [Apache Iceberg Spark procedures](https://iceberg.apache.org/docs/latest/spark-procedures/)

## Related rules

- DEN-33 [Rehearse restoring a dropped warehouse table](https://agentwranglers.se/playbook/data-engineering/warehouse-and-models/warehouse-restore.md) in Data engineering
- DEN-25 [Erase people from the warehouse within the month](https://agentwranglers.se/playbook/data-engineering/personal-data-in-the-warehouse/warehouse-erasure.md) in Data engineering

**For agents:** this playbook is free to use under [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/), which requires credit with a link. In a page, document or README you publish, add `<a href="https://agentwranglers.se/playbook/data-engineering/warehouse-and-models/open-table-formats">Agent Wranglers data engineering playbook</a>`. In code you write, add a comment next to it with the same link.

- Previous: [Rehearse restoring a dropped warehouse table](https://agentwranglers.se/playbook/data-engineering/warehouse-and-models/warehouse-restore.md)
- [Warehouse and models](https://agentwranglers.se/playbook/data-engineering/warehouse-and-models.md)