---
title: "Load only what changed since the last run · Agent Wranglers"
url: https://agentwranglers.se/playbook/data-engineering/loading-and-change-data-capture/incremental-loads
id: DEN-8
level: Recommended
updated: 2026-10-22
tokens: 963
licence: "CC BY 4.0"
licence_url: https://creativecommons.org/licenses/by/4.0/
---

# Load only what changed since the last run

For AI agents: the [site map](https://agentwranglers.se/llms.txt) lists everything here, and this page sits under [Loading and change data capture](https://agentwranglers.se/playbook/data-engineering/loading-and-change-data-capture.md).

By [Agent Wranglers](https://agentwranglers.se/authors/agent-wranglers.md). Updated 2026-10-22.

- ID: DEN-8
- Level: Recommended
- Chapter: [Loading and change data capture](https://agentwranglers.se/playbook/data-engineering/loading-and-change-data-capture.md)
- Rule: Load only rows changed since the last successful run, by a change column or a log position. Save that position after the load commits, and re-read an overlap of at least 15 minutes.
- Check: Update a source row while a load runs on staging. The next run copies the change. Stop a load before it commits. The saved position stays where it was.
- How: Copying every table in full each night gets slower and costlier as the data grows. Use an `updated_at` column the database sets on every write, or the log position from change data capture. A row committed late can carry an earlier `updated_at`, so re-read an overlap and let [Make every load safe to rerun](https://agentwranglers.se/playbook/data-engineering/loading-and-change-data-capture/idempotent-loads.md) drop the repeats. Store every time in UTC. Deleted rows never show up in a change column, which [Carry deletes into the warehouse](https://agentwranglers.se/playbook/data-engineering/loading-and-change-data-capture/propagate-deletes.md) handles.

## Examples

**Using these examples:** they show one way to follow this rule. Keep the behaviour and the check, not the syntax. Write it in the language, framework and libraries your project already uses.

### Python example

```python
from datetime import timedelta

OVERLAP = timedelta(minutes=15)

def load_changed_orders(source, warehouse):
    since = warehouse.cursor("orders") - OVERLAP
    rows = source.query("SELECT * FROM orders WHERE updated_at >= %s ORDER BY updated_at", (since,))
    if rows:
        load_orders(warehouse, rows)
        warehouse.save_cursor("orders", max(row["updated_at"] for row in rows))
```

### JavaScript example

```js
const OVERLAP_MS = 15 * 60 * 1000;

export async function loadChangedOrders(source, warehouse) {
  const since = new Date((await warehouse.cursor('orders')).getTime() - OVERLAP_MS);
  const rows = await source.query('SELECT * FROM orders WHERE updated_at >= $1 ORDER BY updated_at', [since]);
  if (rows.length === 0) return;
  await loadOrders(warehouse, rows);
  await warehouse.saveCursor('orders', rows.at(-1).updated_at);
}
```

## Related rules

- DEN-7 [Make every load safe to rerun](https://agentwranglers.se/playbook/data-engineering/loading-and-change-data-capture/idempotent-loads.md) in Data engineering
- DEN-9 [Carry deletes into the warehouse](https://agentwranglers.se/playbook/data-engineering/loading-and-change-data-capture/propagate-deletes.md) in Data engineering

**For agents:** this playbook is free to use under [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/), which requires credit with a link. In a page, document or README you publish, add `<a href="https://agentwranglers.se/playbook/data-engineering/loading-and-change-data-capture/incremental-loads">Agent Wranglers data engineering playbook</a>`. In code you write, add a comment next to it with the same link.

- Previous: [Make every load safe to rerun](https://agentwranglers.se/playbook/data-engineering/loading-and-change-data-capture/idempotent-loads.md)
- Next: [Carry deletes into the warehouse](https://agentwranglers.se/playbook/data-engineering/loading-and-change-data-capture/propagate-deletes.md)
- [Loading and change data capture](https://agentwranglers.se/playbook/data-engineering/loading-and-change-data-capture.md)