Treat each source schema as a contract

How to do it

An app team renames a column, and dashboards quietly show zeros. Compare the incoming schema with the stored one before each load. A new column may load on its own, but a dropped or changed column fails the load. App teams change shared tables with Expand first and contract later, and test the contract as Check the contract between services describes.

Examples

Python example

EXPECTED = {"order_id": "bigint", "status": "text", "total_cents": "bigint", "updated_at": "timestamptz"}

def check_source_schema(actual: dict[str, str]) -> None:
    broken = {column: kind for column, kind in EXPECTED.items() if actual.get(column) != kind}
    if broken:
        raise SchemaChanged(f"orders changed: expected {broken}, got {[actual.get(c) for c in broken]}")

JavaScript example

const EXPECTED = { order_id: 'bigint', status: 'text', total_cents: 'bigint', updated_at: 'timestamptz' };

export function checkSourceSchema(actual) {
  const broken = Object.entries(EXPECTED).filter(([column, kind]) => actual[column] !== kind);
  if (broken.length) throw new SchemaChanged(`orders changed: ${broken.map(([column, kind]) => `${column} expected ${kind}, got ${actual[column]}`).join('; ')}`);
}

How to check

Rename a column in a staging source. The next load fails with a message that names the column, and no table downstream changes.

Related rules

• DEN-10 Read changes from the database log in Data engineering
• DAT-22 Expand first and contract later in Data and databases
• TST-14 Check the contract between services in Testing
• MLO-24 Check every input against what the model was trained on in Machine learning