Copy only the personal columns an analysis needs

How to do it

GDPR Article 5(1)(c) limits personal data to what the purpose needs, and a full copy of the app database breaks that. An allow list keeps a new column out until someone adds it with a purpose, which a block list cannot do. Free text, such as support messages, stays out unless a purpose needs it. The list builds on Store only the personal data you need and feeds the data map.

Examples

Python example

ALLOWED_COLUMNS = {
    "customers": {"customer_id": "join key", "country": "sales per country", "created_at": "cohorts"},
    "orders": {"order_id": "join key", "customer_id": "join key", "total_cents": "revenue", "created_at": "revenue per day"},
}

def extract(source, table: str):
    columns = ", ".join(ALLOWED_COLUMNS[table])
    return source.query(f"SELECT {columns} FROM {table}")

JavaScript example

export const ALLOWED_COLUMNS = {
  customers: { customer_id: 'join key', country: 'sales per country', created_at: 'cohorts' },
  orders: { order_id: 'join key', customer_id: 'join key', total_cents: 'revenue', created_at: 'revenue per day' },
};

export function extract(source, table) {
  return source.query(`SELECT ${Object.keys(ALLOWED_COLUMNS[table]).join(', ')} FROM ${table}`);
}

How to check

A script compares the columns in the raw layer with the allow list and fails on any extra column. Every personal column in the warehouse has a purpose in its metadata.

Sources

• GDPR Article 5 on principles of processing

Related rules

• DEN-4 Give every table an owner and a description in Data engineering
• DAT-30 Store only the personal data you need in Data and databases
• CMP-7 Map every store that holds personal data in Compliance and governance
• DEN-27 List the warehouse and every BI tool in the data map in Data engineering
• DEN-29 Mask personal columns for everyone outside a named role in Data engineering