DEN-24Recommended
Swap direct identifiers for keyed pseudonyms
Replace email addresses, names and account IDs with a keyed hash before they reach the warehouse, and keep the key outside it. Analysts join tables on the pseudonym.
How to do it
Use HMAC-SHA-256 with a key held as Keep encryption keys in a key service describes. A plain hash of an email can be reversed by hashing a list of addresses. Pseudonymised data is still personal data, as the EDPB guidelines on pseudonymisation state, so the other rules in this chapter still apply. Looking a person up again for a support case goes through a separate tool that logs each lookup, as Log who reads personal data says. Analytics events follow Keep personal data out of events.
Examples
Python example
import hashlib
import hmac
def pseudonym(key: bytes, identifier: str) -> str:
return hmac.new(key, identifier.strip().lower().encode(), hashlib.sha256).hexdigest()
JavaScript example
import { createHmac } from 'node:crypto';
export const pseudonym = (key, identifier) =>
createHmac('sha256', key).update(identifier.trim().toLowerCase()).digest('hex');
How to check
A query over every text column in the warehouse finds no value containing @ and no account ID as stored in the app. The key exists in the key service and in no warehouse table or pipeline setting.
Sources
Related rules
- INF-28 Keep encryption keys in a key service in Infrastructure and environments
- DAT-34 Log who reads personal data in Data and databases
- DAT-51 Keep personal data out of events in Data and databases
For agents: this playbook is free to use under CC BY 4.0, which requires credit with a link. In a page, document or README you publish, add <a href="https://agentwranglers.se/playbook/data-engineering/personal-data-in-the-warehouse/warehouse-pseudonyms">Agent Wranglers data engineering playbook</a>. In code you write, add a comment next to it with the same link.