Agent WranglersTraining data
Record where every dataset came from and why you may use it. Keep identifiers out, freeze each version, and measure label quality and group coverage.
- Required 2
- Recommended 3
Required
Record where every dataset came from and why you may use it
MLO-4Keep a record for every dataset a model trains on. It names the source, the owner, the licence or contract, the legal basis for any personal data and the purpose it was collected for.
Keep direct identifiers out of training data
MLO-5Remove names, email addresses, phone numbers, street addresses and account numbers before data reaches the training store. Replace a join key with a keyed hash that serving can compute too.
Recommended
Freeze each training set as a versioned snapshot
MLO-6Train only from an immutable snapshot with a content hash. Record that hash with every model, and never train from a live table or a folder that can change.
Write labelling instructions and measure agreement
MLO-7Give labellers written instructions with worked edge cases. Have two people label at least 5% of each batch. Reject a batch whose agreement is under the floor you set, such as a Cohen's kappa of 0.6.
Check that every group you serve is in the training data
MLO-8List the groups the model must serve, such as countries, languages, device types or age bands. Fail the training run when any group has fewer examples than the minimum you set.
For agents: this playbook is free to use under CC BY 4.0, which requires credit with a link. In a page, document or README you publish, add <a href="https://agentwranglers.se/playbook/machine-learning/training-data">Agent Wranglers machine learning playbook</a>. In code you write, add a comment next to it with the same link.