Machine learning playbook

A machine learning playbook for teams that train their own models. It covers classifiers, ranking, recommendations, forecasting and fine-tuned models that score or sort things inside a product. It works with any framework, such as scikit-learn, PyTorch or XGBoost, and any tracking tool, such as MLflow.

A trained model is code that nobody wrote by hand. Its behaviour comes from the data, so a mistake in the data becomes a mistake in every answer. These rules make each model traceable to its data, tested against a simple baseline, and watched once it runs.

The chapters follow the life of a model. First come the decision the model serves and the baseline it must beat. Then come training data, splits that keep the test honest, and evaluation. Next are training runs and the model registry, serving and release, and monitoring with retraining. The last chapter covers people affected by the model, from fairness per group to explaining a decision.

Start with Beat a simple baseline before you ship a model. Then read Splits and leakage before you trust any score.

Product features built on language models, such as chat, summaries and retrieval, live in the AI features playbook. That playbook also holds the rules on fine-tuning a language model, such as when to fine-tune. The legal duties of the AI Act live in the compliance playbook.

Chapters

1. Problem and baseline

   Write down the decision a model serves and what each error costs. Then build a simple baseline, and ship a model only when it beats that baseline.

   • Recommended 3

2. Training data

   Record where every dataset came from and why you may use it. Keep identifiers out, freeze each version, and measure label quality and group coverage.

   • Required 2
   • Recommended 3

3. Splits and leakage

   Split data the way the model meets it in production, by time and by person. Fit every step on training data only, and keep the test set untouched.

   • Recommended 5
   • Avoid 1

4. Evaluation

   Judge a model with metrics that fit the decision, with their uncertainty and with behaviour tests. Then let an automatic gate decide on each release.

   • Recommended 4
   • Avoid 1

5. Training runs and the registry

   Record the data, code, seed and environment of every training run. Keep each model in a registry with versions, and describe it in a model card.

   • Recommended 3

6. Serving and release

   Compute features once for training and serving, check every input, and log what the model saw. Release in shadow and canary steps, with a fallback ready.

   • Required 1
   • Recommended 5

7. Monitoring and retraining

   Alert when inputs or predictions drift, measure live quality once outcomes arrive, retrain on written triggers, and keep feedback loops out of the data.

   • Recommended 4
   • Avoid 1

8. People affected by the model

   Measure quality for each group of people, keep proxies for protected traits out, and give every person a decision touches an explanation and a record.

   • Required 4
   • Avoid 1

All 38 rules on one page

This is a lot to check. An Expert can check it for you.

Related playbooks

• Evals and quality

  In the AI features playbook. Fine-tuned language models get the same treatment there, with an eval set, a stored training file and a comparison before each switch.

• AI Act

  In the Compliance and governance playbook. Check whether a model that scores people is high-risk under the AI Act, and which duties and dates apply to you.

• Personal data

  In the Data and databases playbook. Keep training extracts as small as the app tables, with masked copies and automatic deletion of old rows.

• Data quality tests

  In the Data engineering playbook. The warehouse tables that feed training need freshness, volume and schema tests before a model learns from them.

• Alerts and on-call

  In the Operations and observability playbook. Drift and quality alerts follow the same rules as every alert, with one owner, one action and a page only when it cannot wait.