Fine-tune only after prompts and retrieval miss the target

How to do it

Fine-tuning teaches a format, a tone or a narrow task. It is a poor way to add facts that change, which retrieval does better. A fine-tune also brings a training pipeline and a model to retrain when its base model retires. It brings data duties too, in Train only on data people agreed to share and Keep each customer's data in its own fine-tuned model. Changing a general-purpose model can make you a provider under the EU AI Act, as Know when you become a model provider explains. Two reasons hold up well. A small fine-tuned model passes where only a large model did, so each call costs less. Or prompts alone cannot hold a strict output format.

How to check

The decision note for each fine-tuned model lists three eval scores. They are the best prompt alone, the best prompt with retrieval and the fine-tuned model. Only the fine-tuned model meets the spec, or it meets it at a lower cost per call.

Sources

• OpenAI API docs, model optimization

Related rules

• AIF-50 Train only on data people agreed to share in AI features
• AIF-58 Keep each customer's data in its own fine-tuned model in AI features
• CMP-45 Know when you become a model provider in Compliance and governance