Grade AI features against fixed examples

How to do it

Collect questions people ask and edge cases, each with the answer you expect, including attempts to make the model misbehave. Grade each answer with exact checks where you can, such as a schema or a required fact, and with a scoring model where you cannot. Set a pass rate, such as 95 percent, and fail below it. Run the set whenever the prompt, the model or its version changes. Tools such as promptfoo or DeepEval run the set from the command line. See also grow the eval set from production.

Examples

YAML example

# promptfoo: each test has an input and checks on the answer
tests:
  - vars: { question: "Can I return a sale item?" }
    assert:
      - type: contains
        value: "30 days"
      - type: llm-rubric
        value: "Does not promise a refund for opened software"

How to check

Change the prompt so one known answer goes wrong. The eval run fails in CI and names the failing example.

Sources

• NIST AI 100-1, AI Risk Management Framework

Related rules

• AID-50 Test changes to the agent setup on fixed tasks in AI-assisted development
• AIF-2 Define success before you build in AI features
• AIF-8 Review and test every prompt change in AI features
• AIF-59 Pin the AI model version you depend on in AI features
• AIF-26 Grow the eval set from production in AI features
• AIF-27 Grade with code before a grading model in AI features
• STK-91 Use Promptfoo only when it passes your own tests in Software choices
• SRH-17 Test search results with a fixed query set in Search