AI software

Recommended

• Use an open-weight model on vLLM or Ollama when it passes your evals

  Recommended Open source

  Run your eval set on an open-weight model served by an open source server, such as vLLM or Ollama. Use it whenever it reaches your pass threshold.

• Use each provider's official SDK to call models

  Recommended Paid

  Call each model provider through its own official SDK, inside one module of your own. Add a shared abstraction only when you call two or more providers.

• Use Zod or Pydantic to define structured output

  Recommended Open source

  Define each structured answer once, as a Zod schema in TypeScript or a Pydantic model in Python. Send its JSON Schema to the model and validate the reply with the same definition.

• Use pgvector in PostgreSQL before a separate vector database

  Recommended Open source

  Store embeddings in a pgvector column next to the rows they describe. Add a separate vector database only after a load test shows a limit PostgreSQL cannot meet.

• Use the Model Context Protocol to give agents your tools

  Recommended Open source

  Offer tools and data to outside AI agents as a Model Context Protocol server, built on an official MCP SDK.

• Use your test runner to run evals on a fixed example set

  Recommended Open source

  Run AI evals as ordinary tests in the runner the project already uses, over a fixed set of examples kept in the repository.

• Use the coding agent that passes most of your own task set

  Recommended Open source

  Pick a coding agent by running your own fixed task set on each candidate. Keep its instructions in one AGENTS.md file that any agent can read.

Optional

• Use Promptfoo only when it passes your own tests

  Optional Open source

  Add an open source eval tool, such as Promptfoo, only when it runs your example set in CI and fails the build on a drop.