Use an open-weight model on vLLM or Ollama when it passes your evals

Cost

Open source. vLLM is Apache-2.0 and Ollama is MIT, both free to run on your own servers. Model weights carry their own licences, and the hardware is yours to pay for.

How to do it

An open-weight model runs on your own machines, so prompts and data stay with you and no provider can change or retire it. vLLM is Apache-2.0 licensed. Its first PyPI release was in June 2023, and 0.32.0 came out on 18 October 2026, with about 423,000 weekly downloads. Ollama is MIT licensed. Its first release was in July 2023, and 0.40.7 came out on 18 October 2026. Its Docker image has over 189 million pulls. Both fail the five-year test, like all AI software here. Both answer the same chat API as the openai SDK, so you switch by changing one base URL. Read each model's licence before you ship, because some limit commercial use. Choose something else when no open-weight model you can host reaches your eval threshold. Then call a paid model API, as the next rule says. Pick the size by Choose the smallest model that passes.

Examples

Using these examples: they show one way to follow this rule. Keep the behaviour and the check, not the syntax. Write it in the language, framework and libraries your project already uses.

TypeScript example

import OpenAI from 'openai';

const client = new OpenAI({ baseURL: process.env.MODEL_BASE_URL ?? 'http://localhost:11434/v1', apiKey: 'unused' });

export async function generate(prompt: string): Promise<string> {
  const response = await client.chat.completions.create({
    model: process.env.MODEL_NAME!,
    messages: [{ role: 'user', content: prompt }],
  });
  return response.choices[0].message.content ?? '';
}

Python example

import os

from openai import OpenAI

client = OpenAI(base_url=os.environ.get("MODEL_BASE_URL", "http://localhost:8000/v1"), api_key="unused")

def generate(prompt: str) -> str:
    response = client.chat.completions.create(
        model=os.environ["MODEL_NAME"],
        messages=[{"role": "user", "content": prompt}],
    )
    return response.choices[0].message.content or ""

How to check

A decision record holds the eval pass rate of at least one open-weight model. Every paid model API in the project has a record showing it beat that rate.

Related rules

• Choose the smallest model that passes in AI features