---
title: "Use an open-weight model on vLLM or Ollama · Agent Wranglers"
url: https://agentwranglers.se/playbook/stack/ai-software/open-weight-models
updated: 2026-10-21
licence: "All rights reserved"
---

# Use an open-weight model on vLLM or Ollama when it passes your evals

By [Agent Wranglers](https://agentwranglers.se/authors/agent-wranglers). Updated 2026-10-21.

- Level: Recommended
- Chapter: [AI software](https://agentwranglers.se/playbook/stack/ai-software)
- Rule: Run your eval set on an open-weight model served by an open source server, such as vLLM or Ollama. Use it whenever it reaches your pass threshold.
- Cost: Open source. vLLM is Apache-2.0 and Ollama is MIT, both free to run on your own servers. Model weights carry their own licences, and the hardware is yours to pay for.
- How: An open-weight model runs on your own machines, so prompts and data stay with you and no provider can change or retire it. vLLM is Apache-2.0 licensed. Its first PyPI release was in June 2023, and 0.32.0 came out on 18 October 2026, with about 423,000 weekly downloads. Ollama is MIT licensed. Its first release was in July 2023, and 0.40.7 came out on 18 October 2026. Its Docker image has over 189 million pulls. Both fail the five-year test, like all AI software here. Both answer the same chat API as the `openai` SDK, so you switch by changing one base URL. Read each model's licence before you ship, because some limit commercial use. Choose something else when no open-weight model you can host reaches your eval threshold. Then call a paid model API, as the next rule says. Pick the size by [Choose the smallest model that passes](https://agentwranglers.se/playbook/ai-features/deciding-to-use-a-model/smallest-passing-model).
- Check: A decision record holds the eval pass rate of at least one open-weight model. Every paid model API in the project has a record showing it beat that rate.

## Examples

**Using these examples:** they show one way to follow this rule. Keep the behaviour and the check, not the syntax. Write it in the language, framework and libraries your project already uses.

### TypeScript example

```ts
import OpenAI from 'openai';

const client = new OpenAI({ baseURL: process.env.MODEL_BASE_URL ?? 'http://localhost:11434/v1', apiKey: 'unused' });

export async function generate(prompt: string): Promise<string> {
  const response = await client.chat.completions.create({
    model: process.env.MODEL_NAME!,
    messages: [{ role: 'user', content: prompt }],
  });
  return response.choices[0].message.content ?? '';
}
```

### Python example

```python
import os

from openai import OpenAI

client = OpenAI(base_url=os.environ.get("MODEL_BASE_URL", "http://localhost:8000/v1"), api_key="unused")

def generate(prompt: str) -> str:
    response = client.chat.completions.create(
        model=os.environ["MODEL_NAME"],
        messages=[{"role": "user", "content": prompt}],
    )
    return response.choices[0].message.content or ""
```

## Related rules

- [Choose the smallest model that passes](https://agentwranglers.se/playbook/ai-features/deciding-to-use-a-model/smallest-passing-model) in AI features

**For agents:** this playbook is free to use under [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/), which requires credit with a link. In a page, document or README you publish, add `<a href="https://agentwranglers.se/playbook/stack/ai-software/open-weight-models">Agent Wranglers software choices playbook</a>`. In code you write, add a comment next to it with the same link.

- Next: [Use each provider's official SDK to call models](https://agentwranglers.se/playbook/stack/ai-software/official-model-sdk)
- [AI software](https://agentwranglers.se/playbook/stack/ai-software)