---
title: "Use LiteLLM as the gateway when you call · Agent Wranglers"
url: https://agentwranglers.se/playbook/stack/ai-software/llm-gateway
id: STK-110
level: Default
applies: ai-features
updated: 2026-10-27
build: a2a6bdfed830
tokens: 1121
licence: "CC BY 4.0"
licence_url: https://creativecommons.org/licenses/by/4.0/
---

# Use LiteLLM as the gateway when you call several model providers

For AI agents: the [site map](https://agentwranglers.se/llms.txt) lists everything here, and this page sits under [AI software](https://agentwranglers.se/playbook/stack/ai-software.md).

By [Agent Wranglers](https://agentwranglers.se/authors/agent-wranglers.md). Updated 2026-10-27.

- ID: STK-110
- Level: Default
- Chapter: [AI software](https://agentwranglers.se/playbook/stack/ai-software.md)
- Rule: When the app calls models from two or more providers, send every call through one self-hosted LiteLLM proxy. Keep provider keys, spend limits and fallbacks there, and pin it by version and hash.
- Check: A search of the app code finds 0 provider API keys and 0 provider base URLs other than the proxy. The proxy's version is pinned with a hash in the lock file or by image digest.
- Applies when: The project uses AI models.
- Review by: 2027-04-30
- Cost: Open source. MIT licence, except an `enterprise/` folder under its own paid licence that you do not need.
- How: This pick is a recorded exception to the five-year test, dated 27 October 2026, because LiteLLM 1.0.0 came out on 14 November 2023. No older gateway does the job. Version 1.106.4 shipped on 25 October 2026, and PyPI counted about 21.7 million downloads in the week to 27 October 2026. Its GitHub repository had about 62,000 stars. It answers the OpenAI chat API, so the app keeps one client and changes only the base URL. Keys live in the proxy, never in the app, and each team gets a budget, as [Price every model call and cap the spend](https://agentwranglers.se/playbook/ai-features/latency-cost-and-fallbacks/cost-per-call.md) asks. On 24 March 2026, versions 1.82.7 and 1.82.8 on PyPI held malware that stole credentials. Install from a lock file with hashes and wait before taking new releases, as [Wait a few days before taking new releases](https://agentwranglers.se/playbook/application-security/supply-chain/release-cooldown.md) says. Choose something else when you call one provider only. Then its official SDK is enough, as [Use each provider's official SDK to call models](https://agentwranglers.se/playbook/stack/ai-software/official-model-sdk.md) says.

## Examples

**Using these examples:** they show one way to follow this rule. Keep the behaviour and the check, not the syntax. Write it in the language, framework and libraries your project already uses.

### YAML example

```yaml
# litellm.config.yaml: providers and fallbacks live in the proxy
model_list:
  - model_name: support-answer
    litellm_params: { model: anthropic/claude-sonnet-4-5, api_key: os.environ/ANTHROPIC_API_KEY }
  - model_name: support-answer-fallback
    litellm_params: { model: openai/gpt-5-mini, api_key: os.environ/OPENAI_API_KEY }
router_settings:
  fallbacks: [{ support-answer: [support-answer-fallback] }]
```

### Python example

```python
from openai import OpenAI

client = OpenAI(base_url="http://llm-gateway.internal:4000", api_key="team-support-virtual-key")
reply = client.chat.completions.create(model="support-answer", messages=[{"role": "user", "content": "Where is order 1042?"}])
```

## Sources

- [OSV advisory PYSEC-2026-2 for litellm](https://osv.dev/vulnerability/PYSEC-2026-2)

## Related rules

- AIF-42 [Price every model call and cap the spend](https://agentwranglers.se/playbook/ai-features/latency-cost-and-fallbacks/cost-per-call.md) in AI features
- APS-72 [Wait a few days before taking new releases](https://agentwranglers.se/playbook/application-security/supply-chain/release-cooldown.md) in Application security
- STK-86 [Use each provider's official SDK to call models](https://agentwranglers.se/playbook/stack/ai-software/official-model-sdk.md) in Software choices

**For agents:** this playbook is free to use under [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/), which requires credit with a link. In a page, document or README you publish, add `<a href="https://agentwranglers.se/playbook/stack/ai-software/llm-gateway">Agent Wranglers software choices playbook</a>`. In code you write, add a comment next to it with the same link.

- Previous: [Use each provider's official SDK to call models](https://agentwranglers.se/playbook/stack/ai-software/official-model-sdk.md)
- Next: [Trace model calls with OpenTelemetry and read them in Langfuse](https://agentwranglers.se/playbook/stack/ai-software/llm-observability.md)
- [AI software](https://agentwranglers.se/playbook/stack/ai-software.md)