LiteLLM

LiteLLM is an open-source AI gateway: a Python SDK and a proxy server that put one OpenAI-shaped interface in front of every provider, with retries, fallbacks, budgets, and spend tracking built in. It's the layer a lot of the ecosystem already sits on — Aider and OpenHands both talk to models through it — so adding Melious to a LiteLLM config usually means every tool behind that gateway reaches European infrastructure at once.

Two ways in, and you'll probably use both. The SDK is a drop-in completion() call for your own Python. The proxy is a long-lived server that speaks /v1/chat/completions and /v1/messages on localhost:4000, so anything that can point at an OpenAI or Anthropic base URL gets Melious without knowing we exist.

Setup

pip install litellm

pip install 'litellm[proxy]'

docker run -p 4000:4000 \
  -v $(pwd)/config.yaml:/app/config.yaml \
  -e MELIOUS_API_KEY=sk-mel-<YOUR_API_KEY> \
  -e LITELLM_MASTER_KEY=sk-<YOUR_PROXY_KEY> \
  ghcr.io/berriai/litellm:main-stable \
  --config /app/config.yaml

export MELIOUS_API_KEY=sk-mel-<YOUR_API_KEY>

import os
from litellm import completion

response = completion(
    model="openai/glm-5.1",                        
    api_base="https://api.melious.ai/v1",          
    api_key=os.environ["MELIOUS_API_KEY"],         
    messages=[{"role": "user", "content": "Name three Hanseatic cities."}],
)
print(response.choices[0].message.content)

model_list:
  - model_name: glm-5.1
    litellm_params:
      model: openai/glm-5.1
      api_base: https://api.melious.ai/v1
      api_key: os.environ/MELIOUS_API_KEY

litellm_settings:
  drop_params: true

litellm --config config.yaml

curl http://localhost:4000/v1/chat/completions \
  -H "Authorization: Bearer sk-<YOUR_PROXY_KEY>" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "glm-5.1",
    "messages": [{"role": "user", "content": "Name three Hanseatic cities."}]
  }'

Model routing

model_name is the alias clients ask for; litellm_params.model is what we actually run. That indirection is the whole point of the proxy — swap the model underneath and no client changes.

One entry per model when you want a curated list:

model_list:
  - model_name: fast
    litellm_params:
      model: openai/qwen3.5-9b
      api_base: https://api.melious.ai/v1
      api_key: os.environ/MELIOUS_API_KEY
  - model_name: coding
    litellm_params:
      model: openai/qwen3-coder-next
      api_base: https://api.melious.ai/v1
      api_key: os.environ/MELIOUS_API_KEY

A wildcard when you'd rather expose the whole catalog and let callers name the model:

model_list:
  - model_name: melious/*
    litellm_params:                                    
      model: openai/*
      api_base: https://api.melious.ai/v1
      api_key: os.environ/MELIOUS_API_KEY

Clients then ask for melious/glm-5.2, melious/kimi-k3, or anything else our GET /v1/models lists, with no config change per model. One wart: the proxy's own /v1/models expands openai/* from LiteLLM's built-in OpenAI catalog, so the list it advertises is full of melious/gpt-… names that don't exist here. Routing is unaffected — a real Melious model ID works whether or not the listing mentions it — but if a client populates its model picker from that endpoint, prefer explicit entries over the wildcard.

Routing flavors survive the trip. Append :eco, :price, :speed, or :balanced to bias which European provider serves the request — LiteLLM passes the model string through untouched, so openai/glm-5.1:eco reaches us as glm-5.1:eco. See Routing for what each flavor weights.

Fallbacks chain models when one errors out:

router_settings:
  fallbacks: [{ "glm-5.1": ["qwen3.5-397b-a17b"] }]
  num_retries: 2

Listing two Melious entries under the same model_name also load-balances between them, which is mostly useful for splitting traffic across models rather than across providers — provider-level failover already happens on our side.

Streaming

Set stream=True and LiteLLM hands you an iterator of OpenAI-shaped chunks:

for chunk in completion(
    model="openai/glm-5.1",
    api_base="https://api.melious.ai/v1",
    api_key=os.environ["MELIOUS_API_KEY"],
    messages=[{"role": "user", "content": "Summarize GDPR in two sentences."}],
    stream=True,
):
    print(chunk.choices[0].delta.content or "", end="", flush=True)

curl -N http://localhost:4000/v1/chat/completions \
  -H "Authorization: Bearer sk-<YOUR_PROXY_KEY>" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "glm-5.1",
    "messages": [{"role": "user", "content": "Summarize GDPR in two sentences."}],
    "stream": true
  }'

Reasoning models stream their thinking as reasoning_content deltas before the answer arrives in content, so a naive loop that only prints content looks idle for the first few seconds. That's the model working, not a stalled stream.

The Anthropic-shape endpoint

Melious implements POST /v1/messages too, and LiteLLM can front it. Add a second entry using the anthropic/ transport against our root URL — no /v1, because the Anthropic path appends its own:

model_list:
  - model_name: claude-sonnet-4
    litellm_params:
      model: anthropic/claude-sonnet-4
      api_base: https://api.melious.ai
      api_key: os.environ/MELIOUS_API_KEY

Claude Code and anything else built on the Anthropic SDK can then use the proxy as its ANTHROPIC_BASE_URL. We map claude-* names onto open-weight models and echo the requested name back, which is what keeps those clients working unmodified — see From Anthropic for the mapping table. To run a specific model instead, name it directly: anthropic/glm-5.1.

Cost tracking

LiteLLM prices requests from its own model map, which doesn't know our rates yet, so declare them per deployment:

model_list:
  - model_name: glm-5.1
    litellm_params:
      model: openai/glm-5.1
      api_base: https://api.melious.ai/v1
      api_key: os.environ/MELIOUS_API_KEY
    model_info:
      input_cost_per_token: 0.0000013
      output_cost_per_token: 0.00000405

Every response then comes back with an x-litellm-response-cost header, and /spend/logs aggregates per key and per model once you've given the proxy a DATABASE_URL. Current per-model rates are on the models hub and in _meta.pricing from GET /v1/models?include_meta=true. Ours are in EUR per million tokens; LiteLLM's fields are per single token, so divide by a million.

Melious' own accounting rides along independently. Every response carries environment_impact and billing_cost objects, and both survive the trip: response.environment_impact on an SDK call, and the same keys in the JSON body through the proxy. The usage dashboard aggregates them if you'd rather not read them per request.

What's different

• Two base URLs, one key. The OpenAI-compatible surface is https://api.melious.ai/v1; the Anthropic one is the root, https://api.melious.ai. Mixing them up is the most common misconfiguration here.
• No explicit cache control. There's no field to mark a prefix cacheable. Transparent prefix caching still happens on our side and comes back as usage.cached_tokens, billed at the cheaper cache-read rate. See Models.
• GET /v1/models on the proxy lists your aliases, not our catalog. It reports what model_list exposes, and under a wildcard it reports LiteLLM's built-in list for the transport you borrowed. Query us directly to see everything we run.
• Tool calling, JSON mode, and vision work through the standard OpenAI fields on models that support them. Filter GET /v1/models?include_meta=true by _meta.capabilities before assuming.

When it breaks

• NotFoundError: Model not found: <id> — the ID after the transport prefix isn't one of ours. GET /v1/models for the canonical list; remember the prefix is openai/, and the part after it is the Melious model ID.
• NotFoundError: The requested resource was not found — the model is fine, api_base is wrong for the surface. Chat completions needs the /v1 suffix; the Anthropic path must not have it.
• AuthenticationError from the proxy, not from us — you sent your Melious key where the proxy wanted its own. Clients authenticate with LITELLM_MASTER_KEY or a generated virtual key; only the config holds MELIOUS_API_KEY.
• Empty content with finish_reason: "stop" — a reasoning model spent the whole max_tokens budget on its thinking pass before emitting an answer. It bites hardest with response_format, where you get {} or an empty string back. Raise max_tokens or pick a non-reasoning model; _meta.reasoning_type on GET /v1/models?include_meta=true tells you which is which.
• Rate limit errors under a fan-out workload — LiteLLM will happily run more concurrency than your plan's per-minute token cap allows. Set rpm/tpm on the deployment, or see Rate limits for which plan lifts which limit.

Errors and retry patterns: Errors.