LiteLLM with a custom OpenAI-compatible endpoint

By the aitokensai editors · Verified 2026-11-09 Sources: LiteLLM docs · OpenAI API docs

LiteLLM exists to stop you writing vendor-specific code. Point it at one OpenAI-compatible base URL and the same completion() call reaches every model your gateway exposes. Then add a fallback so a provider outage becomes a footnote rather than an incident.

What you need

• A gateway API key.
• The base URL: https://aicomp.ai/v1.
• At least one exact model ID from GET /models, LiteLLM does not guess these.

curl https://aicomp.ai/v1/models \
  -H "Authorization: Bearer sk-your-gateway-key"

Step 1 — the smallest working call

Read this one carefully, because the prefix is where most people lose twenty minutes:

import os
from litellm import completion

# LiteLLM needs the provider prefix to know which wire format to speak.
# Everything OpenAI-compatible uses "openai/", including when you point it
# at a gateway rather than at OpenAI.
resp = completion(
    model="openai/deepseek-v4-pro",
    api_base=os.environ["GATEWAY_BASE_URL"],   # https://aicomp.ai/v1
    api_key=os.environ["GATEWAY_API_KEY"],
    messages=[{"role": "user", "content": "Summarise this file."}],
    max_tokens=400,
)
print(resp.choices[0].message.content)

The openai/ prefix selects the request format, not the vendor. With api_base set, it is the format that matters (the request goes where you told it to go).

Step 2 — streaming (what every chat UI needs)

from litellm import completion

resp = completion(
    model="openai/deepseek-v4-pro",
    api_base=os.environ["GATEWAY_BASE_URL"],
    api_key=os.environ["GATEWAY_API_KEY"],
    messages=[{"role": "user", "content": "Write the migration SQL."}],
    stream=True,
)
for chunk in resp:
    delta = chunk.choices[0].delta.content
    if delta:
        print(delta, end="", flush=True)

Step 3 — add a fallback before you need one

Fallbacks are cheap insurance and they compound well with mixed pricing: put an inexpensive model first and promote to a stronger one only when the cheap path fails.

from litellm import completion

# Cheap model first; if it errors, retry on the stronger one.
# The caller never sees which one answered.
resp = completion(
    model="openai/deepseek-v4-flash",
    api_base=os.environ["GATEWAY_BASE_URL"],
    api_key=os.environ["GATEWAY_API_KEY"],
    messages=[{"role": "user", "content": "Explain this stack trace."}],
    fallbacks=[{"deepseek-v4-flash": ["deepseek-v4-pro"]}],
    num_retries=2,
)

Step 4 — move it into the proxy when more than one process needs it

Once several services share the endpoint, repeating api_base per call is how drift starts. The proxy takes the same values once:

model_list:
  - model_name: default
    litellm_params:
      model: openai/deepseek-v4-pro
      api_base: https://aicomp.ai/v1
      api_key: os.environ/GATEWAY_API_KEY
  - model_name: cheap
    litellm_params:
      model: openai/deepseek-v4-flash
      api_base: https://aicomp.ai/v1
      api_key: os.environ/GATEWAY_API_KEY

litellm_settings:
  drop_params: true      # ignore params the upstream model does not accept
  num_retries: 2
  request_timeout: 60

# Routes: callers ask for "default", the proxy decides what that means today.
router_settings:
  fallbacks: [{ default: ["cheap"] }]

Start it with litellm --config config.yaml --port 4000 and callers use one stable URL. Changing which model answers becomes a config edit instead of a deploy.

How this fails in practice

Why changing one URL is enough

An OpenAI-compatible client does not have a provider hardcoded into it. It holds a base URL, and at request time it concatenates that base URL with a resource path — /chat/completions, /models, then sends the same JSON body it would have sent anywhere else, with your key in an Authorization: Bearer header. The base URL is therefore the only thing that decides where the request physically goes.

That is why this change is one line and not a rewrite. Your message array, your tool definitions, your streaming handling and your response parsing are all unchanged, because the wire format is unchanged. What changes is who answers, and what they charge per token. The two values that travel with the new destination are the base URL itself and the API key; the model ID is the third, and it has to be one the new endpoint actually serves, which you can list with a single call to /v1/models rather than guess at.

The practical consequence is that this is reversible. Keep the base URL in configuration rather than inline, and moving between providers is a config change you can undo in a minute — which is also the right way to test it: point a single non-critical call at the new endpoint first, confirm it in the usage log, then move the rest.

What a day of this actually costs

Fallback strategies look free until you read the invoice. Below is one heavy agent day (200k input, 60k output) which is the shape of work these tools generate:

One month at 20 working days, gateway rates checked 2026-11-09. These are the token volumes, not a quote Your own shape may be larger or smaller.

Worth reading both tables together: the agent-day table is one point on a curve, and this one shows that the curve does not rank models the same way everywhere. Where a workload is mostly input, the input rate decides it; where it is mostly output, the output rate does. A fallback chain that sends cheap traffic to one model and hard traffic to another is only saving money if the two models are actually cheaper on the shapes you are sending them.

FAQ

Related

• All tools with one endpoint — the matrix
• Pydantic AI, the other Python library, and its route trap
• Aider (same routing), in the terminal
• Continue — per-role models in the IDE
• Vercel AI SDK with a custom endpoint
• LangChain with a base URL override
• Attribute spend per request yourself
• Ranking models by token mix, not input price
• Full price index

Get API access