LiteLLM with a custom OpenAI-compatible endpoint

LiteLLM exists to stop you writing vendor-specific code. Point it at one OpenAI-compatible base URL and the same completion() call reaches every model your gateway exposes — then add a fallback so a provider outage becomes a footnote rather than an incident.

What you need

• A gateway API key.
• The base URL: https://aicomp.ai/v1.
• At least one exact model ID from GET /models — LiteLLM does not guess these.

curl https://aicomp.ai/v1/models \
  -H "Authorization: Bearer sk-your-gateway-key"

Step 1 — the smallest working call

Read this one carefully, because the prefix is where most people lose twenty minutes:

import os
from litellm import completion

# LiteLLM needs the provider prefix to know which wire format to speak.
# Everything OpenAI-compatible uses "openai/", including when you point it
# at a gateway rather than at OpenAI.
resp = completion(
    model="openai/deepseek-v4-pro",
    api_base=os.environ["GATEWAY_BASE_URL"],   # https://aicomp.ai/v1
    api_key=os.environ["GATEWAY_API_KEY"],
    messages=[{"role": "user", "content": "Summarise this file."}],
    max_tokens=400,
)
print(resp.choices[0].message.content)

The openai/ prefix selects the request format, not the vendor. With api_base set, it is the format that matters — the request goes where you told it to go.

Step 2 — streaming (what every chat UI needs)

from litellm import completion

resp = completion(
    model="openai/deepseek-v4-pro",
    api_base=os.environ["GATEWAY_BASE_URL"],
    api_key=os.environ["GATEWAY_API_KEY"],
    messages=[{"role": "user", "content": "Write the migration SQL."}],
    stream=True,
)
for chunk in resp:
    delta = chunk.choices[0].delta.content
    if delta:
        print(delta, end="", flush=True)

Step 3 — add a fallback before you need one

Fallbacks are cheap insurance and they compound well with mixed pricing: put an inexpensive model first and promote to a stronger one only when the cheap path fails.

from litellm import completion

# Cheap model first; if it errors, retry on the stronger one.
# The caller never sees which one answered.
resp = completion(
    model="openai/deepseek-v4-flash",
    api_base=os.environ["GATEWAY_BASE_URL"],
    api_key=os.environ["GATEWAY_API_KEY"],
    messages=[{"role": "user", "content": "Explain this stack trace."}],
    fallbacks=[{"deepseek-v4-flash": ["deepseek-v4-pro"]}],
    num_retries=2,
)

Step 4 — move it into the proxy when more than one process needs it

Once several services share the endpoint, repeating api_base per call is how drift starts. The proxy takes the same values once:

model_list:
  - model_name: default
    litellm_params:
      model: openai/deepseek-v4-pro
      api_base: https://aicomp.ai/v1
      api_key: os.environ/GATEWAY_API_KEY
  - model_name: cheap
    litellm_params:
      model: openai/deepseek-v4-flash
      api_base: https://aicomp.ai/v1
      api_key: os.environ/GATEWAY_API_KEY

litellm_settings:
  drop_params: true      # ignore params the upstream model does not accept
  num_retries: 2
  request_timeout: 60

# Routes: callers ask for "default", the proxy decides what that means today.
router_settings:
  fallbacks: [{ default: ["cheap"] }]

Start it with litellm --config config.yaml --port 4000 and callers use one stable URL. Changing which model answers becomes a config edit instead of a deploy.

How this fails in practice

What a day of this actually costs

Fallback strategies look free until you read the invoice. Below is one heavy agent day — 200k input, 60k output — which is the shape of work these tools generate:

FAQ

Related

• All tools with one endpoint — the matrix
• Vercel AI SDK with a custom endpoint
• LangChain with a base URL override
• Attribute spend per request yourself
• Ranking models by token mix, not input price
• Full price index

Get API access