Jan with a Custom OpenAI-Compatible Endpoint
Jan is a local-first desktop assistant: it runs models on your machine, and it also connects to remote providers that speak the OpenAI or Anthropic wire format. Adding a gateway is a four-field dialog, and the order of those fields matters more than it looks.
The API format is chosen before the URL, and picking the wrong one fails every request even when the base URL and key are both correct. The format tells Jan how to build requests and parse responses — it describes the protocol your endpoint speaks, not who made the model. The base URL has to include /v1 for OpenAI-compatible servers, and the API key is mandatory even when nothing on the other end checks it.
https://aicomp.ai/v1).
Create one free →
This page covers the dialog, why a wrong format cannot be debugged from the URL, and what a month of Jan sessions costs once a remote provider is in the mix.
The four fields
Open Settings, go to Model Providers, and click Add Provider. The dialog asks for:
| Field | What goes in it |
|---|---|
| Provider name | Any unique name — this is what shows in the model selector |
| API format | OpenAI-compatible (default) or Anthropic-compatible |
| Base URL | The root of the API, including the version path |
| API key | Required |
For an OpenAI-compatible gateway:
Provider name: My Gateway
API format: OpenAI-compatible
Base URL: https://gw.example.com/v1
API key: sk-your-gateway-keyJan strips trailing slashes automatically, so a URL ending in /v1/ is fine. What it does not do is add the version path for you: entering https://gw.example.com for a server that expects /v1 produces a 404 on every request, because Jan appends the operation path — /chat/completions — onto whatever you supplied and ends up at /chat/completions instead of /v1/chat/completions.
Do not append /chat/completions yourself either. Jan builds that part.
The format field describes the endpoint, not the model
This is the field people get wrong, and it is wrong in a specific direction.
The choice is between OpenAI-compatible and Anthropic-compatible, and it tells Jan which request shape to send and which response shape to parse. It has nothing to do with which company made the model. If your gateway exposes the OpenAI Chat Completions API — most do, including ones that go on to serve Claude models — you pick OpenAI-compatible, even for an Anthropic-created model.
Pick Anthropic-compatible against an OpenAI-speaking endpoint and every request fails, with the base URL and key both correct and both irrelevant. The error does not mention the format, so the natural next move is to re-check the URL, which is already fine.
The key is required even when nothing checks it
Jan requires a non-empty API key for custom providers. For a local server that does not authenticate — vLLM, Ollama, llama.cpp server on your own machine — put any placeholder there. The documented example is sk-no-key. Leave it blank and the provider will not save.
Models: discovered once, then by hand
When you save the provider, Jan requests {base_url}/models and populates the list from what comes back. If your endpoint does not expose that route, or returns an empty list, nothing appears — and the fix is not to change the base URL. Open the provider's Models section, click +, and enter the model id exactly as your server expects it.
There is a second, quieter consequence. Jan does not infer capabilities for custom providers. It cannot tell whether a model supports tools, vision or audio, so those switches stay off until you enable them per model. A model that supports tool calling will simply never call a tool. Enable only what the model actually declares — a UI toggle cannot add an API capability the endpoint does not have.
Key rotation covers three status codes
Jan accepts several keys per provider. Under the provider's API keys section, click Advanced and add them; #1 is primary and the rest are fallbacks. There is a Test keys action that calls {base_url}/models per key and reports the result for each.
Rotation is narrower than it sounds: Jan only tries the next key on HTTP 401, 403 or 429. Every other error — a 404 from a wrong path, a 500 from the upstream, a model id that does not exist — returns immediately without trying another key. Treat the fallback list as resilience against auth and rate limits, not as a general retry.
A 429 across every configured key means you are being rate limited at the account level, and no amount of rotation helps.
What remote sessions cost
Jan's default posture is local, where the bill is electricity. Adding a remote gateway changes that, and the shape of a chat session is predictable: each turn re-sends the conversation so far, so input grows with the length of the thread while output stays roughly one message per turn.
A day of work on this page's workload is 140k input and 30k output tokens — 2.8M input and 0.6M output across 20 working days. That is smaller than a coding agent's footprint, because Jan is a chat interface rather than a loop that reads files it did not ask for. The local models stay available alongside the remote one, so the cheapest configuration is usually a cheap remote model for long threads and a local one for quick questions. See output vs input pricing for the general rule.
| Model | Gateway rate in / out per 1M tokens | One day | 20 days |
|---|---|---|---|
| gpt-5.6-luna | $0.10 / $0.60 | $0.03 | $0.64 |
| gemini-3.7-flash | $0.375 / $1.875 | $0.11 | $2.17 |
| llama-4-maverick | $0.07 / $0.35 | $0.02 | $0.41 |
| gpt-5-mini | $0.125 / $1.00 | $0.05 | $0.95 |
| o3 | $1.00 / $4.00 | $0.26 | $5.20 |
One day = 140k input + 30k output on this page's workload. 20 days = 2.8M input and 0.6M output. Rates checked 2026-10-11 — gateway rates move with upstream promotions, so verify the current number in your dashboard before committing to a budget.
Verifying the route
- Choose
OpenAI-compatibleunless your gateway documents the Anthropic Messages API. The format is about the protocol, not the model's maker. - Include
/v1in the base URL and do not append/chat/completions. Jan adds the operation path. - Put something in the key field even for a local server —
sk-no-keyis the documented placeholder. - Run Test keys. It reports per-key status against
{base_url}/models, which separates a bad key from a wrong URL. - Add models by hand if the list is empty, then enable the capabilities that model actually has.
FAQ
FAQ
Why does every request fail when the URL and key are right?
The API format is wrong. It must match the protocol your endpoint speaks — an OpenAI-speaking gateway needs OpenAI-compatible even when it serves Anthropic-created models.
Do I need /v1 in the base URL?
Yes, for OpenAI-compatible servers. Without it the request lands on /chat/completions instead of /v1/chat/completions and returns 404.
Can I leave the API key blank for a local server?
No. The field is required; use any non-empty placeholder such as sk-no-key when the server does not check.
No models appear in the selector
Jan queried {base_url}/models when you saved and got nothing usable. Add the model id manually in the provider's Models section — do not change the base URL to chase it.
Why does my model never call tools?
Capabilities are not detected for custom providers. Enable tools for that specific model in the provider settings, and only if the model actually supports them.
Will Jan try my backup key on any error?
Only on 401, 403 and 429. Other errors return immediately.
Related
- Cherry Studio with a custom endpoint — another desktop client, with a different URL-appending rule
- LobeChat with a custom endpoint — self-hosted chat UI configured by environment variable
- Open WebUI with a custom endpoint — the other major self-hosted chat UI
- OpenAI-compatible error codes — what a 401 or 404 from your gateway means
- Tools that take one endpoint — the full matrix