Zencoder with a custom OpenAI-compatible endpoint
Zencoder is an AI coding agent that ships with its own model catalog and a credit system. Bringing your own endpoint is possible, but there are two routes that do different things, and the documentation describes them in different places Which is why people end up on the wrong one and conclude the feature is missing.
The routes:
| Route | Where | What it actually accepts |
|---|---|---|
| User API key | Zencoder menu → ⋯ → User API key | Anthropic and OpenAI only |
| Custom model provider | settings.json | anything speaking the OpenAI-compatible chat surface |
If your endpoint is not literally Anthropic or OpenAI, the UI key field is the wrong route. It will not accept a third-party base URL at all, and nothing on that screen says so.
https://aicomp.ai/v1).
Create one free →
The settings.json provider map
Custom models are declared as providers in a JSON file that works at two scopes:
- Machine-wide:
~/.zencoder/settings.json - Per project:
.zencoder/settings.json— overrides the machine-wide file for that repo
A provider block takes four things that matter:
| Key | Value |
|---|---|
mode | direct for a bring-your-own endpoint |
type | openai-compatible (also openai, gemini, anthropic) |
baseUrl | the root URL of the inference API, localhost, a VPC address, or a public gateway |
apiKey | inline credential, or omit it and use an environment variable |
models are then declared under that provider, each with a name (the identifier the provider expects on the wire) and a displayName (what you see in the selector). Those two are separate fields for a reason: the selector label is free text, while name has to match the identifier your endpoint publishes exactly.
The type value is worth pausing on. openai-compatible is the general case and is what a gateway wants. gemini is a distinct integration that does not need a baseUrl (pasting a URL into it accomplishes nothing).
The double underscore in the key variable
This is the field that costs the most time, because getting it wrong produces the wrong symptom.
The documentation gives two names in two places, and they are not the same variable:
- A provider-specific variable:
ZENCODER_<PROVIDER>_API_KEY - A generic fallback:
ZENCODER__API_KEY
Look at the generic one closely. It is ZENCODER_ followed by _API_KEY, with two underscores in a row, the provider slot left empty. Type it with one underscore, the way it reads at a glance, and it is simply a different, unused variable name. Zencoder finds no key.
The symptom is what makes this expensive. A missing key does not always surface as an auth error against your endpoint. Frequently the built-in catalog is still present, the selector still works, and requests quietly go to a Zencoder-managed model instead of yours. You conclude the provider block "did not load" and go looking at the JSON structure, when the JSON is fine and the shell profile has a typo.
If you set the key inline instead, the environment variable is not consulted at all Which is a valid way to sidestep the whole class of problem while testing.
The quotes vs values trap in shell exports
Related and equally quiet: if the key is exported with literal quotes in the value (export ZENCODER__API_KEY="'sk-...'", quotes inside quotes), the variable is set, non-empty, and wrong. Zencoder sends it, the endpoint rejects it, and the error points at the credential rather than at the profile line that produced it.
Check the variable the way the process will see it, not the way you wrote it.
What the endpoint has to support
A custom provider is not just a URL — Zencoder states the requirements for the endpoint itself:
- an OpenAI-compatible
/chat/completionsroute - streaming, via server-sent events, for real-time responses
- function calling, optionally, for tool-use capabilities
The streaming requirement is the one that disqualifies otherwise-working endpoints. A small proxy that buffers the whole response and returns it as one JSON body will pass a curl test and fail in the editor, because the editor's first read is a stream that never arrives. If your gateway is a thin wrapper you wrote yourself, this is the first thing to check.
Hiding the built-in catalog
By default your custom providers sit alongside Zencoder's own models in the selector. To show only yours, add a top-level flag:
{
"useDefaultProviders": false,
"providers": { "...": {} }
}Worth doing while you are validating a new endpoint. With the built-ins hidden, there is no ambiguity about which model answered a request, and the "it silently used a Zencoder model" failure mode becomes impossible rather than merely unlikely.
Per-project files override machine-wide ones, so a repository can pin its own endpoint without changing the setup of everything else on the machine.
Reloading, and why edits appear to do nothing
Editing settings.json does not take effect in a running session. The docs give an explicit reload step for the CLI and UI. Skipping it produces the most misleading symptom of all: a file that is definitely correct on disk, and a selector that definitely does not show it.
Two adjacent gotchas from the same parser:
- Invalid JSON fails silently. A missing comma or bracket makes the whole provider block disappear rather than raising a visible error in the UI.
- A malformed provider does not necessarily break the others, which is good, but it also means a typo can hide for a long time behind a working built-in model.
Cost note
Routing an agent through a gateway changes who bills you, not how the models are priced per token. The relevant comparison is the endpoint's rate against the vendor's published list, and that varies by model and by load shape.
An agent workload here is roughly 300k input and 45k output tokens a day, or 6M input and 0.9M output across a 20-day month. It is input-heavy by construction, because the loop resends context on every turn, which is the opposite of what a completion workload looks like.
| Model | Gateway rate in / out per 1M tokens | One day | 20 days |
|---|---|---|---|
| claude-sonnet-5 | $1.00 / $5.00 | $0.52 | $10.50 |
| gpt-5.6-sol | $2.5 / $15.00 | $1.42 | $28.50 |
| deepseek-v4-pro | $0.66 / $1.98 | $0.29 | $5.74 |
| qwen3.8-max | $1.00 / $3.00 | $0.43 | $8.70 |
One day = 300k input + 45k output on this page's workload. 20 days = 6M input and 0.9M output. Rates checked 2026-10-15 (gateway rates move with upstream promotions), so verify the current number in your dashboard before committing to a budget.
For an agent workload, the number that decides the bill is the output-to-input multiple, not the input rate, agent loops resend context on every turn, so input dominates token counts while output dominates price.
FAQ
FAQ
Does Zencoder support any OpenAI-compatible endpoint, or only OpenAI and Anthropic?
Both, through different routes. The User API key screen in the UI is limited to Anthropic and OpenAI. Anything else goes in settings.json as a provider with type: "openai-compatible".
Why is my custom model not appearing in the selector?
In order: the session was not reloaded after editing the file; the JSON is invalid, so the block was dropped; or the file is at the wrong scope and is being overridden. Reloading first eliminates the most common cause.
My key is set but requests still hit a built-in model. Why?
Check the variable name character by character. The generic fallback has two consecutive underscores. A one-underscore spelling is a different, unused name, and the request falls through to a Zencoder-managed model rather than failing loudly.
Can I point one provider at a local server?
Yes — localhost and private addresses are accepted in baseUrl. A machine-wide file is usually the right scope for a local model, since it is not tied to one repository.
Do I need to keep the built-in models?
No. useDefaultProviders: false hides them, which is the recommended state while validating a new endpoint.
Related
- Tools that take one endpoint (the full matrix), and how each client names the endpoint it talks to
- Base URL not working, the doubled-path failure, which a provider block is not immune to just because it lives in JSON
- OpenAI-compatible endpoints — the streaming chat surface a custom provider has to serve
- Cline 401 errors (the same auth-first failure), but loud instead of silent
- Kilo Code with a custom endpoint — another agent that reaches a gateway through a provider map
- Official vs gateway prices — how the two price columns on this site are measured