Tabby with a Custom OpenAI-Compatible Endpoint

By the aitokensai editors · Verified 2026-10-14 Sources: OpenAI API docs

Tabby is a self-hosted coding assistant: a server you run, plus editor plugins that talk to it. Its selling point is that the model can be local, and the second-most-common reason people run it is that the model can be someone else's. Pointing it at a gateway is a config file edit, and the file has three independent model slots.

Chat and embedding are straightforward. Code completion is the one that needs extra work, because completion is not a chat call It is a fill-in-the-middle call with a template, and a gateway that does not echo that template back produces completions that are visibly wrong rather than absent.

Everything lives in ~/.tabby/config.toml, or /data/config.toml inside the container if you deployed with Docker.

Three slots, configured separately

Nothing is shared. A working chat slot tells you nothing about completion, and the usual symptom of an unfinished setup is a side panel that answers questions while suggestions never appear.

Each slot picks a connector with kind, and the connector decides which HTTP route is called:

# ~/.tabby/config.toml

[model.chat.http]
kind = "openai/chat"
model_name = "claude-sonnet-5"
api_endpoint = "https://gw.example.com/v1"
api_key = "sk-your-gateway-key"

[model.embedding.http]
kind = "openai/embedding"
model_name = "text-embedding-3-small"
api_endpoint = "https://gw.example.com/v1"
api_key = "sk-your-gateway-key"

[model.completion.http]
kind = "openai/completion"
model_name = "deepseek-v4-pro"
api_endpoint = "https://gw.example.com/v1"
api_key = "sk-your-gateway-key"
prompt_template = "{prefix}{suffix}"

openai/chat posts to the chat completions route and openai/completion posts to the legacy completions route. An endpoint that only implements chat completions cannot serve the completion slot through this connector, no matter how the key is set.

The prompt template, and why completion breaks without it

A code completion request is not a conversation. It asks the model to continue text from a cursor position, and Tabby expresses that with a FIM template: {prefix} is the code before the cursor, {suffix} is the code after it.

Different model families were trained on different FIM formats, and the template has to match the one your model expects. For a CodeLlama-lineage model the conventional value is "<PRE> {prefix} <SUF>{suffix} <MID>"; for a Qwen Coder model it is closer to "{prefix}{suffix}" with the model's own control tokens.

Get it wrong and the failure is quiet. The request succeeds, tokens come back, and the suggestion contains fragments of your own prompt, or repeats the line above the cursor, or ignores the code below the cursor entirely. A completion that arrives but is useless is a prompt template problem, not a connectivity problem, and it is the single most common misdiagnosis with this tool.

The version segment is connector-specific

This is the second trap, and it runs the opposite way depending on kind.

• vllm/* connectors expect api_endpoint in the form http://host:port/v1. The version segment is part of the value.
• llama.cpp/* connectors document the opposite: the endpoint should not carry a /v1 suffix.

Because both are described as "the endpoint", copying a connection string from one connector's example into another connector's section produces a route that 404s. For an OpenAI-compatible gateway the openai/* connectors follow the versioned convention, so include /v1 and stop there, not the operation path.

A chat slot needs a chat model

kind = "openai/chat" tells Tabby which route and which request shape to use. It cannot make a completion model behave as a chat model. Pointing the chat slot at a base-model checkpoint returns answers that trail off or loop, and the vendor note attached to the connector says as much: use a chat model.

The same applies in reverse to completion. A chat-tuned model asked to continue code will produce something that reads fine and completes nothing.

Verifying the route

Tabby exposes the state of each connector in its own UI, which is unusual and worth using before you debug from the editor.

1. Open Information → System. The model cards show whether each configured model connected. A card that is not connected is a config problem, not an editor problem.
2. Confirm the connector's URL convention. vllm/* carries /v1; llama.cpp/* must not.
3. Give completion a real prompt template matched to the model family, then check a suggestion against the code surrounding the cursor.
4. Point supported_models at real IDs if you want to switch models from the chat UI without editing the file.
5. Restart the server after any edit. The config file is read at start.
6. Test in the editor last. Every failure above looks the same from inside a plugin.

[model.chat.http]
kind = "openai/chat"
model_name = "claude-sonnet-5"
supported_models = ["claude-sonnet-5", "claude-haiku-4-5", "deepseek-v4-pro"]
api_endpoint = "https://gw.example.com/v1"
api_key = "sk-your-gateway-key"

Where the tokens go

Tabby's three slots have very different shapes, and the cheapest one is not the one people worry about.

Completion fires whenever you stop typing. Each request carries the code around the cursor, and the reply is tens of tokens. That is a high-frequency, large-input, small-output pattern: a hundred or more calls in a working hour, each one maybe a few thousand input tokens.

Chat is a conversation with code context attached, so it is moderate frequency and grows with the thread.

Embedding runs in bursts when a repository is indexed, then goes quiet until the repository changes.

A day on this page's workload is 260k input and 30k output tokens, which is 5.2M input and 0.6M output across 20 working days. Because completion dominates the call count and the input side, the input rate of whichever model fills that slot sets the month.

See output vs input pricing for why the ratio matters more than the headline rate, and cheapest models by workload shape for picking per slot rather than per tool.

One day = 260k input + 30k output on this page's workload. 20 days = 5.2M input and 0.6M output. Rates checked 2026-10-14 (gateway rates move with upstream promotions), so verify the current number in your dashboard before committing to a budget.

FAQ

FAQ

Related

• Void with a custom endpoint (an editor with three model slots of its own), assigned through a UI rather than a file
• Continue with a custom endpoint — a plugin that splits roles across models with per-role settings
• Base URL not working, the trailing slash and doubled path failures, in general form
• Cheapest models by workload shape (pick per slot), not per tool
• Codecompanion with a custom endpoint — another FIM-dependent setup in a different editor
• Tools that take one endpoint — the full matrix

Get API access