RAGFlow with a Custom OpenAI-Compatible Endpoint
RAGFlow connects to a model through a provider, and since v0.27.0 a provider is no longer a single row of settings. It is an instance you create, name, and attach models to. That change is what most older walkthroughs get wrong, and it is also the reason the same base URL can behave differently on two installs of the same product.
Pick OpenAI-API-Compatible as the provider type. It exists precisely for endpoints that are not on the shortlist of named vendors, and it is the only one that will accept an arbitrary base URL.
https://aicomp.ai/v1).
Create one free →
Instances, not rows
Go to User settings → Model providers, find OpenAI-API-Compatible under Available models, and create an instance.
| Field | Value | Notes |
|---|---|---|
| Instance name | anything but default | default is taken by the built-in entry |
| Base URL | https://gw.example.com/v1 | must carry the version segment |
| API Key | your gateway key | sent as a Bearer token, scoped to this instance |
One instance holds one endpoint. Separate instances are the intended way to keep a test endpoint and a production endpoint from bleeding into each other, which is why the instance name matters more than it looks: it is how you tell them apart later, both in the UI and in the model ID, which is rendered as OpenAI-API-Compatible/<instance>/<model>.
After saving, RAGFlow verifies the connection. Only then can you add models.
The /v1 rule changed at v0.27.0
This is the single most common reason a correct-looking base URL fails.
| Version | What RAGFlow does with a base URL that has no /v1 |
|---|---|
| v0.27.0 – v0.27.2 | appends the missing /v1 for you |
| earlier | sends it as written |
The practical rule that works on every version: always write the version segment yourself. Never rely on RAGFlow to add it, and never paste a full /chat/completions URL, because that gets the operation appended to it and produces a path with the operation twice.
If you are on 0.27.0 through 0.27.2 and you wrote the version segment, the auto-append does not fire and you are fine. If you are on an earlier version and you omitted it, nothing adds it and every call 404s.
The error message tells you nothing
Saving an instance runs a test call, and when it fails the message is:
Fail to access model(OpenAI-API-Compatible/…).No valid response receivedThis is RAGFlow's blanket conclusion for a failed test chat. It is the same string for a wrong key, a wrong path, an unreachable host, an exhausted quota, and a model name that does not exist. Treating it as a diagnosis wastes the first hour.
What actually distinguishes the causes is the endpoint's own log and the shape of the failure:
- Nothing arrives at the gateway → the URL, the host, or DNS. Check the path first.
- A 401 arrives → the key or the header, not the model.
- A 404 arrives with the operation twice in it → the base URL includes
/chat/completions. - A 400 about the model arrives → the model row exists but its name does not match what the gateway serves.
Verification runs a chat completion, so an endpoint that only serves embeddings cannot be verified this way. Add the embedding models and use them in a knowledge base instead.
Max tokens is not a generation cap
This field is widely misread, and the misreading costs money in one direction and truncated answers in the other.
RAGFlow's Max tokens is the context budget it reserves for chat history plus retrieved chunks. It is not sent to the model, and it does not cap the model's reply. Setting it to 128 does not make answers short; it makes RAGFlow drop retrieved content to fit.
Setting it to 0 falls back to 8192. That is the quiet default that surprises people who deliberately set a small number to save money and then wonder why retrieval got worse instead.
The generation cap is a separate setting on the model row, and on many gateways it is not enforced at all.
Adding models, and the catalog prefill
Two routes in.
List models adds models RAGFlow already knows about for this provider type. If your model is not there, use Add custom model: the name must match the model ID your gateway exposes exactly, character for character. A near miss is not fixed up and not reported; the row saves and calls fail.
One behaviour worth knowing before you click: the list route can prefill the entire catalog as Chat type, including model IDs that are image or embedding models. Those rows are wrong, and they sit in the picker looking valid. Remove the ones you will not use.
Tool call is a per-model toggle. Enable it only for models your gateway lists as function-calling capable. Turning it on for a model that does not return structured tool calls makes agent runs fail in a way that reads like a prompt problem.
Where the tokens go
A RAGFlow answer is not one request per question. The query is embedded, chunks are retrieved, and the question plus every retrieved chunk plus the conversation history goes to the model on every turn.
So the input side scales with retrieval settings, not with the question. Raising the chunk count raises every turn, including the turns where the answer is one sentence. Output is a single answer per turn.
A day on this page's workload is 200k input and 40k output tokens: 4M input and 0.8M output across 20 working days. Retrieval depth and the Max tokens budget are the two levers, and both are on the input side.
| Model | Gateway rate in / out per 1M tokens | One day | 20 days |
|---|---|---|---|
| gpt-5.6-terra | $1.00 / $6.00 | $0.44 | $8.80 |
| claude-sonnet-5 | $1.00 / $5.00 | $0.40 | $8.00 |
| deepseek-v4-pro | $0.66 / $1.98 | $0.21 | $4.22 |
| qwen3.8-max | $1.00 / $3.00 | $0.32 | $6.40 |
| glm-5.3 | $0.70 / $2.2 | $0.23 | $4.56 |
One day = 200k input + 40k output on this page's workload. 20 days = 4M input and 0.8M output. Rates checked 2026-10-14, gateway rates move with upstream promotions, so verify the current number in your dashboard before committing to a budget.
Verifying the route
- Choose
OpenAI-API-Compatible, not a named vendor with a similar-looking URL field. - Name the instance something other than
default, so you can tell instances apart later. - Include
/v1yourself. The auto-append only exists on 0.27.0–0.27.2 and only fires when it is missing. - Never paste a URL ending in
/chat/completions. - Read the failure, do not trust the message. Check the gateway log to tell URL from key from model.
- Set Max tokens deliberately. It budgets context, not the reply, and 0 silently becomes 8192.
- Prune the prefilled catalog so embedding and image IDs are not sitting in the chat picker.
FAQ
FAQ
Why does saving the instance fail with "No valid response received"?
That message is RAGFlow's generic test-chat failure. It covers a wrong key, a wrong path, an unreachable host and a missing model equally. Check the gateway's log to see whether a request arrived and what it looked like.
Does the base URL need /v1?
Write it yourself. Versions 0.27.0 to 0.27.2 add it when absent and earlier versions do not, so including it is the only value that works everywhere.
Why does the model row save but every call fail?
The name in the row does not match the model ID the gateway serves. Add it as a custom model and copy the ID from the gateway's models route rather than typing it.
Does Max tokens cap how long the answer is?
No. It budgets the context RAGFlow assembles from history and retrieved chunks, and it is not sent to the model. A value of 0 falls back to 8192.
Why is an image model listed as a chat model?
The list route can prefill the whole catalog as Chat type. Those rows are not valid for chat; delete the ones you do not need.
Can I use two endpoints in one RAGFlow install?
Yes, as two instances under the same provider. Instances are independent, which is what they were introduced for.
Related
- Langflow with a custom endpoint, the same version-drift problem, with an SSRF block on top
- MaxKB with a custom endpoint (the same provider-and-model-name-must-match rule), with a different field layout
- Base URL not working, the doubled path failure, in general form
- OpenAI-compatible error codes (what 401), 404 and 429 mean with a gateway in the middle
- Context length exceeded — what happens when the assembled prompt outgrows the window
- Tools that take one endpoint — the full matrix