Point AnythingLLM at an OpenAI-compatible endpoint
AnythingLLM runs two models, not one: an embedder that ingests your documents once, and a chat model that answers every query forever. Conflating the two is how people end up optimising the wrong bill.
/v1 and a key from that endpoint. The setting below is the only thing that changes — request and response handling stay identical. Rates checked 2026-09-26.The setting
The connector for arbitrary OpenAI-compatible endpoints is the generic provider. In an environment file it looks like this:
LLM_PROVIDER=generic-openai
GENERIC_OPEN_AI_BASE_PATH=https://aicomp.ai/v1
GENERIC_OPEN_AI_API_KEY=sk-...
GENERIC_OPEN_AI_MODEL_PREF=claude-sonnet-5
GENERIC_OPEN_AI_MODEL_TOKEN_LIMIT=8192
# Embedding is a SEPARATE, one-off cost — 'native' makes it free
EMBEDDING_ENGINE=native
VECTOR_DB=lancedbOr the same thing on the command line:
docker run -d \
--name anythingllm \
-p 3001:3001 \
-v anythingllm-data:/app/server/storage \
-e LLM_PROVIDER=generic-openai \
-e GENERIC_OPEN_AI_BASE_PATH="https://aicomp.ai/v1" \
-e GENERIC_OPEN_AI_API_KEY="sk-..." \
-e GENERIC_OPEN_AI_MODEL_PREF="claude-sonnet-5" \
mintplexlabs/anythingllmTwo things about those variables. GENERIC_OPEN_AI_BASE_PATH is
mandatory — the provider throws if it is unset rather than falling back to
a default — and it needs the /v1 segment. The key, by contrast, is optional and
defaults to unset, which is what you want against a local server that does no
authentication.
Embedding is a separate decision with its own engine setting. The built-in one runs locally and costs nothing per document, which is the right default for most self-hosted setups.
The two bills, and which one matters
This is the part worth reading twice. Ingesting a thousand documents at roughly 2k tokens each is about 2M tokens — and you pay that once. Answering two hundred queries a day is about 48M input and 2.4M output a month, and you pay that every month, indefinitely.
| Model | Vendor | Rate in / out per 1M | 4,000 queries | Input share of the bill | Per month |
|---|---|---|---|---|---|
| gpt-5.6-luna | OpenAI | $0.1 / $0.6 | $6.24 | 77% | $6 |
| MiniMax-M3 | MiniMax | $0.15 / $0.6 | $8.64 | 83% | $9 |
| deepseek-v4-flash | DeepSeek | $0.22 / $0.66 | $12.14 | 87% | $12 |
| gemini-3.7-flash | $0.375 / $1.875 | $22.50 | 80% | $22 | |
| claude-haiku-4-5-20251001 | Anthropic | $0.5 / $2.5 | $30.00 | 80% | $30 |
| deepseek-v4-pro | DeepSeek | $0.66 / $1.98 | $36.43 | 87% | $36 |
| glm-5.3 | Zhipu | $0.7 / $2.2 | $38.88 | 86% | $39 |
| qwen3.8-max | Alibaba | $1 / $3 | $55.20 | 87% | $55 |
| claude-sonnet-5 | Anthropic | $1 / $5 | $60.00 | 80% | $60 |
| gpt-5.6-terra | OpenAI | $1 / $6 | $62.40 | 77% | $62 |
| kimi-k3 | Moonshot | $1.5 / $7.5 | $90.00 | 80% | $90 |
| claude-opus-5 | Anthropic | $2.5 / $12.5 | $150.00 | 80% | $150 |
Input is between 77% and 87% of that. Now put the other bill next to it. Ingesting a thousand documents is 2M tokens, paid once, and it is 3.1%–3.5% of the first month's total — after which it is zero, forever, because nothing re-embeds a document that has not changed. A model that costs twice as much to embed with and ten per cent less to query with is still the cheaper choice by the second month, by a wide margin.
The ratio on queries is about 20:1, so input decides the ranking here as it does everywhere on this site. The difference is that on this page there is a second, separate cost most people never look at, and it is the one you can make disappear entirely by using the built-in embedder.
One setting moves the query figure more than the model does, and it is not a model setting at all: chunk size. Every query pays for whatever retrieval returns, so chunking that yields more and smaller chunks raises the tokens per query without necessarily improving the answer. If a query is sending twelve thousand tokens of context to produce three paragraphs, the chunking is the thing to fix — and fixing it is free, where changing model is not.
How this fails in practice
| What you see | What it usually is | Fix |
|---|---|---|
| GenericOpenAI must have a valid base path | GENERIC_OPEN_AI_BASE_PATH is unset | Set it, including /v1, then restart |
| 404 on every chat request | Version segment missing from the base path | End the path with /v1 |
| Requests hit a doubled path | Trailing slash after the version segment | Remove the trailing slash |
| UI still shows the previous provider | Variables read at startup, or a workspace overrides the default | Restart, then check the workspace's own chat settings |
| Embedding fails while chat works | Embedding engine configured independently of the LLM | Set EMBEDDING_ENGINE and its model preference separately |
| Answers ignore uploaded documents | Nothing was embedded, or the workspace points elsewhere | Re-embed the workspace and check the vector store |
https://aicomp.ai/v1).
Create one free →
Confirming it took effect
Ask a question whose answer appears only in a document you uploaded. If it answers from the document, both halves are working — the embedder ingested it and the chat model is reading the retrieved context. Then read the usage log on the endpoint: a request recorded there confirms which provider actually answered, because the UI will happily show a provider that is not the one being used.
FAQ
Why does AnythingLLM say it needs a valid base path?
Because the generic OpenAI-compatible connector treats GENERIC_OPEN_AI_BASE_PATH as mandatory and throws immediately if it is unset, rather than quietly falling back to a default. This is deliberate: the whole point of the generic provider is to talk to an endpoint you specify, so there is no sensible default to fall back to. Set the variable to your endpoint including the /v1 segment and restart.
Does the base path need /v1 on the end?
Yes. The provider builds its request paths on top of whatever you give it, so omitting the version segment produces requests against paths that do not exist — usually a 404 that reads like an endpoint outage rather than a configuration typo. Equally, do not add a trailing slash, because the version segment is the last thing it expects and a slash after it produces a doubled separator in the final URL.
Is the API key required?
Not strictly — the key is optional on the generic provider and defaults to unset, which is what you want for a local server that does no authentication. For a hosted endpoint you will set it. Either way the base path is the one setting that is genuinely mandatory.
Why is embedding configured separately from the chat model?
Because they are two different jobs with two different cost profiles. Embedding converts your documents into vectors and happens once per document, at ingest. The chat model runs on every single query, forever. AnythingLLM lets you pick them independently, which is the right design: it means you can use the free built-in embedder and still use a strong model for answers.
Should I choose a model based on its embedding price?
No — that is the most expensive mistake available on this page. Embedding is a one-off cost paid when documents are ingested, while queries are paid continuously, thousands of times a month. A model that is marginally more expensive to embed with and much cheaper to query is overwhelmingly the better pick. Decide the embedding engine first, then choose the chat model on query cost alone.
How much context does each query actually send?
Whatever the retrieval returns, which is a consequence of your chunk settings and similarity threshold rather than of the model. Typical queries carry several retrieved chunks — on the order of ten thousand tokens — against a few hundred tokens of answer. That makes this an input-heavy workload, so the input rate is the one that decides the bill.
The UI still shows the old provider after I changed the environment file.
Two likely causes. The process reads environment variables at startup, so a restart is required. And a workspace can override the system default: if a specific workspace has its own LLM selection, it wins for that workspace regardless of what the system setting says. Check the workspace's own chat settings before concluding the environment change failed.