Haystack with a custom OpenAI-compatible endpoint
Haystack is a Python orchestration framework, and its LLM components are thin wrappers over the OpenAI SDK. That is good news for a gateway: the same api_base_url parameter that redirects to Azure OpenAI also redirects to any OpenAI-compatible endpoint, and nothing in your pipeline graph changes.
It is also the source of the only real trap. Two components read that parameter, they append different paths to it, and neither complains about a wrong value until you run the pipeline.
https://aicomp.ai/v1).
Create one free →
One parameter, two components
The generator and the embedder are separate components with separate base URLs:
| Component | Import | What it appends |
|---|---|---|
| Generator | haystack.components.generators | the chat completions route |
| Chat generator | haystack.components.generators.chat | the chat completions route |
| Embedder | haystack.components.embedders | the embeddings route |
Because both are built on the OpenAI SDK, both expect a base URL that stops at the version segment. The version segment is yours; the operation path is appended by the component. Pasting a full request URL into api_base_url produces a doubled route, and the error that comes back is a 404 that names the concatenated path rather than the field you set.
The rule is the opposite of the one some other clients in this index use. Aider, for instance, wants the full path in some configurations. Haystack wants the root.
The key is a Secret object, not a string
api_key accepts a plain string, and that is the fastest way to get a first call working And the fastest way to ship a credential. The intended shape is a Secret:
Secret.from_env_var("MY_KEY")reads the variable at call time, not at import time.Secret.from_token("...")wraps a literal for cases where no secret exists, such as a local server behind a placeholder key.
The distinction matters at deploy time. from_env_var defers the lookup, so a variable set by your process manager after import is still found. A plain string captured at import is not.
Note that a local or self-hosted endpoint still needs some key value. The OpenAI SDK refuses to construct a client without one, so a placeholder token is the standard workaround rather than a misconfiguration.
Embeddings are a second Integration, not a detail
The most expensive mistake in a Haystack migration is treating the embedder as a copy of the generator. Vector dimensions are a property of the model, and they are written into the document store when you index.
If you keep the same embedding model name against a different endpoint, dimensions match and nothing changes. If you switch the embedding model, the store's configured dimension no longer matches the vectors the embedder returns, and the failure appears at write time rather than at configuration time. Recreating the store is the fix, and it is worth checking the returned vector length on a single document before indexing a corpus.
Prompt caching travels in extra body fields
Provider-side caching is not part of the OpenAI schema, so it is exposed through extra_body and forwarded by whichever gateway you use. In Haystack that means generation_kwargs, which are passed through to the request rather than interpreted by the component:
generation_kwargs={
"temperature": 0,
"extra_body": {"cache_control": {"type": "ephemeral"}},
}Because the component does not validate these, a gateway that ignores an unknown field is indistinguishable from one that honours it until you compare bills. Keep the cacheable prefix, the system prompt and tool definitions, stable and in a fixed order, or the cache never hits regardless of the header.
What a RAG pipeline actually costs
A document question-answering pipeline is an unusual workload: the retriever supplies most of the input, and the answer is short.
A working configuration here runs about 200k input and 40k output tokens a day, which is 4M input and 0.8M output across 20 working days. The ratio is the reason the generator and the embedder should be sized separately The embedder has no output tokens at all, so a blended per-model comparison hides it entirely.
| Model | Gateway rate in / out per 1M tokens | One day | 20 days |
|---|---|---|---|
| gpt-5.6-terra | $1.00 / $6.00 | $0.44 | $8.80 |
| claude-sonnet-5 | $1.00 / $5.00 | $0.40 | $8.00 |
| deepseek-v4-pro | $0.66 / $1.98 | $0.21 | $4.22 |
| glm-5.3 | $0.70 / $2.2 | $0.23 | $4.56 |
One day = 200k input + 40k output on this page's workload. 20 days = 4M input and 0.8M output. Rates checked 2026-10-15, gateway rates move with upstream promotions, so verify the current number in your dashboard before committing to a budget.
If you are picking a generation model for retrieval-augmented work, the <a href="/guides/cheapest-models-by-workload/">re-ranked index</a> sorts by workload shape rather than by input price, which is the comparison a retrieved-context pipeline actually needs.
FAQ
FAQ
Does Haystack need the base URL to include /v1?
Yes, as the trailing element. The components append the operation path, so the base URL has to include the version segment and stop there. A full request address in that field produces a doubled path.
Can I use one endpoint for the generator and the embedder?
Yes. Both take api_base_url, and a gateway that serves both routes can back the whole pipeline. Check that the embeddings route is actually exposed, some gateways proxy chat completions only.
Why does my document store reject embeddings after I changed the endpoint?
Almost always a dimension mismatch rather than an endpoint problem. Dimensions belong to the embedding model, and the store was sized when you indexed. Keeping the same model name keeps the dimension; changing the model requires recreating the store.
Do I need a real API key for a local server?
No. The SDK requires a non-empty value to construct a client, so a placeholder token is expected. That is a construction requirement, not authentication.
How do I pass provider-specific options like cache control?
Through generation_kwargs, which are forwarded to the request rather than interpreted by the component. Whether they take effect depends on the gateway, and the component cannot tell you either way.
Related
- Tools that take one endpoint — the full matrix of clients and how each one names the endpoint
- Base URL not working (the doubled-path failure), which is what a full URL in this field produces
- OpenAI-compatible endpoints — the request shape both the generator and the embedder target
- LangChain with a base URL override — the same provider-plus-base-URL pattern in a different framework
- LlamaIndex with a custom endpoint — another Python framework where the embedder and the LLM are configured apart
- Official vs gateway prices — how the two price columns on this site are measured