LangChain4j with a Custom OpenAI-Compatible Endpoint

LangChain4j is the Java stack for LLM applications: a ChatModel interface, AI Services that turn interfaces into prompted calls, and RAG and tool plumbing on top. Pointing it at a gateway is one builder call, but the builder validates before it sends anything — and the thing it insists on is the one field a gateway sometimes makes you want to leave out.

apiKey is required, even when your endpoint does not use one. OpenAiChatModel.builder() throws on a null key at construction, before any request exists. A local or IP-allowlisted endpoint that needs no credential still has to be given one; the accepted workaround is a placeholder string.

This page covers the builder, the two spellings of the same property depending on whether you are in Spring, and the streaming setting that silently mangles tool calls on gateways that are not OpenAI.

The builder

ChatModel model = OpenAiChatModel.builder()
        .baseUrl("https://gw.example.com/v1")
        .apiKey(System.getenv("GATEWAY_API_KEY"))
        .modelName("gpt-5.6-terra")
        .logRequests(true)
        .logResponses(true)
        .build();

// keyless endpoint — the key is mandatory, so pass a placeholder
ChatModel model = OpenAiChatModel.builder()
        .baseUrl("http://10.0.0.4:8000/v1")
        .apiKey("none")
        .modelName("qwen3.8-max")
        .build();

baseUrl is the version root. LangChain4j appends the operation path, so the value ends at /v1 (or whatever your gateway serves under) and never at /chat/completions.

Three of these fields fail at different times, which is what makes the setup confusing:

• apiKey fails at build time, with a message about the key. It cannot be a connectivity problem yet, so that is the one place the error is actually accurate.
• modelName is sent verbatim. A wrong id is a model not found from your gateway at the first request.
• baseUrl fails at the first request, usually as a connection or 404 error that names the operation rather than the URL.

The Spring spelling

langchain4j.open-ai.chat-model.api-key=${GATEWAY_API_KEY}
langchain4j.open-ai.chat-model.base-url=https://gw.example.com/v1
langchain4j.open-ai.chat-model.model-name=gpt-5.6-terra
langchain4j.open-ai.chat-model.timeout=PT60S
langchain4j.open-ai.chat-model.max-retries=2

The prefix is open-ai, with a hyphen, not openai — the module name keeps the hyphen (langchain4j-open-ai, langchain4j-open-ai-spring-boot-starter). Write langchain4j.openai.chat-model.base-url and Spring Boot ignores it silently: no startup error, no warning, and the model falls back to api.openai.com with your gateway key, which returns 401.

That failure reads as "wrong key" and is really "wrong property name". If a Spring app is reaching OpenAI when you have configured a gateway, check the hyphen before you check the credential.

Streaming tool calls: accumulateToolCallId

StreamingChatModel model = OpenAiStreamingChatModel.builder()
        .baseUrl("https://gw.example.com/v1")
        .apiKey(System.getenv("GATEWAY_API_KEY"))
        .modelName("deepseek-v4-pro")
        .accumulateToolCallId(false)
        .build();

This one has no wrong-value symptom you would recognise. Some OpenAI-compatible APIs — DeepSeek and Qwen among them — send the complete tool call id in every streaming chunk, where OpenAI sends it in pieces. LangChain4j's default is OpenAI's behaviour: it concatenates the pieces across chunks. On an API that repeats the full id, the default concatenates the repeats and you get a mangled id like abcabcabc.

The result surfaces downstream as a tool that cannot be matched to its result, or a tool call that seems to vanish. Nothing in that error mentions the id, and the request and response both look correct in the logs. If streaming tool calls misbehave only on one gateway model, set accumulateToolCallId(false) before debugging anything else.

Two more that cost time

HTTP/2. Some endpoints and proxies — LM Studio is the documented case, and reverse-proxied gateways behave the same way — do not handle HTTP/2. The symptom is a hang or a protocol error rather than a clean failure. Forcing HTTP/1.1 needs both the alternative client module and a builder:

HttpClient.Builder httpClientBuilder = HttpClient.newBuilder()
        .version(HttpClient.Version.HTTP_1_1);

ChatModel model = OpenAiChatModel.builder()
        .baseUrl("https://gw.example.com/v1")
        .apiKey(System.getenv("GATEWAY_API_KEY"))
        .modelName("gpt-5.6-terra")
        .httpClientBuilder(JdkHttpClient.builder().httpClientBuilder(httpClientBuilder))
        .build();

Retries. maxRetries defaults to a small number and covers connection failures and 5xx. A retried request re-sends the whole input, so a gateway that is slow under load can bill you twice for a call that you only see once in your own logs. Set it explicitly and check it against monitoring API spend rather than assuming the default is free.

Where the tokens go

LangChain4j apps are usually RAG or AI Services rather than chat loops, and the shape follows from that. A RAG call sends the system prompt, the retrieved chunks and the question; the answer is short. An AI Service with tools sends the whole conversation again on every tool round, so one user call can be several full-context requests.

That makes the input side a function of your retrieval settings — how many chunks, how large, whether the history is trimmed. It is not a function of what the user typed, which is why these workloads surprise people: the question is 20 tokens and the request is 4,000.

A day of work on this page's workload is 200k input and 35k output tokens — 4M input and 0.7M output across 20 working days. Prompt caching is worth more here than in a chat workload because the system prompt and the retrieval template are stable across calls; see prompt caching savings for how much that changes the month, and output vs input pricing for the general rule.

One day = 200k input + 35k output on this page's workload. 20 days = 4M input and 0.7M output. Rates checked 2026-10-12 — gateway rates move with upstream promotions, so verify the current number in your dashboard before committing to a budget.

Verifying the route

1. Always set apiKey, even to a placeholder. A null key throws at build time, before any request.
2. Point baseUrl at the version root, not at the operation path.
3. In Spring, use open-ai with the hyphen. openai is ignored silently and falls back to OpenAI.
4. Set accumulateToolCallId(false) if streaming tool calls misbehave on a non-OpenAI model.
5. Force HTTP/1.1 if requests hang instead of failing.
6. Set maxRetries explicitly. Retries re-send the full input.

FAQ

FAQ

Related

• Spring AI with a custom endpoint — the other Java stack, where base-url carries no /v1
• Prism PHP with a custom endpoint — same idea in PHP, with the opposite default for /v1
• Semantic Kernel with a custom endpoint — the C# equivalent
• OpenAI SDK with a custom endpoint — what OpenAiChatModel is built on
• Monitoring API spend — catch retry amplification before it shows up on the invoice
• Tools that take one endpoint — the full matrix

Get API access