LangChain with a custom endpoint

LangChain wraps the OpenAI client rather than replacing it, so pointing it at a different endpoint is one kwarg. The part that costs money is subtler: changing the base URL also changes whether LangChain asks for token usage on streamed responses, and that decision is what your per-request cost numbers rest on.

curl https://aicomp.ai/v1/models \
  -H "Authorization: Bearer sk-your-gateway-key"

What you need before you start

• A gateway API key and the base URL (https://aicomp.ai/v1)
• pip install -U langchain-openai — the integration is a separate package
• A model ID copied from the endpoint's /v1/models list

Python — the minimal change

Everything else in your chain survives untouched. Prompts, output parsers, retrievers and agent loops sit above the model object, so replacing the model is the whole change.

import os
from langchain_openai import ChatOpenAI

llm = ChatOpenAI(
    model="gpt-5.6-luna",                 # exact ID from /v1/models
    api_key=os.environ["GATEWAY_API_KEY"],
    base_url="https://aicomp.ai/v1",
    temperature=0.2,
    max_retries=2,
)

msg = llm.invoke("In one sentence: what is a rate limit?")
print(msg.content)

Two arguments carry the routing. base_url decides where the request goes and must include the /v1 segment, because the client appends the resource path itself. api_key supplies the credential — and it is worth passing explicitly rather than relying on OPENAI_API_KEY, because a stale variable left in a long-lived shell is the most common cause of a 401 that looks like a broken base URL.

TypeScript — a different shape

The JS client does not mirror the Python one. The base URL belongs inside a configuration object, and putting it at the top level of the constructor does nothing.

import { ChatOpenAI } from "@langchain/openai";

// baseURL belongs inside `configuration`, not at the top level.
const llm = new ChatOpenAI({
  model: "gpt-5.6-luna",
  apiKey: process.env.GATEWAY_API_KEY,
  maxRetries: 2,
  configuration: {
    baseURL: "https://aicomp.ai/v1",
  },
});

const msg = await llm.invoke("In one sentence: what is a rate limit?");
console.log(msg.content);

The one behaviour worth testing before you trust your cost numbers

LangChain inspects whether a non-OpenAI base URL is configured when it decides whether to default-enable stream_usage. When one is set, the default is switched off — the stated reason being that many non-OpenAI endpoints do not support streaming token usage. The consequence is specific and easy to miss: a chain that used to report usage_metadata can start reporting None, with no exception, no warning, and a response that looks entirely correct.

That is not a LangChain bug. It is a reasonable default applied to an endpoint whose capabilities it cannot know. What makes it expensive is that per-request cost attribution usually reads exactly that field, so the dashboards quietly go empty while the bill keeps accruing.

import os
from langchain_openai import ChatOpenAI

# The point of this snippet: find out whether the endpoint actually reports
# token usage back. If usage_metadata is None, your cost dashboards are
# blind from here on -- not wrong, just empty.
llm = ChatOpenAI(
    model="gpt-5.6-luna",
    api_key=os.environ["GATEWAY_API_KEY"],
    base_url="https://aicomp.ai/v1",
    stream_usage=True,          # ask for usage on streamed responses
)

msg = llm.invoke("Say: ok")
print("usage_metadata:", getattr(msg, "usage_metadata", None))
print("response_metadata keys:", sorted(getattr(msg, "response_metadata", {}).keys()))

# Non-streaming call: the usage block here is what a cost callback would read.
raw = llm.invoke("Say: ok", config={"callbacks": []})
print("input_tokens :", (raw.usage_metadata or {}).get("input_tokens"))
print("output_tokens:", (raw.usage_metadata or {}).get("output_tokens"))

If the probe prints None, you have three options in increasing order of effort: ask the endpoint whether it can return a usage block on streamed responses and enable it if so; fall back to non-streaming calls for the subset of traffic you need metered; or meter tokens at the call site with your own counter and use the gateway's usage log only for reconciliation. The third is the one that survives an endpoint change.

What it costs

Rates checked 2026-09-26. Gateway rates move with upstream promotions — verify the current number in your dashboard before committing to a budget.

Rates checked 2026-09-26. Gateway rates move with upstream promotions — verify the current number in your dashboard before committing to a budget. Across the 216 models in our cleaned catalogue the median output-to-input ratio is 4.0×, which is why the model column matters more than any code change in this page.

What actually goes over the wire

Every OpenAI-compatible call is an HTTP POST to a base URL plus a resource path, with two things that decide everything else: the path, and the bearer token.

POST /v1/chat/completions HTTP/1.1
Host: aicomp.ai
Authorization: Bearer sk-...
Content-Type: application/json

{"model": "gpt-5.6-luna",
 "messages": [{"role": "user", "content": "..."}],
 "stream": true}

Read that literally, because three separate failure modes hide in it. The Host comes from your base URL — if the request arrives somewhere unexpected, this is the line that was wrong. The Authorization header is the only thing identifying you, so a shared key means indistinguishable spend. And the model value inside the body is not validated against anything client-side: send an ID the server does not serve and you get a server-side error, not a client-side warning.

Nothing else in the request is provider-specific. The message array, the tool schemas, the stream flag and the response shape are the wire format, and they are what makes one base URL substitutable for another. This is also the reason the change is reversible: you are not recompiling anything, you are editing one string.

LangChain failures and what each one is actually telling you

When this is the wrong move

Cases where plain ChatOpenAI is the wrong abstraction

Rolling it out without finding out the hard way

The failure mode you want to avoid is discovering a problem through a production bill or a customer-visible error. Four steps, in order:

1. Prove it on one non-critical chain that already has a cost callback. One call, from a script, with an explicit timeout and the model ID echoed back. You are testing reachability, authentication and model availability — three things that can fail independently.
2. Measure before you switch. Record tokens per task and cost per task on the current path first. Without that baseline, "it got cheaper" is an impression, not a result.
3. Move one workload, not everything. Pick the workload with the most predictable shape — batch jobs over interactive traffic. Leave the interactive path on the old configuration until the batch numbers are in.
4. Decide the rollback condition in advance. Write down what makes you revert (error rate above X, cost per task above Y, latency above Z) before you start, so the decision is not made under pressure.

Keep the base URL in configuration, never inline. That single choice is what makes step four take a minute instead of an afternoon.

FAQ

Get API access