Use Claude in LangChain with a custom endpoint

By the aitokensai editors · Verified 2026-11-01 Sources: Anthropic docs · LangChain docs

LangChain talks to models through a chat model object, so changing where requests go is a constructor argument, not a rewrite of your chains, agents or parsers.

In short: In LangChain, set the endpoint through the chat model's URL parameter rather than monkey-patching the SDK (ChatAnthropic takes anthropic_api_url, ChatOpenAI takes base_url). Everything downstream (chains, agents, memory) keeps working unchanged.

You need a key before the code below runs. Create an account, generate a key, and copy the base URL (https://aicomp.ai). Create one free →

Check current rates → Free to sign up · $1 minimum top-up · No prepayment

Confirm the key works first. One command, no SDK, costs nothing:

Don't have an account yet? You need an API key from the gateway before the steps below will work — how the discount works and what you take on. Create one free → (no prepay, $1 minimum top-up)

curl https://aicomp.ai/v1/models \
  -H "Authorization: Bearer sk-your-gateway-key"

A JSON list of model IDs means the key is good. Invalid token means it was copied wrong.

Anthropic model

from langchain_anthropic import ChatAnthropic

llm = ChatAnthropic(
    model="claude-sonnet-5",
    anthropic_api_url="https://aicomp.ai",   # older versions: base_url
    anthropic_api_key="sk-your-gateway-key",
    max_tokens=1024,
    temperature=0,
)

print(llm.invoke("Say OK").content)

OpenAI-compatible path

The base URL here is not the same value as above. ChatOpenAI sends /chat/completions and takes the versioned base, while ChatAnthropic appends /v1/messages itself and takes the bare host.

from langchain_openai import ChatOpenAI

llm = ChatOpenAI(
    model="gpt-5.6-luna",
    base_url="https://aicomp.ai/v1",
    api_key="sk-your-gateway-key",
)

print(llm.invoke("Say OK").content)

Pick one path, not both. The Anthropic path is the safer choice for Claude models because tool calling and streaming follow the native contract; use the OpenAI path when you want one client across many vendors.

Everything above still works

Once the model object is constructed, nothing downstream changes, chains, agents, memory, retrievers and output parsers all read from it as before:

from langchain_core.prompts import ChatPromptTemplate

prompt = ChatPromptTemplate.from_messages([("human", "{question}")])
chain = prompt | llm
print(chain.invoke({"question": "Summarise rate limiting in one line."}).content)

Streaming and callbacks

for chunk in llm.stream("Explain prompt caching briefly."):
    print(chunk.content, end="", flush=True)

Streaming requires the endpoint to support SSE. If chunks arrive as one block, the endpoint is buffering rather than streaming (functionally fine, but you lose the time-to-first-token benefit).

Cost note. Agent loops are where LangChain bills surprise people. Every step resends the conversation, so a fifteen-step agent run can cost more than a hundred plain calls. Log usage per step — see how to monitor API spend.

Rates

Model rates reachable from LangChain — gateway rate vs official list
Model	Gateway rate in / out per 1M tokens	Official list in / out per 1M tokens	Diff
claude-haiku-4-5	$0.5 / $2.5	$1 / $5	50%
claude-sonnet-5	$1 / $5	$2 / $10	50%
claude-opus-5	$2.5 / $12.5	$5 / $25	50%

Rates checked 2026-11-01. Gateway rates move with upstream promotions, verify the current number in your dashboard before committing to a budget.

Troubleshooting

Symptom	Fix
Unknown field anthropic_api_url	Version mismatch (use base_url on older releases).
404 on messages	Doubled `/v1`. Check what the base URL already ends with.
401 authentication_error	Key not passed to the model object, or overridden by an env var. Pass it explicitly.
Tools not called	Bind tools with `llm.bind_tools(...)` and confirm the endpoint supports tool use.

Rate-limit failures and what each one is actually telling you

Symptom	What is actually happening	How to confirm	Fix
429s arrive in bursts, not under steady load	Limits are usually per-minute and per-token; a batch job can exhaust a minute's allowance instantly.	Plot requests per minute alongside the errors, bursts align with job starts.	Spread batch work with a rate limiter instead of firing it concurrently.
Retries make it worse instead of better	Immediate retries consume the same window that just refused you, so every retry deepens the hole.	Watch whether error rate rises after you add retries.	Back off exponentially with jitter, and honour any retry-after value returned.
One service starves another	Several services share one key, so one noisy caller consumes the shared allowance.	Split usage by model and by time; a single caller's spike will be visible.	Issue a separate key per service, which also makes spend attributable.
Limits hit sooner than the request count suggests	Long-context requests consume a token budget that is separate from the request count.	Compare input tokens per request against the errors, not just the request rate.	Trim context, or throttle on tokens rather than on requests.
Streaming connections appear to leak slots	A stream holds a connection for its whole duration, occupying concurrency that request-count limits do not model.	Count concurrent open streams at the moment errors begin.	Cap concurrent streams and close them explicitly on error paths.
The documented limit does not match what you observe	Limits vary by model and by account tier, and are not uniform across a catalogue.	Test one model at a time to isolate which limit you are actually hitting.	Treat published numbers as a starting point and measure your own ceiling.

When this is the wrong move

Three fixes that make rate limiting worse

Situation	Why it breaks	Do this instead
You are about to raise concurrency to fix a 429	Adding callers to a saturated window converts throttling into a queue that hides the real limit.	Reduce concurrency, then add back slowly while watching the error rate.
You plan to retry indefinitely	Unbounded retries convert a rate limit into an unbounded bill.	Cap attempts, back off, and surface the failure to a queue you can replay later.
You are sharing one key across environments	Production and development compete for the same allowance, and neither is attributable.	Separate keys per environment before you tune anything else.

FAQ

How do I point LangChain at a custom Anthropic endpoint?

Set the URL parameter on the chat model constructor instead of patching the SDK globally. In current LangChain that is anthropic_api_url on ChatAnthropic; some versions call it base_url. Everything downstream keeps working because it talks to the model object, not the client.

Does streaming still work?

Yes, if the endpoint supports server-sent events. Pass streaming=True or use the .stream() method as usual, and test it once before relying on it in production.

Can I use the OpenAI-compatible path instead?

Yes, when the endpoint exposes one. ChatOpenAI with base_url and a non-OpenAI model ID works for many gateways, but tool calling and structured output are the features most likely to diverge (test them explicitly).

Where do costs get out of hand in LangChain?

Agent loops. Each step resends the accumulated context, so a ten-step agent run costs far more than ten individual calls. Log token usage per step or you will not see it coming.

Do I need to change my prompts or parsers?

No. The endpoint change is transparent to everything above the model layer, prompts, output parsers, memory and retrieval behave identically.

Get API access