context_length_exceeded on a custom endpoint

Your client does not know how big the window is — the server does. Your client sends whatever you handed it, and the server either rejects the request or, worse, quietly trims it to fit. Both outcomes produce a valid-looking response; only one tells you what happened.

curl https://aicomp.ai/v1/models \
  -H "Authorization: Bearer sk-your-gateway-key"

What the error actually says

The response body carries a code, a message naming the model's limit, and often the count you sent:

{
  "error": {
    "message": "This model's maximum context length is 1048576 tokens. "
               "You requested 1203433 tokens in the messages.",
    "type": "invalid_request_error",
    "param": "messages",
    "code": "context_length_exceeded"
  }
}

Two numbers, and the gap between them is your whole diagnosis. Everything else — how big you thought the file was, how many messages you remember sending — is unreliable, because the count includes the system prompt, the entire history, every tool result appended during the session, and every retrieved document. None of that is visible from where you are sitting.

Two independent limits

The input window and the maximum output are separate caps, and the failure modes differ:

• Window exceeded — the request is rejected before inference. You get an error and, usefully, the number.
• Output cap reached — the request runs and stops early. You get a valid response with finish_reason of length, truncated mid-thought, and no error at all.

Among the models in our catalogue where both figures are published, the input window is between 3.1× and 8.2× the maximum single output. So on generation-heavy work the output cap is the binding constraint even though nobody talks about it.

Only 7 of the models we track publish both figures — the rest leave one or both unspecified, which is itself worth knowing before you plan around them. Rates checked 2026-09-26; gateway rates move with upstream promotions.

Stop estimating, start measuring

The only authoritative count of what you are sending is the one the server reported on your last successful call. Make it non-streamed, because the usage field is always populated there, and read prompt_tokens.

# The only authoritative count of what you are sending is the one the
# server reported on your last successful call. Estimates are always wrong
# because system prompts, tool results and retrieved documents are all
# counted and none of them are visible in your editor.
last = client.chat.completions.create(
    model="claude-sonnet-5",
    messages=messages,
    stream=False,          # non-streamed: the usage field is always populated
)

used = last.usage.prompt_tokens
print(f"last successful call: {used} input tokens")
print(f"implied headroom   : {WINDOW - used}")   # WINDOW = the model's published value

# Then, when it finally fails, read the number the server says rather than
# arguing with it -- the error body carries the model's own limit.
# If the response instead comes back short and complete, nothing was rejected:
# something upstream truncated, and finish_reason is the tell.
print("finish_reason:", last.choices[0].finish_reason)   # "length" = capped by max_tokens

That gives you the number to reason with, and it gives you a baseline to watch. Plot it across a session and the shape tells you which part of the context is growing — a steady linear climb is history, a jump is a tool result or a retrieved document.

The version that never errors

Worth repeating, because it is the one that costs the most: clients infer the window from the model name. Serve a model under a custom id and the client may assume a small window and trim your prompt before it is sent. No rejection, no error message, a perfectly valid response — built on less context than you provided.

If your client has a model configuration section, fill in the window and the maximum output explicitly for every custom id. If it does not, treat any unexplained quality drop on long inputs as this failure until proven otherwise. Our /v1/models page covers where the id comes from in the first place.

The order to fix things in

1. Compact the conversation. Replace history with a summary of decisions made. Usually the single largest recovery.
2. Trim tool results. A file read or a shell output is often tens of thousands of tokens for a few hundred that mattered. Keep the relevant extract.
3. Drop retrieved documents that did not change the answer. Retrieval returns candidates, not all of which are needed in context.
4. Set max_tokens deliberately. An unnecessarily large value does not cause the error but it does cap output incorrectly the other way.
5. Then change models. Bigger window, same task — and re-measure, because a larger window on a per-token basis also means a larger worst-case request.

Steps one to three also cut cost, which is the reason to do them before the model swap rather than after. Across the 216 models in our cleaned catalogue the median output-to-input ratio is 4.0×, and on agent-shaped work the input side dominates the invoice — the input/output breakdown shows how to tell which half yours sits on.

Related guides

• One endpoint for every tool — the hub for tool-specific setup
• Reading /v1/models — where the id and the window hint come from
• Streaming and usage — getting the token count you need to see this coming
• Output vs input pricing — which side of the sum your workload lives on
• Cheapest model by workload — repricing the same task
• Claude Code 429 — the other limit that stops a session mid-flight

Context-limit failures and what each one is actually telling you

When this is the wrong move

Four moves that fix the wrong limit

Rolling it out without finding out the hard way

The failure mode you want to avoid is discovering a problem through a production bill or a customer-visible error. Four steps, in order:

1. Prove it on one long session with prompt_tokens and finish_reason logged per turn. One call, from a script, with an explicit timeout and the model ID echoed back. You are testing reachability, authentication and model availability — three things that can fail independently.
2. Measure before you switch. Record tokens per task and cost per task on the current path first. Without that baseline, "it got cheaper" is an impression, not a result.
3. Move one workload, not everything. Pick the workload with the most predictable shape — batch jobs over interactive traffic. Leave the interactive path on the old configuration until the batch numbers are in.
4. Decide the rollback condition in advance. Write down what makes you revert (error rate above X, cost per task above Y, latency above Z) before you start, so the decision is not made under pressure.

Keep the base URL in configuration, never inline. That single choice is what makes step four take a minute instead of an afternoon.

FAQ

Get API access