Streaming through a custom endpoint: the usage chunk you are not getting

Turn on stream: true and the token counts you were relying on quietly disappear. Every chunk reports usage: null except one, that one has an empty choices array, and if the stream breaks you never receive it at all. None of this raises an error, which is what makes it expensive.

curl https://aicomp.ai/v1/models \
  -H "Authorization: Bearer sk-your-gateway-key"

What streaming does to the usage field

OpenAI's API reference is explicit about the shape. With stream_options set to include usage, an additional chunk is streamed before the data: [DONE] message. On that chunk, usage carries the statistics for the whole request and choices is always an empty array. On every other chunk, usage is present but null.

Two consequences follow directly, and both are the kind that show up as a bug report rather than as an exception:

• Without the flag there is no count at all. Not a partial count — no usage object to read. A per-request cost line that reads usage.prompt_tokens records nothing.
• With the flag, the useful chunk breaks naive code. It is the only chunk with an empty choices list, so the line that reads choices[0].delta.content raises on exactly the chunk carrying the numbers.

from openai import OpenAI

client = OpenAI(base_url="https://aicomp.ai/v1", api_key=KEY)

stream = client.chat.completions.create(
    model="gpt-5.6-luna",
    messages=[{"role": "user", "content": "Summarise this file in three lines."}],
    stream=True,
    stream_options={"include_usage": True},   # <- produces the final usage chunk
)

text, usage = [], None
for chunk in stream:
    if not chunk.choices:
        # This is the usage chunk: choices is an empty list, usage is populated.
        usage = chunk.usage
        continue
    delta = chunk.choices[0].delta
    if delta.content:
        text.append(delta.content)

print("".join(text))
print("usage:", usage)
#   -> None  means the endpoint ignored stream_options. No error is raised.
#             Your token counter is now zero for every request it sends.

That guard — if not chunk.choices — is the whole fix for the second problem. It is also the only place you can distinguish "the endpoint returned no usage" from "the request produced no tokens".

The flag is a request, not a guarantee

This is the part that costs money. stream_options is a field you send; whether the endpoint implements it is up to the endpoint. One that does not will accept the request, stream perfectly good text, and never emit the usage chunk. No error surfaces. Your counter reads null and your dashboard reads zero.

So the check is not "did I set the flag" but "did a usage object come back". Run the snippet above once per endpoint and per model — behaviour can differ between models behind the same base URL — and record the answer. If it is null, you have two honest options: count tokens client-side, or stop streaming for the requests you need to attribute.

A missing chunk is unknown, not zero

OpenAI's documentation carries a note that deserves more attention than it gets: if the stream is interrupted, you may not receive the final usage chunk, and not receiving it does not mean the request consumed no tokens. A client disconnect, a timeout, a dropped connection mid-generation — all of them produce a request that generated output and a tracker that recorded none.

The correct representation in your data model is therefore not 0 but unknown, and unknowns should be reconciled against the provider's own usage view rather than left as zeros in a sum.

What the gap is worth

Below: one day of 2,000 streamed requests at 8k input and 700 output each — a modest assistant-shaped workload, not a heavy one. The first column is what the work actually costs; the last is what a tracker that never receives a usage chunk records.

Rates checked 2026-09-26; gateway rates move with upstream promotions. The workload shape is illustrative — substitute your own request count and token shape.

# How much spend a missing usage chunk actually hides.
# Shape: 2,000 streamed requests a day, 8k input / 700 output each.
REQUESTS, TIN, TOUT = 2000, 8_000, 700

# $ per 1M tokens -- replace with the rates for the model you route to.
IN_RATE, OUT_RATE = 1.0, 5.0


def day_cost(recorded_fraction):
    """recorded_fraction = share of requests whose usage you actually captured."""
    return (REQUESTS * TIN * recorded_fraction) / 1e6 * IN_RATE \
        + (REQUESTS * TOUT * recorded_fraction) / 1e6 * OUT_RATE


for label, f in [("non-streamed (usage always present)", 1.0),
                 ("streamed, stream_options ignored", 0.0),
                 ("streamed, 4% of streams interrupted", 0.96)]:
    print(f"{label:40} ${day_cost(f):7.2f}/day   ${day_cost(f) * 20:8.2f}/20d")

Two streaming failures that are not about usage

Worth separating, because both get reported as "the endpoint is broken":

• Implemented /chat/completions but not stream: true. A plain curl returns a clean response and the client hangs, because clients default to streaming. Always test with streaming on before concluding anything.
• Buffered responses. An intermediary that buffers the stream produces no tokens until the end, which reads as a hang and trips client-side timeouts that a non-streamed call would pass. Our LangChain page covers the related case where a non-OpenAI base URL turns streaming usage off inside the framework.

Related guides

• One endpoint for every tool — the hub for tool-specific setup
• OpenAI SDK with a custom endpoint — the non-streamed baseline
• Monitoring API spend — reconciling what you record against what you are billed
• Output vs input pricing — which half of the sum your workload lives on
• LangChain on a custom endpoint — stream_usage being disabled silently
• Endpoint error codes — the failures that return a 200

Streaming-accounting failures and what each one is actually telling you

When this is the wrong move

Four habits that make a streaming integration quietly expensive

Rolling it out without finding out the hard way

The failure mode you want to avoid is discovering a problem through a production bill or a customer-visible error. Four steps, in order:

1. Prove it on one streamed request per model, with the usage object printed back. One call, from a script, with an explicit timeout and the model ID echoed back. You are testing reachability, authentication and model availability — three things that can fail independently.
2. Measure before you switch. Record tokens per task and cost per task on the current path first. Without that baseline, "it got cheaper" is an impression, not a result.
3. Move one workload, not everything. Pick the workload with the most predictable shape — batch jobs over interactive traffic. Leave the interactive path on the old configuration until the batch numbers are in.
4. Decide the rollback condition in advance. Write down what makes you revert (error rate above X, cost per task above Y, latency above Z) before you start, so the decision is not made under pressure.

Keep the base URL in configuration, never inline. That single choice is what makes step four take a minute instead of an afternoon.

FAQ

Get API access