DSPy with an OpenAI-compatible endpoint

DSPy is a framework for optimising prompts, and that is exactly why its bills surprise people. The requests that dominate are not the ones your program makes at runtime — they are the ones compile() makes while searching for a better prompt.

curl https://aicomp.ai/v1/models \
  -H "Authorization: Bearer sk-your-gateway-key"

Step 1 — configure the LM

import dspy

lm = dspy.LM(
    "openai/gpt-5.6-luna",
    api_base="https://aicomp.ai/v1",
    api_key="sk-your-gateway-key",
)
dspy.configure(lm=lm)

The openai/ prefix is what selects the OpenAI-compatible route through the provider resolver. A gateway-served model generally needs it.

Step 2 — model_type, when the endpoint needs it

# Endpoints that are not OpenAI's sometimes need the model type spelled out.
lm = dspy.LM(
    "openai/deepseek-v4-flash",
    api_base="https://aicomp.ai/v1",
    api_key="sk-your-gateway-key",
    model_type="chat",
)

Endpoints that are not OpenAI's own sometimes need the model type stated explicitly. The failure it produces tends to read like a model or format problem rather than a configuration one, so it is worth trying early when a call that works with curl fails through the framework.

Step 3 — the part nobody budgets: compile()

An optimiser runs your module across the trainset repeatedly: generating candidates, scoring them against the metric, retrying the ones that fail. Each of those is a request. A single compile() call can be hundreds of them.

import dspy
from dspy.teleprompt import BootstrapFewShot

optimiser = BootstrapFewShot(metric=answer_exact_match)
# 每一次 compile 都会在训练集上反复调用 LM —— 这是账单的主体
compiled = optimiser.compile(student=classifier, trainset=train_set)

The figures are computed from the rate table, at the stated call counts — your own trainset and optimiser settings will differ. The point is the order of magnitude: compiling is not a rounding error next to inference.

Compared with a single inference call at $0.00039, the ratio is the point. If your bill looks inexplicable, this is usually where it went.

Step 4 — count the calls once

Rather than estimating, wrap the LM and count. It is a few lines, and it converts the largest unknown in a DSPy project into a number you can act on.

import dspy

class CountCalls(dspy.Module):
    def __init__(self):
        self.n = 0

# 包装 LM，统计 compile 期间到底打了多少次
class CountingLM(dspy.LM):
    def __init__(self, inner):
        self.inner = inner
        self.calls = 0

    def __call__(self, *a, **kw):
        self.calls += 1
        return self.inner(*a, **kw)

wrapped = CountingLM(lm)
dspy.configure(lm=wrapped)
# ... 跑完 compile 之后
print("calls during compile:", wrapped.calls)

Two levers follow from the count. Optimise against a cheaper model — the optimiser is searching over prompts, and a cheaper model often finds a comparable one — then run the compiled program on the model you intend to ship. And shrink the trainset before shrinking anything else, because cost scales linearly with rows.

DSPy failures, ordered by how much they cost

When this is the wrong move

Where the optimisation loop costs more than it returns

Rolling it out without finding out the hard way

The failure mode you want to avoid is discovering a problem through a production bill or a customer-visible error. Four steps, in order:

1. Prove it on one compile against a 20-row subset. One call, from a script, with an explicit timeout and the model ID echoed back. You are testing reachability, authentication and model availability — three things that can fail independently.
2. Measure before you switch. Record tokens per task and cost per task on the current path first. Without that baseline, "it got cheaper" is an impression, not a result.
3. Move one workload, not everything. Pick the workload with the most predictable shape — batch jobs over interactive traffic. Leave the interactive path on the old configuration until the batch numbers are in.
4. Decide the rollback condition in advance. Write down what makes you revert (error rate above X, cost per task above Y, latency above Z) before you start, so the decision is not made under pressure.

Keep the base URL in configuration, never inline. That single choice is what makes step four take a minute instead of an afternoon.

Where the money actually goes, ranked

Cost advice tends to be a list of tricks with no ordering. These five are ordered by how much they typically move a real bill, and each one names the number that tells you whether it worked.

Concretely: one day at 1500k input / 400k output costs $0.390 on gpt-5.6-luna and $0.594 on deepseek-v4-flash — a difference of $0.204 per day, or $4.08 over a 20-day month.

The ordering is not universal — it depends on your token shape. On agent loops, context size outranks model choice; on bulk generation, model choice wins because there is no context to trim. Work out which half of the sum your workload sits on before spending effort on the wrong lever. Our output vs input pricing breakdown covers how to tell.

FAQ

Related

• Every tool that speaks OpenAI, pointed at one endpoint
• LangChain with a base URL override
• Pydantic AI with a custom endpoint
• Attributing spend per run
• Why output dominates most bills
• DeepSeek V4 Flash pricing

Get API access