AutoGen with a custom endpoint

AutoGen 0.4 models an application as agents in conversation, and each agent carries its own model client. Pointing one at a different endpoint takes two arguments — but the second one, the capability declaration, is what decides whether your agents can use tools at all.

curl https://aicomp.ai/v1/models \
  -H "Authorization: Bearer sk-your-gateway-key"

What you need before you start

• A gateway API key and base URL (https://aicomp.ai/v1)
• Python 3.10 or later — on 3.9 pip silently installs the incompatible 0.2 release
• pip install autogen-agentchat "autogen-ext[openai]"

The client, with capabilities declared

import asyncio, os
from autogen_ext.models.openai import OpenAIChatCompletionClient
from autogen_agentchat.agents import AssistantAgent
from autogen_agentchat.teams import RoundRobinGroupChat
from autogen_agentchat.conditions import MaxMessageTermination

client = OpenAIChatCompletionClient(
    model="gpt-5.6-luna",
    base_url="https://aicomp.ai/v1",
    api_key=os.environ["GATEWAY_API_KEY"],
    model_info={
        "vision": False,
        "function_calling": True,     # REQUIRED for tool-using agents
        "json_output": True,
        "structured_output": True,
        "family": "unknown",
    },
)

async def main():
    writer = AssistantAgent("writer", client, system_message="Draft.")
    critic = AssistantAgent("critic", client, system_message="Critique.")
    team = RoundRobinGroupChat([writer, critic],
                               termination_condition=MaxMessageTermination(6))
    await team.run(task="Explain what a rate limit is, in two sentences.")

asyncio.run(main())

The model_info mapping is not decoration. AutoGen cannot probe an unknown endpoint for what the model supports, so anything you leave out is treated as unsupported. function_calling is the one that matters most: AssistantAgent and most other AgentChat agents require it, and the framework only tells you so after you attach a tool.

That ordering is what makes this hard to debug. A plain conversation succeeds — which is what everyone tests first — and then the first tool-using agent either raises or, worse, answers in prose while the workflow reports completion.

Cost: the product, not the sum

The arithmetic that catches people out is that a group chat bills one call per agent per turn. A team of three running eight turns is twenty-four calls for one task, and each turn carries the conversation history forward, so per-call input tokens grow as the run proceeds.

# A group chat does not cost one call per turn -- it costs one call per
# agent per turn. Two agents over six turns is twelve billed calls, and each
# turn carries the accumulated conversation history forward.

def autogen_calls(n_agents, n_turns):
    """Upper bound on billed calls for a RoundRobinGroupChat run."""
    return n_agents * n_turns

def autogen_month_cost(n_agents, n_turns, runs_per_day,
                       in_tok_per_call, out_tok_per_call,
                       in_rate, out_rate):
    calls = autogen_calls(n_agents, n_turns) * runs_per_day * 30
    return calls * (
        in_tok_per_call / 1_000_000 * in_rate
        + out_tok_per_call / 1_000_000 * out_rate
    )

# 3 agents, 8 turns, 50 runs/day, 4k input and 500 output per call:
#   3 * 8 * 50 * 30 = 36,000 calls a month, not 1,500.

Run that before you deploy a team rather than after. The two levers it exposes are both structural rather than pricing-related: fewer agents, and a termination condition that actually terminates. A team that runs to max_turns because the termination string never matched costs the full budget every single time.

Caching is off by default in v0.4

v0.2 cached by default through cache_seed. v0.4 does not, so re-running the same conversation bills again — invisible during prompt iteration, and visible only as an unexplained line in the monthly total.

from autogen_ext.models.cache import ChatCompletionCache
from autogen_ext.cache_store.diskcache import DiskCacheStore
from diskcache import Cache

# Caching is OFF by default in v0.4 (it was on by default in v0.2).
# A re-run of the same conversation is billed again unless you opt in.
cache_client = ChatCompletionCache(
    client, DiskCacheStore[CHAT_CACHE_VALUE_TYPE](Cache("/tmp/autogen-cache"))
)

What it costs

Rates checked 2026-09-26. Gateway rates move with upstream promotions — verify the current number in your dashboard before committing to a budget.

Rates checked 2026-09-26. Multiply these by agents × turns × runs before comparing — a per-million-token difference of a few dollars becomes the dominant term at thirty-six thousand calls a month.

AutoGen failures and what each one is actually telling you

When this is the wrong move

Cases where a multi-agent team is the wrong shape

Rolling it out without finding out the hard way

The failure mode you want to avoid is discovering a problem through a production bill or a customer-visible error. Four steps, in order:

1. Prove it on one two-agent team with one harmless tool and max_turns set. One call, from a script, with an explicit timeout and the model ID echoed back. You are testing reachability, authentication and model availability — three things that can fail independently.
2. Measure before you switch. Record tokens per task and cost per task on the current path first. Without that baseline, "it got cheaper" is an impression, not a result.
3. Move one workload, not everything. Pick the workload with the most predictable shape — batch jobs over interactive traffic. Leave the interactive path on the old configuration until the batch numbers are in.
4. Decide the rollback condition in advance. Write down what makes you revert (error rate above X, cost per task above Y, latency above Z) before you start, so the decision is not made under pressure.

Keep the base URL in configuration, never inline. That single choice is what makes step four take a minute instead of an afternoon.

FAQ

Get API access