Home / Guides / Model fallback

Model fallback: the difference between a retry and a chain

A retry sends the same request to the same model again. A fallback sends it somewhere else. They solve different problems, and mixing them up is why some "highly available" setups go down anyway.

Retries help with transient failures, a blip, a momentary refusal. Fallback helps when the destination itself is unavailable: a model under sustained rate limiting, a provider having an incident, an endpoint that has started refusing a model you depend on. Retrying the second kind just generates more failures.

You need a key before the code below runs. Create an account, generate a key, and copy the base URL (https://aicomp.ai/v1). Create one free →
Check current rates → Free to sign up · $1 minimum top-up · No prepayment

What a chain is made of

A fallback chain is an ordered list of models to try, with rules for when to move down it. Three pieces:

  1. The order. Which model is tried first, second, third.
  2. The trigger. Which failures advance the chain and which do not.
  3. The shape check. Whether the replacement is allowed to behave differently.

Most implementations get the first two right and skip the third, which is where the quiet damage happens.

Order by capability first, price second

The instinct is to order cheapest-first and fall back to expensive. That inverts the point.

Fallback exists to preserve capability under failure. If your primary is a frontier model because the task needs one, a cheap backup that cannot do the task has not preserved anything It has converted a visible failure into a wrong answer. A wrong answer is more expensive than an error, because you often act on it.

So: pick the primary for the capability the task needs, then pick backups that can do the same job, and only then prefer the cheaper of those. Price is the tiebreaker between candidates that can both do the work, not the ordering principle.

Don't have an account yet? You need an API key from the gateway before the steps below will work — how the discount works and what you take on. Create one free → (no prepay, $1 minimum top-up) 
# Order by capability, fall back within the capable set
CHAIN = ["claude-sonnet-5", "gpt-5", "deepseek-v4-pro"]

# Advance only on failures a different model would fix,
# and never on a bad request, that will fail everywhere.
ADVANCE_ON = {429, 500, 502, 503, 504}
STOP_ON    = {400, 401, 403, 404}   # your request is wrong; retrying is pointless

That second set matters more than people expect. A 400 or 404 means the request itself is malformed or the model does not exist. Sending it to three models produces three identical failures and three times the latency.

The trap: a backup that silently changes the output

Two failure modes are worse than the outage they were meant to cover.

Schema breakage. Your primary returns JSON matching a schema; the backup returns JSON with different field names, or prose wrapped around it. Nothing errors, so your parser fails somewhere downstream and the cause is not obvious.

Capability cliffs. The backup has a smaller context window, so long requests are refused. Or it does not support tool calling, so a tool-using agent stops mid-loop. Both surface as strange behaviour far from the fallback code.

Guard against both by validating the response, not just the status code:

def acceptable(resp):
    return resp.get("choices") and len(resp["choices"][0]["message"]["content"] or "") > 0

And record which model actually served each request. A chain that never logs its own decisions is unauditable You will not know you spent a week on the backup tier, or that quality dropped on Tuesday.

When a gateway is the chain

Doing this with direct vendor keys means one credential, one client, and one error-handling path per vendor. The fallback logic is the easy part; the plumbing is not.

With one endpoint, the chain is a list of model IDs in one config, against one bill, with one place that logs which model served what. That is the practical argument, not that fallback is impossible without it, but that per-vendor fallbacks are usually where the complexity lives.

Get API access

FAQ

FAQ

Is fallback the same as retrying?

No. A retry sends the same request to the same destination and helps with transient failures. Fallback sends it to a different destination and helps when that destination is unavailable. Retrying a sustained outage produces more failures, not fewer.

Should the cheapest model be the fallback?

Only if it can do the task. Fallback preserves capability under failure, so a backup that cannot perform the job converts an error into a wrong answer And wrong answers get acted on. Pick backups that are capable, then prefer the cheaper capable option.

Which errors should advance the chain?

Rate limits and server-side failures: 429, 500, 502, 503, 504. Client-side errors, 400, 401, 403, 404, mean the request itself is the problem, so the next model will reject it identically. Advancing on those triples latency and changes nothing.

How do I know if fallback is hurting quality?

Log which model served every request, and validate the response shape rather than only the status. A chain with no per-request attribution will run on a weaker model for days before anyone notices.

Can fallback hide an outage I should fix?

Yes, and that is the real risk. A chain that absorbs every failure removes the signal that something is wrong. Alert on fallback rate, not just on hard failures (sustained fallback is an incident, even when requests keep succeeding).

Related

Get API access