Model router

Chooses how much a call costs, before it is made: which model it goes to, and how hard that model thinks.

Slot: resilience · Ships: off · Enable per agent in agent.toml

When you want it

Send short requests to a cheaper model, or have them think less, and keep the expensive work untouched. Or put a fixed share of real traffic on a new model to see how it does before switching.

If you only do one of these, do the effort one. Lowering reasoning effort does not change the prompt, so the provider's cached prefix and the model's tool-calling behaviour both stay put, where switching models gives up both.

What it does

One signal drives two levers: whether the conversation's messages total small_max_chars or fewer. The system prompt is not counted - it is identical on every call, so counting it would push every request over any threshold worth setting.

Cheaper model. A small conversation goes to cheap_model. Once it grows past the threshold it moves to the agent's own model and stays there for the rest of the conversation, even if compaction later shrinks it back under the line.

Less thinking. A small request uses low_effort instead of the effort the agent configured. This one is decided per call rather than per conversation, because effort is a request parameter rather than part of the prompt, so changing it between turns costs no cached prefix.

low_effort only ever replaces an effort that is already there. A model that cannot reason arrives with none, and the router will not introduce one, because the provider would reject the parameter.

Canary. A share of conversations goes to canary_model instead. Which ones is derived from the session id, so every turn of a conversation routes alike. The canary outranks the sizing rule: a conversation picked for it is measured on that model whatever its size, or the sample skews toward large prompts.

Both decisions are per conversation. Switching models forfeits the provider's cached prompt prefix and slows the first token, so a conversation that flipped models between turns would pay that on every flip. Deciding once and sticking to it means at most one such boundary per conversation.

Both are decisions made before the call. Recovering from a call that already failed by trying a different model is not expressible here - see Limits.

Model names are checked at boot against the models your provider actually serves, so a typo fails startup rather than every routed call.

Turn it on

[middleware.model_router]
enabled = true
small_max_chars = 2000
low_effort = "low"                          # small requests think less
cheap_model = "claude-haiku-4-5-20251001"   # and go to a cheaper model

Either lever works on its own. Effort alone is the lower-risk starting point:

[middleware.model_router]
enabled = true
small_max_chars = 2000
low_effort = "low"

Or to canary a new model on a tenth of traffic:

[middleware.model_router]
enabled = true
canary_model = "claude-sonnet-5"
canary_percent = 10

Settings

A config that would route nothing - none of the three set, a zero small_max_chars alongside cheap_model or low_effort, or a zero share alongside canary_model - is rejected at boot rather than sitting in the stack doing nothing. A typo in any key fails the boot rather than being silently ignored.

Limits

The set of conversations that have outgrown cheap_model is bounded at 10,000 per instance, oldest evicted first, so a long-lived process cannot grow one entry per session forever. A conversation that falls out of it and comes back small routes as cheap again, costing one cached-prefix boundary on a conversation nobody has touched in a long time.

Model fallback - retrying a failed call on a different model - is not available. The loop's terminal emits the run's terminal error event to the client before it returns the failure, so a middleware that retried after seeing it would produce content behind an error the client was already shown.

Check it is loaded

axl weave print --config axl-config/agents/<your-agent>/agent.toml

model_router appears in the resilience band of the printed stack.

Next

• LLM response cache - pairs with routing to cut spend further.
• Choose a model - what the agent asks for before routing applies.
• All middleware - the full stack and slot ordering.