Evaluate agent quality

Offline evaluations run a versioned dataset against an agent or model and aggregate scorer results. Use deterministic scorers for contractual behavior and judge-based scorers for qualities that require interpretation.

Keep evaluation cases small, explicit, and representative. Store the expected behavior and why the case exists, run the dataset before releasing prompt or model changes, and gate CI on a metric only after its baseline is stable.

RAG retrieval has its own axl rag eval workflow described in the RAG guide. See the interactive HTTP API for agent evaluation endpoints.

Monitor production quality

Online evaluations sample completed production runs and score them asynchronously with reference-free scorers. They do not add model or scorer work to the request path.

Enable them per agent:

[online_evals]
enabled = true
sample_rate = 0.1
daily_judge_budget_usd = 1.0

[[online_evals.scorers]]
type = "no_tool_error"

[[online_evals.scorers]]
type = "latency_at_most"
ms = 30000

Only top-level completed runs are sampled. Use trajectory, structural, cost, latency, or judge scorers that do not require an expected answer; reference-comparing scorers are rejected when the agent loads.

Judge scorers stop at the configured daily budget and are recorded as skipped. A skipped scorer is a run that went out ungraded, not one that passed.

Reading the scores

axl eval online summary support-bot            # pass rates over the last day
axl eval online summary support-bot --days 7   # a week
axl eval online scored support-bot             # the scored runs, newest first
axl eval online scored support-bot --scorer no_tool_error   # only where that scorer failed

summary reports each pass rate with a Wilson confidence interval, and the interval is the part to read. A 100% pass rate over 4 runs and one over 400 are the same number and completely different evidence; the interval is what separates them, so a rate whose interval still spans half the range is not yet telling you anything.

It also reports unscored runs separately from failing ones. They are different faults: a failing run was graded and found wanting, an unscored run was never graded at all - which is what an exhausted judge budget looks like from the outside.

The console's Evals board shows the same window, alongside judge spend against run cost.

Turning a failure into a test

A failed production run makes a regression case, because nobody had to invent it:

axl eval online promote <run-id> --dataset regressions --tag refunds

That reconstructs a scripted, deterministic case, so it replays in CI without a provider call. Its expected is empty - score it with trajectory or judge scorers, or add gold by hand.

See Online evaluation internals for storage, queue, and multi-instance behavior.

Next

• Observe runs - the traces these scores are computed over.
• Add RAG - axl rag eval for retrieval quality specifically.
• Verify configuration content - signed evaluation datasets.