LLM cache

Answers a repeated model call from a stored turn instead of calling the provider.

Slot: resilience · Ships: off · Enable per agent in agent.toml

When you want it

Workloads that ask the same thing more than once: evaluation suites, scheduled jobs over unchanged input, a support agent fielding the same question from the same user. Each hit is a provider call you do not pay for and do not wait on.

What it does

The key covers the whole request - the model, every message, the system prompt, the tools, and the sampling levers - so a hit means the provider was asked exactly this before. Anything that would change the answer changes the key.

The scope is one agent and one user, deliberately not the session: the repeat worth serving is the same person asking again in a new conversation. No entry is ever visible to another user.

A served hit is reported as costing nothing, because it did. Charging the run's budget for a call that never happened would end long runs early.

A cached turn that requested tools still dispatches them on the replay. The tools really run; only the model call is skipped.

Turns carrying reasoning blocks, redacted thinking, or citations are never stored. Replay rebuilds a turn from its text and tool calls alone, so one of those would come back visibly poorer than the call it stands in for.

Entries live in the agent's process. They are not shared between instances, and they are gone on restart.

Turn it on

[middleware.llm_cache]
enabled = true

Settings

Key	Default	What it does
`ttl_secs`	`300`	How long an entry stays fresh.
`max_entries`	`256`	Cached turns before the oldest is evicted. Bounds the cache by count, not bytes: worst case is roughly `max_entries * max_entry_bytes`.
`max_entry_bytes`	`131072`	Largest turn that may be cached, counting its text and its tool calls.

A zero in any of these is rejected at boot rather than silently caching nothing. A typo in any key fails the boot rather than being silently ignored.

Check it is loaded

axl weave print --config axl-config/agents/<your-agent>/agent.toml

llm_cache appears in the resilience band of the printed stack.

Model router - runs before this one, so cached answers stay keyed to the model.
Tool result cache - the same idea for tool calls.
All middleware - the full stack and slot ordering.