Local model setup

Operator guide for running an open-weight model on your own machine and pointing AXL at it. AXL has no Ollama-specific or MLX-specific code: local inference reaches it through the same generic OpenAI-compatible provider that serves DeepSeek, Qwen, and Z.ai. Anything that answers POST /v1/chat/completions with tool calls works, and no AXL rebuild is involved.

For the axl-side knobs see (canonical OPENAI_COMPAT_* reference); for picking a model per agent see Choose a model.

At a glance

There are four things, in order:

An inference server running on your machine that speaks the OpenAI chat-completions API and emits tool calls while streaming.
A model small enough to fit your memory at a workable quantization.
A [local] entry in axl-config/models.toml with the real served context window.
LLM_PROVIDER=openai-compat plus OPENAI_COMPAT_BASE_URL. No API key needed.

The one hard constraint

AXL's ReAct loop only ever calls chat_stream. There is no non-streaming path through the loop, so the server must emit tool calls inside the SSE stream, not only in unary responses.

This rules out Ollama's OpenAI-compatible endpoint. Ollama emits tool calls correctly on its native /api/chat, but drops them when streaming through /v1/, returning empty content with finish_reason: "stop". The common workaround is to send stream: false, which AXL cannot do. Ollama's native protocol is not one AXL speaks. Use llama.cpp or LM Studio instead.

Install an inference server

llama-server from llama.cpp is the recommended default on every platform: mature OpenAI-compatible endpoint, tool calls that survive streaming, GBNF grammars for structured output, and long prompts handled by micro-batching rather than a hard prefill ceiling.

macOS (Apple Silicon)

brew install llama.cpp

Metal support is built in. There is nothing else to configure.

Raise the GPU wired-memory cap before loading a large model. macOS caps it near 96 GB on a 128 GB machine, and a model that fits on paper will otherwise fail to load:

sudo sysctl iogpu.wired_limit_mb=122880   # 120 GB; leave 8-16 GB for the OS

This takes effect immediately and resets on every reboot. Add it to a login script if you want it to persist.

Windows

Download a prebuilt release from llama.cpp releases and pick the binary that matches your GPU: cuda for NVIDIA, hip for AMD, or vulkan as a vendor-neutral fallback. Unzip and run llama-server.exe from that directory.

Windows sizing is different from a Mac's. There is no unified memory: the model has to fit in discrete VRAM to run at full speed, and a typical 16-24 GB card is far below what the large models below need. Use -ngl to control how many layers go to the GPU, and expect a steep slowdown once layers spill to system RAM. On a 24 GB card, qwen3.6-27b at 4-bit (~18 GB) is the realistic ceiling for a fully-resident model.

WSL2 also works and behaves like the Linux instructions, but it takes a memory cut from the Windows host - allocate deliberately in .wslconfig.

Alternative: LM Studio

Cross-platform GUI (macOS, Windows, Linux) that downloads models and exposes an OpenAI-compatible server, including tool calls and JSON-schema structured output. Slower to script than llama-server. Start its local server, then treat its URL as OPENAI_COMPAT_BASE_URL.

A note on MLX

On Apple Silicon, MLX is roughly twice as fast on decode as GGUF. But plain mlx_lm.server has a broken prefix cache for hybrid attention architectures, which makes an agent loop reprocess the entire context every turn - a much slower step at long context. Prefer llama.cpp until you have measured otherwise on your own model.

Pick and start a model

Sizes below are for a single 128 GB unified-memory Mac. Scale down for a discrete GPU.

Model id	Params	Footprint	GGUF repo	Best for
`qwen3.5-122b-a10b`	122B MoE, 10B active	~77 GB at 4-bit	`unsloth/Qwen3.5-122B-A10B-GGUF`	Strongest local tool-calling that fits
`gpt-oss-120b`	117B MoE, 5.1B active	~63 GB at MXFP4	`unsloth/gpt-oss-120b-GGUF`	Cleanest tool calls of the set
`qwen3.6-35b-a3b`	35B MoE, 3B active	~23 GB at 4-bit	`unsloth/Qwen3.6-35B-A3B-GGUF`	Fast iteration
`qwen3.6-27b`	27B dense	~18 GB at 4-bit	`unsloth/Qwen3.6-27B-GGUF`	Best correctness per call at its size

Prefer a mixture-of-experts model with few active parameters. Decode speed is set by memory bandwidth, so a dense 120B model reads roughly five times more per token than a 122B MoE with 10B active and feels correspondingly slower.

Start with the smallest model that will do. It is an ~18 GB download rather than ~70 GB, and it proves the whole path end to end:

llama-server -hf unsloth/Qwen3.6-27B-GGUF:Q4_K_M \
    --alias qwen3.6-27b \
    -c 131072 --jinja --host 127.0.0.1 --port 8080

Then move up once that works. Large quants are published as multiple shards inside a quant-named subdirectory, so there is no single filename to pass to -m; -hf resolves the quant tag and fetches every shard:

sudo sysctl iogpu.wired_limit_mb=122880   # required at this size, see above

llama-server -hf unsloth/Qwen3.5-122B-A10B-GGUF:UD-Q4_K_XL \
    --alias qwen3.5-122b-a10b \
    -c 131072 --jinja --host 127.0.0.1 --port 8080

Prefer Unsloth's UD- dynamic quants where they exist. UD-Q4_K_XL keeps the quantization- sensitive layers at higher precision and costs about half a gigabyte over plain Q4_K_M (77.0 GB against 76.5 GB), so there is little reason to take the plain one.

At ~77 GB against a 120 GB wired limit, that leaves roughly 40 GB for the KV cache and overhead, which is comfortable at a 128k window. Models are cached under ~/.cache/llama.cpp, so the download happens once. Use -m instead of -hf only when you already have the file locally.

--jinja is required for tool calling. It applies the model's own chat template from the GGUF instead of a generic one. Without it, tool calls arrive as plain prose.
--alias sets the id the server reports, so it matches the models.toml entry. AXL sends the model id and llama.cpp serves whatever is loaded regardless, so this is not strictly required to work - but without it GET /v1/models disagrees with the catalog, which is confusing to read and makes scripts/local-model.sh warn.
-c sets the served context window. llama.cpp allocates only what you ask for, so this is the number that has to match context in your models.toml entry.

Confirm the server is up and reporting the id you expect:

curl -s http://127.0.0.1:8080/v1/models

Vision

The Qwen models above are natively multimodal, and their GGUF repos ship an mmproj file that -hf downloads alongside the weights. llama.cpp loads it automatically and logs loaded multimodal model. gpt-oss-120b is text-only.

Their [local] catalog rows carry vision = true to match. This has to be right in both places: AXL strips image content before sending when the catalog says a model cannot see, so an understated row silently drops images that the server would have handled. Add vision = true to any local entry whose server loads an mmproj.

Register the model in AXL

axl-config/models.toml ships a [local] section with the four models above. If you serve something else, add an entry keyed to the id the server reports:

[[local.models]]
id = "my-local-model"
name = "Local: My Model"
context = 131072      # the served window, matching -c
max_output = 32768
input_per_m = 0.0     # self-hosted; zero is valid and keeps cost accounting correct
output_per_m = 0.0
supports = ["tools", "temperature", "reasoning"]

Do not skip this step. When the configured model id is absent from the catalog, AXL synthesizes a stub assuming a 200k context and 4096 max output. If your real served window is smaller, context budgeting and compaction miscalculate long before the server ever rejects a request - the failure looks like bad model behavior, not a config error.

Turn it on in AXL

LLM_PROVIDER=openai-compat
OPENAI_COMPAT_BASE_URL=http://127.0.0.1:8080/v1   # /chat/completions is appended for you
OPENAI_COMPAT_MODEL=qwen3.5-122b-a10b             # must match a [local] id
OPENAI_COMPAT_PROVIDER_NAME=local                 # binds the [local] catalog section

OPENAI_COMPAT_API_KEY is deliberately absent. Local servers serve no credential, and leaving it unset sends no Authorization header at all rather than an empty bearer that some servers reject. Set it only if you put an authenticating proxy in front of your server.

OPENAI_COMPAT_PROVIDER_NAME must equal the models.toml section name, or none of the curated entries are found.

Using more than one local model

Every model in the [local] section is registered and routable, not just the default. Select one per agent:

# axl-config/agents/researcher/agent.toml
model = "qwen3.6-27b"

or per request, with "model": "qwen3.6-27b" in the run body. AXL routes by model id and sends that id to the server. No rebuild and no restart is needed to switch models - only to add a new models.toml entry.

What you cannot currently do is register two different local servers at once, for example llama.cpp on :8080 alongside LM Studio on :1234. There is a single set of OPENAI_COMPAT_* variables, so one AXL instance points at one endpoint. Serving several models from that one endpoint is fine.

Verify

scripts/local-model.sh

Boots a real axl-server against your already-running inference server and drives the tester-local-model agent through a streamed run that must produce a completed tool call. It fails loudly on the two failure modes that otherwise look like success: tool calls dropped mid-stream, and tool-call syntax arriving as assistant prose.

Knobs: LOCAL_SERVER_URL, LOCAL_MODEL_ID, LOCAL_MODEL_PORT, LOCAL_MODEL_AGENT, LOCAL_REDIS_URL.

The tester uses the blob tool set, which needs only the Redis that AXL already requires, so the check runs with no cloud credentials configured.

What AXL still needs from the network

Local inference removes the LLM dependency, not every dependency:

Redis is required. It is where sessions, tasks, and streams live.
RAG embeddings are already local by default: in-process fastembed ONNX, no API key. Nothing here changes that.
PII detection is already local: a quantized ONNX NER model vendored into the binary.
Voice STT and TTS are cloud-only (Deepgram, ElevenLabs, Cartesia) and have no base-URL override. Voice activity detection is local. A fully offline voice pipeline is not currently possible.
Web tools still call out. The web tool set self-disables without TAVILY_API_KEY, which leaves an agent whose only tools are web with nothing to call.

Troubleshooting

The model answers but never calls a tool. Check that --jinja is set, and confirm the agent actually has tools: a tool set that self-disables for a missing key registers nothing, and the run looks normal while testing nothing. scripts/local-model.sh distinguishes these.

Tool calls appear as text like <function=name>. Wrong tool-call parser for the model family. Qwen3.5 was trained on XML tool calls, not Hermes-style JSON; Qwen3.6 does not have this problem. With llama.cpp, --jinja normally picks the right template out of the GGUF.

Every turn is slow even on short prompts. The prompt cache is probably missing. Both llama.cpp and MLX match on a byte prefix, so a timestamp, version string, or session id near the start of the system prompt forces full reprocessing every turn. Keep the system prompt byte-stable and put volatile content at the end of the context.

Gibberish at long context. Quantized KV cache degrades these models. Use --cache-type-k bf16 --cache-type-v bf16. There is little to reclaim anyway: these hybrid architectures keep only a minority of layers in a full KV cache.

Model fails to load despite fitting in RAM (macOS). The wired-memory cap. See the iogpu.wired_limit_mb sysctl above, and remember it resets on reboot.

UnknownModel at boot. OPENAI_COMPAT_MODEL is not in the catalog section named by OPENAI_COMPAT_PROVIDER_NAME. The two must agree.