Workflow operations

How to compile, submit, run, observe, and operate workflows: the CLI, the HTTP API, run lifecycle, configuration, and failure modes. For the language itself, see the Language Guide.

All HTTP routes are under /api/v1 and require a Bearer token (a JWT or an axl_key_… API key). Successful JSON responses are wrapped in a standard envelope:

{ "status": 202, "msg": "Accepted", "data": { … } }

Your payload is always under data. Set $AXL_HOST and $TOKEN for the examples below.

The two ways to run a workflow

1. Ad-hoc - submit a .axg (or compiled JSON) directly; the engine creates a one-off run.
2. Registered - register a definition by name once, then start runs of it repeatedly by name.

Use ad-hoc while authoring; register definitions you run on a schedule or trigger from code.

CLI

The axl workflow subcommands cover the whole loop: author locally, then run and watch.

Author (local, no server)

axl workflow check   ./flow.axg                       # parse + validate; non-zero exit on error
axl workflow compile ./flow.axg --out flow.json       # lower to canonical JSON (no run)
axl workflow graph   ./flow.axg [--layout vertical|horizontal|auto]   # ASCII flow chart

• check runs the exact compiler + validator the server runs, so every language and graph error is caught before you submit. It cannot see your deployment's catalog, so agent keys, tool names and sub-workflow names are not checked here - see the CLI page for where each one does surface.
• compile emits the canonical WorkflowDefinition JSON (to stdout, or --out).
• graph compiles and draws the graph as an ASCII/Unicode flow chart - boxes per node (id (kind: detail)), arrows for depends_on, and labeled arms for branch guards. --layout defaults to vertical (boxes + arrows); horizontal collapses simple chains to one line.

All three are fully local (no server, no token). For example, a branch workflow renders as:

              [classify (agent: triager)]
                           │
                           ↓
           [_axg_branch_classify_3 (branch)]
          ┌────────────────└─────────────────┐
          │                                  │
"category == 'billing'"                    "else"
          ↓                                  ↓
  [billing (agent: ...)]           [general (agent: ...)]

Run and watch

axl workflow submit ./flow.axg --input '{"name":"Ada"}' [--name label]  # submit; prints the run id
axl workflow submit ./flow.axg --input '{"name":"Ada"}' --follow         # submit, then stream it live
axl workflow logs   <run_id> [--since <event_id>]                        # attach to a run and stream
axl workflow list   [--status running] [--limit 20] [--cursor <c>]       # your runs, newest first
axl workflow status <run_id>                                             # snapshot: status, output, nodes
axl workflow cancel <run_id>                                             # stop a run

• submit compiles locally, submits the source, and prints the run_id. Add --follow to hang off the run's event stream and print each transition (nodes starting/finishing, then the final output) until the run reaches a terminal state.
• logs attaches to any run by id and streams its events live. Attaching is read-only: Ctrl-C detaches and leaves the run running - it does not cancel (use cancel for that). A dropped connection reconnects from the last event; --since <event_id> resumes from a known point.
• list shows your runs newest-first; filter with --status, page with --limit / --cursor.
• status prints a one-shot snapshot (no streaming): run status, per-node state, and the output once complete.
• cancel stops a run at its next safe point; an already-finished run is a no-op.

These talk to the server, so they need a configured API host + token (axl config init, or AXL_API_URL / AXL_API_TOKEN). Add --output json to make list / status emit raw JSON and submit --follow / logs emit one JSON event per line (ndjson) for scripting.

Each CLI command maps to an HTTP endpoint documented below - submit → Submit .axg source, logs/--follow → Stream run events, list → List runs, status → Read a run, cancel → Cancel a run.

HTTP API

Submit .axg source

curl -sX POST "$AXL_HOST/api/v1/workflow/flow/source?name=hello&input=%7B%22name%22%3A%22Ada%22%7D" \
  -H "Authorization: Bearer $TOKEN" \
  --data-binary @hello.axg

• ?name= - optional run label.
• ?input= - URL-encoded JSON for the run input ($.input).
• Body - the raw .axg source.
• 202 → data: { run_id, status: "queued" }. A compile error returns 400 with data.diagnostics (the same line-pointed diagnostics the CLI prints).

Submit at most once

A retried submit is a second run doing the work again. Send an Idempotency-Key and the server replays the first response instead:

curl -sX POST "$AXL_HOST/api/v1/workflow/flow/source?name=onboard" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Idempotency-Key: merchant-M-2001" \
  --data-binary @onboarding.axg

Both calls return the same run_id, and the work happens once. The CLI takes the same key as a flag:

axl workflow submit onboarding.axg --idempotency-key merchant-M-2001

Choose a key that identifies the work, not the attempt - the merchant id above, an order number, an upstream event id. The rules:

"Identical" is the whole request - method, path, query string and body - not just the input. That is deliberate: a key is a promise that you are retrying this request, and a narrower fingerprint would let two genuinely different submits collapse into one. The practical consequence is that a retry has to be the same call, down to the ?name= label: retrying the curl above with axl workflow submit --idempotency-key merchant-M-2001 is a different request and is refused. Retry with the tool you submitted with.

Keys are opaque strings up to 255 characters, scoped per caller, and accepted on every mutating route under /api/v1 - not just workflow submits.

Submit compiled JSON

curl -sX POST "$AXL_HOST/api/v1/workflow/flow" \
  -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
  -d '{ "definition": { … }, "input": { … } }'

Compile / check without running

• POST /api/v1/workflow/compile - body is .axg; returns the canonical JSON.
• POST /api/v1/workflow/check - body is .axg; returns diagnostics only. Both are server-side equivalents of the CLI.

Read a run

curl -s "$AXL_HOST/api/v1/workflow/flow/$RUN_ID" -H "Authorization: Bearer $TOKEN"

Returns the run document: status, per-node state, and (when complete) output.

Stream run events (SSE)

curl -sN "$AXL_HOST/api/v1/workflow/flow/$RUN_ID/events" -H "Authorization: Bearer $TOKEN"

A live Server-Sent-Events stream of state transitions - prefer this over tight polling.

List runs (paginated)

curl -s "$AXL_HOST/api/v1/workflow/flow?limit=20" -H "Authorization: Bearer $TOKEN"

Returns { items: [...], next_cursor }. Pass ?cursor=<next_cursor> for the next page; a malformed cursor returns 400. next_cursor is omitted (absent, not null) on the last page.

Cancel a run

curl -sX POST "$AXL_HOST/api/v1/workflow/flow/$RUN_ID/cancel" -H "Authorization: Bearer $TOKEN"

Transitions the run to cancelled and reaps its child tasks.

Cancellation lands at the next safe point rather than instantly: a node already waiting on a model call finishes that call, and you are billed for it. Budget for one in-flight node's cost after a cancel.

Resume a suspended run

curl -sX POST "$AXL_HOST/api/v1/workflow/flow/$RUN_ID/resume" \
  -H "Authorization: Bearer $TOKEN" \
  -d '{ "node": "gate", "data": { "ok": true } }'

node is the suspend node's id; data becomes that node's output.

Registered definitions

# register (or update) a named definition
curl -sX POST "$AXL_HOST/api/v1/workflow/definitions" \
  -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
  -d '{ "name": "summarize_doc", "definition": { … } }'

# list / get / delete
curl -s        "$AXL_HOST/api/v1/workflow/definitions"               -H "Authorization: Bearer $TOKEN"
curl -s        "$AXL_HOST/api/v1/workflow/definitions/summarize_doc" -H "Authorization: Bearer $TOKEN"
curl -sX DELETE "$AXL_HOST/api/v1/workflow/definitions/summarize_doc" -H "Authorization: Bearer $TOKEN"

# start a run of a registered definition
curl -sX POST "$AXL_HOST/api/v1/workflow/definitions/summarize_doc/run" \
  -H "Authorization: Bearer $TOKEN" -d '{ "input": { "doc": "…" } }'

A workflow (sub-workflow) node references a registered definition by name; register the child before submitting a parent that calls it.

The full request/response schemas are in the server's OpenAPI document (every workflow route is annotated, including the { status, msg, data } envelope wrappers).

Run lifecycle

A run moves through these states (status in the run document):

completed, failed, and cancelled are terminal. Individual nodes carry finer states (pending, launching, waiting on a task, delaying, suspended, emitting, completed, failed, cancelled, skipped, retry-pending) visible in the run document's per-node view.

How a run advances ("pulses")

A background consumer advances each run in small, idempotent steps called pulses: read the run, launch whatever nodes are ready, collect finished nodes' outputs, persist, repeat. State is committed after every pulse with optimistic concurrency, so:

• A crash mid-run resumes from the last committed pulse - nothing is lost, nothing double-runs.
• Many server instances can share the same run safely; only one pulse wins each transition.
• Side-effecting steps (write tools, child launches, event emits) are guarded so a retry or restart doesn't fire them twice.

You don't manage any of this - submit, then poll or stream.

Configuration

Workflow-relevant server config (env vars; defaults shown):

A node's own timeout attribute overrides the default for that node. The floors are validated at startup - a value below the floor is rejected.

Deployment note: the engine's multi-key atomic operations assume a single-node (non-clustered) Redis/Valkey. Point it at a single primary (with replicas/failover as you like), not a sharded cluster. The server logs a startup warning if it detects a cluster URL.

Failure modes and what they look like

Retrying

Retries are declared in the workflow, not triggered from outside it. A node with retry <n> after <dur> re-runs itself on failure, n counting the first attempt, and every attempt is kept in the run document's attempts array so you can see the whole history rather than the last error. Route a failure somewhere useful with an on error edge, where {{ node.error }} gives the reason.

A run that has already reached failed is terminal: there is no reset or resume-from-failed-node. Submit it again, ideally with an Idempotency-Key so a retry that races with the original does not run the work twice. resume is for a suspended node awaiting an event, not for a failed one.

Reading a run you did not start

Runs are private to the principal that submitted them, so GET /workflow/flow/{run_id} and its /events stream return 403 to anyone else. A token carrying the admin role reads any run, which is how on-call inspects a run from an alert:

{ "sub": "axl:you@example.com", "roles": ["admin"] }

That grant is read-only - cancelling and resuming still belong to the run's owner.

The same role lists across owners, which is how you find a run you cannot name yet:

curl -s "$AXL/api/v1/workflow/flow?all_owners=true&status=failed" \
  -H "Authorization: Bearer $ADMIN_TOKEN"

axl workflow list --all-owners --status failed

Without all_owners the listing stays your own, so nothing widens by accident, and requesting it without the role returns 403. Each row carries its owner.

The cross-owner history is bounded like any owner's, at WORKFLOW_FLEET_HIST_MULTIPLIER times the per-owner cap (default 10x). Because it is trimmed fleet-wide, a busy tenant can age a quiet one's finished runs out of the window sooner than they would leave that tenant's own listing. Runs still in flight are never trimmed, so nothing running can disappear. Set the multiplier to 0 to switch the cross-owner listing off entirely, leaving admins able to read only a run they name.

See also

• Quickstart
• Language Guide - how AXG works.
• WORKFLOW_SYSTEM.md - engine internals (durability, multi-instance, pulse model).

Next

• Workflow examples - complete files to submit.
• Run recurring work - workflows on a schedule.
• Track asynchronous tasks - the task model underneath.