CLI and interactive workspace

uv tool install evalrouter
evalrouter login
evalrouter

Run the command alone for the interactive workspace, or add --help to any command for its flags. If your shell can't find the command, run uv tool update-shell and restart the terminal. The upgrade command installs new releases. Start with the quickstart.

Command shape#

evalrouter benchmarks list
evalrouter models list
evalrouter quotes create gsm8k --model gpt-5
evalrouter runs create --quote QUOTE_ID --yes --wait
evalrouter runs list
evalrouter runs get RUN_ID
evalrouter runs results RUN_ID
evalrouter runs export RUN_ID --file-format html,json

A command from an earlier release, such as status or wait, prints its replacement and exits 2. The command reference lists every resource command.

Start a run#

The run command quotes the run, shows the price, asks you to confirm, starts it and follows it:

evalrouter run gsm8k --model gpt-5
evalrouter run gsm8k --model gpt-5 --dry-run               # the quote, nothing is charged
evalrouter run gsm8k --model gpt-5 --yes --json
evalrouter run gsm8k --model gpt-5 --max-cost 5            # with a spending cap
evalrouter run --from RUN_ID                               # the same setup again, as a new run

• Names, not IDs. Benchmarks and models take a slug or an unambiguous name, and gsm8k@2 pins a version. Exact IDs work too. At a terminal, a missing or ambiguous name becomes a choice.
• --max-cost USD is the most the run may charge. The run starts only if the quote's maximum fits. Without it, your wallet balance bounds the run and the confirmation says so. Your API key's spend ceiling bounds everything.
• --dry-run prints the quote (maximum charge and plan), starts nothing and needs no confirmation.
• Idempotent. Repeating the same start after a lost connection returns the same run, never a second one.

To review the quote on its own, create it first with quotes create, then start it with run --quote QUOTE_ID --yes.

Several models in one run#

Repeat --model to run two to eight models on the same tasks in one run, under one spending cap or your wallet:

evalrouter run gsm8k --model gpt-5 --model claude-sonnet-5 --dry-run   # one quote for every model
evalrouter run gsm8k --model gpt-5 --model claude-sonnet-5 --benchmark gpqa-diamond
evalrouter runs compare RUN_ID          # rank, model, score, 95% range
evalrouter runs continue RUN_ID         # finish an interrupted run

The quote lists every model and, when the cap is split, each one's share. The reasoning setting applies to every model. The compare command ranks the models on the tasks every model finished:

gsm8k@6 · Accuracy · 1,319 tasks
  claude-sonnet-5-5 92.7% vs gpt-4.1-mini 84.2% on 1,319 tasks · difference 8.5 percentage points (6.9–10.1)
  1  claude-sonnet-5-5   92.7%  (91.3–94.0)
  2  gpt-4.1-mini        84.2%  (82.2–86.1)
  Difference  8.5 percentage points (6.9–10.1)

• The second line summarises the top two, adding "(N models)" when there are more.
• Each row is rank (1= for a tie), model, score and 95% range.
• The difference row (top model minus the other) appears with exactly two models.
• A run compared on fewer tasks than planned says "300 of 1,319 tasks".
• With --json, the output adds every pair.

A run stopped by its cap, your credit or a cancellation shows as interrupted, with the reason. The continue command quotes only its unfinished tasks, asks you to confirm and starts the run that completes it, reusing finished tasks at no charge. The original then shows as continued, and comparing it compares the completed run.

Evaluate an agent#

The --agent flag takes one value, and --model is the model the agent uses through the run's metered gateway:

evalrouter agents list                                   # ACP Registry agents and their integrity
evalrouter run gsm8k --model gpt-5 --agent opencode      # the pinned snapshot's version
evalrouter run gsm8k --model gpt-5 --agent opencode@1.18.34
evalrouter run gsm8k --model gpt-5 --agent "local:my-agent --acp"
evalrouter run gsm8k --model gpt-5 --agent remote:my-agent  # a remote a2a agent, by connection name
evalrouter run gsm8k --model gpt-5 --agent opencode --hosted  # hosted by EvalRouter (or --agent hosted:opencode)

Registry and local agents run on this machine: EvalRouter delivers each task, meters every model call and grades the submission. A remote agent calls the model through EvalRouter. The agent list comes from the ACP Registry snapshot bundled with the CLI, with no API call, and labels each agent's integrity:

The agent contract#

EvalRouter evaluates any agent that meets its agent contract:

On your machine, the agent starts with an empty private home directory. It gets only the locale, path, proxy and certificate variables from your environment. Your EvalRouter and provider keys don't reach it.

Some agents ask permission before they act, as Qwen Code and Claude Code do. The CLI then approves only calls to the evalrouter tools and reads or edits inside the task's workspace. The policy trusts what the agent reports, so it isn't an isolation boundary. OpenCode and Goose run their own shell and file tools without asking.

Agent status#

A status holds for the pinned version. EvalRouter checks a new version again before it lists that version. These statuses are separate from the integrity labels above, which say how an agent is published.

Check an agent yourself with evalrouter agents check AGENT. The check needs no account and makes no model calls. It sees only connections that use the proxy settings, so it reports checks-passed, never a contract status, and then exits 0. An interactive evalrouter run --agent runs the same check on a registry agent's first use. It stops the run only when the check sees a model bypass.

Only a model call that bypasses the run's gateway fails the contract. That is a call to a known model provider, such as api.openai.com, or an agent that ignores the gateway settings. Other network activity doesn't fail it. At startup, OpenCode installs its plugin package from registry.npmjs.org, and Goose downloads the public models.dev catalog. Neither carries the task or a credential.

Any other agent, including a local command, gets the gateway in the standard OpenAI environment variables, such as OPENAI_BASE_URL. The run plan warns that EvalRouter can't stop that agent from calling another provider. The run neither meters nor caps those calls.

Scripts and agents#

A flag wins over its environment variable, which wins over the profile. The options command lists them all. With JSON output, stdout carries exactly one JSON document. Progress goes to stderr, only on a terminal. An error is one object on stderr, with stdout empty:

{"error": {"code": "confirmation_required", "message": "…", "hint": "…", "retryable": false, "request_id": null, "details": null}}

JSON output, CI=true and known agent environments never prompt. Colour is off without a terminal, and with NO_COLOR or a dumb terminal.

The API serves clients from 0.5.0 on. An older CLI or SDK gets HTTP 426 on every command. Nothing runs, and EvalRouter charges nothing. The CLI prints the upgrade command for its installation. An interactive CLI offers to upgrade when a newer release exists. The background check never installs anything or delays startup. Scripts, JSON output and pinned installations skip it, and EVALROUTER_NO_UPDATE_CHECK=1 turns it off.

Exit status#

Lists and pages#

Paging

Output

Browse interactively

• benchmarks list -i
• models list -i
• runs list -i

Cache

Public catalog reads may reuse a recent local copy when the server allows it, and say so. Quotes always check live availability and prices.

Follow, read and export a run#

evalrouter runs wait RUN_ID --progress auto
evalrouter runs events list RUN_ID --follow
evalrouter runs results RUN_ID
evalrouter runs export RUN_ID --file-format html,json --open
evalrouter runs cancel RUN_ID
evalrouter open RUN_ID

Reasoning and output format#

Reasoning defaults belong to the exact model route. The --reasoning flag sets an explicit effort. Without it the route's default applies, and the quote shows the effective choice. The API refuses a route that can't run a requested effort, before any charge.

Benchmark versions that grade a JSON answer may offer a choice of output format with --output-format. Prompts and grading are the same in every format. Results record the format, and scores from different formats are never ranked together. The API refuses a route that can't run the chosen format. Neither setting can change on an existing quote.

Connections and remote agents#

The connections create command registers your own model endpoint. The CLI never takes a key as an argument. At a terminal it asks with hidden input, and scripts pass the key on stdin or name an environment variable.

printf '%s' "$PROVIDER_KEY" | evalrouter connections create --name NAME --endpoint URL --model MODEL --key-stdin
evalrouter connections check CONNECTION_ID

Use a checked connection as a quote's model, or as --model connection:ID on run. Your provider may bill model or agent requests separately.

A provider account holds one key and base URL for several models. Add it once, then attach each model to it:

evalrouter providers add --name munito --endpoint URL --key-ref op://vault/item/key
evalrouter connections create --name qwen --provider munito --model MODEL

Disabling the account with providers disable NAME stops every model on it.

A remote agent is a connection with --protocol a2a. Its credentials come from the environment or stdin, never the command line:

evalrouter connections create --protocol a2a --name my-agent --endpoint https://agent.example.com \
  --header-env Authorization=AGENT_TOKEN
evalrouter run BENCHMARK --model MODEL --agent remote:my-agent --yes

API keys and workspaces#

evalrouter api-keys create --label ci --scope eval:read --scope eval:write --spend-ceiling 20
evalrouter api-keys update KEY_ID --spend-ceiling none
evalrouter api-keys roll KEY_ID
evalrouter workspaces switch NAME_OR_ID

A key's spend ceiling bounds what that key can spend in total, and none removes it. Switching workspaces registers this device for another workspace without a browser.

Preview commands#

These commands are preview. evalrouter --help marks them. They sit outside the stability promise, and any release may change or remove them. Scripts should rely only on the commands in the reference below.

• recommend, releases, traces
• datasets, tune

Command reference#

Every resource command and what it does. Run any of them with --help for its flags.