# CLI and interactive workspace

Install the `evalrouter` command with [uv](https://docs.astral.sh/uv/getting-started/installation/):

```sh
uv tool install evalrouter
evalrouter
```

uv sets up the CLI and a compatible Python in its own environment. If your shell cannot find `evalrouter`, run `uv tool update-shell` and restart your terminal. Update later with `evalrouter upgrade` (or `uv tool upgrade evalrouter`).

The interactive workspace checks for newer releases in the background. The check never installs anything or delays startup; set `EVALROUTER_NO_UPDATE_CHECK=1` to turn it off. Scripts, `--json` and pinned installations skip it.

[Sign in with `evalrouter login`](/developers/docs/cli-login), then follow the [quickstart](/developers/docs/quickstart).

## Command shape

Every resource command is `evalrouter <noun> <verb>`, and each one maps to one API operation:

```sh
evalrouter benchmarks list
evalrouter models list
evalrouter quotes create gsm8k --model gpt-5
evalrouter runs create --quote QUOTE_ID --yes --wait
evalrouter runs list
evalrouter runs get RUN_ID
evalrouter runs results RUN_ID
evalrouter runs export RUN_ID --file-format html,json
```

A few shortcuts stand on their own: `run` (quote, confirm, start and follow in one step), `agents list`, `login`, `logout`, `whoami`, `open`, `upgrade`, `completion` and `api`. `evalrouter api PATH` sends one raw request (`-X METHOD`, `--body FILE`) for anything without its own command.

Run `evalrouter` alone for the interactive workspace, `evalrouter --help` for the commands by purpose, and any command with `--help` for its flags and examples. The [command reference](#command-reference) below lists every resource command. A command from an earlier release (`status`, `wait`, `catalog`, `quote`, `export`, `compare`, `resume` and so on) prints one line naming its replacement and exits 2.

## Start a run

`run` quotes the run, shows the price, asks you to confirm, starts the run and follows it:

```sh
evalrouter run gsm8k --model gpt-5
evalrouter run gsm8k --model gpt-5 --dry-run               # the quote; nothing is charged
evalrouter run gsm8k --model gpt-5 --yes --json
evalrouter run gsm8k --model gpt-5 --max-cost 5            # with a spending cap
evalrouter run --from RUN_ID            # the same setup again, as a new run
```

Benchmarks and models are names: a slug or an unambiguous name (`gsm8k@2` pins a version); exact IDs work too. At a terminal, a missing or ambiguous name is a choice.

`--max-cost USD` is the most the run may charge: the run starts only when the quote's maximum charge fits under it. It is optional; without it the run is bounded by your wallet balance, and the confirmation says so. Your API key's spend ceiling bounds everything. Without a terminal (scripts, CI and agents), a paid start needs `--yes`; without it nothing is quoted or charged and the command exits 5. `--dry-run` prints the quote (maximum charge and plan), starts nothing and needs no confirmation. Starting a quote is idempotent: repeating the same command after a dropped connection returns the same run, never a second one.

The two-step path keeps the quote reviewable on its own:

```sh
evalrouter quotes create gsm8k --model gpt-5 --json
evalrouter run --quote QUOTE_ID --yes --json
```

## Several models in one run

Repeat `--model` to run two to eight models on the same tasks in one run, under one spending cap or your wallet:

```sh
evalrouter run gsm8k --model gpt-5 --model claude-sonnet-5 --dry-run   # one quote for every model
evalrouter run gsm8k --model gpt-5 --model claude-sonnet-5 --benchmark gpqa-diamond
evalrouter runs compare RUN_ID          # rank, model, score, 95% range
evalrouter runs continue RUN_ID         # finish an interrupted run
```

The quote lists every model, and each one's share of the cap when the cap is split. `--reasoning` applies to every model. `runs compare` compares the models on the tasks every model finished:

```text
gsm8k@6 · Accuracy · 1,319 tasks
  claude-sonnet-5-5 92.7% vs gpt-4.1-mini 84.2% on 1,319 tasks · difference 8.5 percentage points (6.9–10.1)
  1  claude-sonnet-5-5   92.7%  (91.3–94.0)
  2  gpt-4.1-mini        84.2%  (82.2–86.1)
  Difference  8.5 percentage points (6.9–10.1)
```

The second line is the top two rows in one line ("(N models)" is added when there are more). Each row is rank (`1=` for a tie), model, score and 95% range. `Difference` (the top model minus the other) appears with exactly two models. A run compared on fewer tasks than planned says "300 of 1,319 tasks". `--json` adds every pair.

A run stopped by its spending cap, your credit or a cancellation shows as `interrupted` with the reason. `runs continue` quotes only its unfinished tasks, asks you to confirm and starts the run that completes it; finished tasks are reused at no charge. The original run then shows as `continued`, and `runs compare` on it compares the completed run.

## Evaluate an agent

`--agent` takes one value; `--model` is the model the agent uses through the run's metered gateway:

```sh
evalrouter agents list                                   # ACP Registry agents and their integrity
evalrouter run gsm8k --model gpt-5 --agent opencode      # the pinned snapshot's version
evalrouter run gsm8k --model gpt-5 --agent opencode@1.18.34
evalrouter run gsm8k --model gpt-5 --agent "local:my-agent --acp"
evalrouter run gsm8k --model gpt-5 --agent remote:my-agent  # a remote a2a agent, by connection name; it calls the model through EvalRouter
evalrouter run gsm8k --model gpt-5 --agent opencode --hosted  # run hosted by EvalRouter (or --agent hosted:opencode)
```

Registry and local agents run on this machine: EvalRouter delivers each task, meters every model call and grades the submission. `agents list` reads the ACP Registry snapshot bundled with the CLI (no API call) and labels each agent `pinned` (a binary with a published sha256), `package-registry` (an exact npm or PyPI package) or `unverified-upstream`. An unverified-upstream agent runs only with `--allow-unverified`; its observed hash is recorded.

## Scripts and agents

| Setting | Effect |
| --- | --- |
| `--format human\|json\|ids` (`EVALROUTER_FORMAT`) | `json`: one JSON document on stdout. `ids`: one ID per line. |
| `--json` | Same as `--format json`. |
| `--no-input` | Never prompt; where a choice is needed, fail with exit status 5. |
| `--profile NAME` (`EVALROUTER_PROFILE`) | A named sign-in and API origin, kept separately (`evalrouter login --profile NAME`). |
| `--base-url`, `--workspace-id`, `--timeout` | API origin, workspace and request deadline for this command. |

A flag wins over its environment variable, which wins over the profile. `evalrouter options` lists them all; they go before or after the command.

With `--json`, stdout carries exactly one JSON document and nothing else. Progress goes to stderr, and only on a terminal. An error is one object on stderr, with stdout left empty:

```json
{"error": {"code": "confirmation_required", "message": "…", "hint": "…", "retryable": false, "request_id": null, "details": null}}
```

`--json`, `CI=true` and known agent environments imply `--no-input`. Colour is off without a terminal, with `NO_COLOR`, or with `TERM=dumb`.

### Upgrades

The API serves published clients from 0.5.0 on. An older CLI or SDK gets `client_upgrade_required` (HTTP 426) on every command, before anything runs or is charged; the CLI prints the upgrade command for how it was installed, such as `uv tool upgrade evalrouter`. When a newer release is available, an interactive CLI offers to upgrade before it runs the command.

### Exit status

| Status | Meaning |
| --- | --- |
| 0 | Success. |
| 1 | The API refused the request, or another error. |
| 2 | Invalid command line or input; nothing was sent. |
| 3 | The run ended without completing (failed, partial or cancelled). |
| 4 | Not signed in, or the credential was rejected. |
| 5 | A confirmation or choice was needed and not given, or the quote exceeds `--max-cost`; nothing started. |
| 6 | The API could not be reached. A run you were watching keeps going on the server. |
| 124 | `--wait-timeout` passed while waiting; the server work continues. |
| 130 | Interrupted (Ctrl-C). Submitted server work is not cancelled. |

### Lists and pages

List commands read up to `--limit` rows (default 50). `--all` reads every page, and `--cursor` continues after an earlier page. With `--json` the document includes `next_cursor`; it is `null` on the last page. In a terminal, lists print a table sized to the window; `--columns`, `--sort`, `--filter KEY=VALUE`, `--wide` and `--jq EXPR` shape it. `benchmarks list -i`, `models list -i` and `runs list -i` browse interactively.

Public catalog reads (`benchmarks list`, `benchmarks get`, `models list`) can reuse a recent local copy when the server allows it, and say so. `--refresh` reads now; `--offline` makes no request. Quotes always check current availability and prices.

## Follow, read and export a run

```sh
evalrouter runs wait RUN_ID --progress auto
evalrouter runs events list RUN_ID --follow
evalrouter runs results RUN_ID
evalrouter runs export RUN_ID --file-format html,json --open
evalrouter runs cancel RUN_ID
evalrouter open RUN_ID
```

`runs wait` retries temporary API failures with growing delays, and stops watching with exit status 6 after `--disconnect-timeout` (default 90 seconds) without an answer. `--progress json` writes one `evalrouter.progress.v1` record per line on stderr. `runs events list` prints one JSON event per line; `--follow` reads until the run ends and exits like `runs wait`.

Exports report the full path of every file saved. Without `--output`, files are named `evalrouter-<id>.<format>` in the current directory, and an existing name gets a `-2` suffix. `--file-format bundle` saves the evidence ZIP. `runs export RUN_ID --include-resumed` saves the chain result of a run and its completed successors.

`comparisons report RUN_A RUN_B --file-format html` downloads one descriptive comparison archive from saved results without running inference; `--evidence` also downloads each full export. `comparisons create` compares exact result versions under a release policy on the server. `result-groups create --name NAME --run RUN --run RUN` saves a group of existing runs for one collective report.

`open RUN_ID` opens the run's live report page and prints its address. It never opens a browser in CI, over SSH, without a display, or with `--no-browser`.

## Reasoning and output format

Reasoning defaults belong to the exact model route. `--reasoning` sets an explicit effort; without it the route's own default applies, and the quote shows the effective choice. A route that cannot run a requested effort is refused before any charge.

Benchmark versions that grade a JSON answer may offer `--output-format default|leaderboard|prompt_json|native_json|json_schema` on `run` and `quotes create`. Results record the format, and scores from different formats are never ranked together. Neither setting can change on an existing quote.

## Connections and remote agents

`connections create` registers your own model endpoint. The CLI never takes a key as an argument: at a terminal it asks with hidden input; scripts use `--key-stdin` or `--key-env NAME`.

```sh
printf '%s' "$PROVIDER_KEY" | evalrouter connections create --name NAME --endpoint URL --model MODEL --key-stdin
evalrouter connections check CONNECTION_ID
```

Use a checked connection as a quote's model, `{"kind": "connection", "connection_id": "…"}`, or `run BENCHMARK --model connection:ID`.

A remote agent is a connection with `--protocol a2a`. Its credentials come from the environment or stdin, never argv:

```sh
evalrouter connections create --protocol a2a --name my-agent --endpoint https://agent.example.com \
  --header-env Authorization=AGENT_TOKEN
evalrouter run BENCHMARK --model MODEL --agent remote:my-agent --yes
```

Headers use `--header-env NAME=ENVVAR` (repeatable, up to four) or `--header-stdin NAME`; OAuth2 client credentials use `--oauth2-token-url`, `--oauth2-client-id`, `--oauth2-client-secret-env ENVVAR` and optional `--oauth2-scope`/`--oauth2-audience`. `--card-path` names a non-default agent card. On `run` and `quotes create`, `--agent-parameters @file.json`, `--agent-gateway-api chat_completions` and `--agent-max-concurrent-tasks N` tune the agent. Your provider may bill model or agent requests separately.

## API keys and workspaces

```sh
evalrouter api-keys create --label ci --scope eval:read --scope eval:write --spend-ceiling 20
evalrouter api-keys update KEY_ID --spend-ceiling none
evalrouter api-keys roll KEY_ID
evalrouter workspaces switch NAME_OR_ID
```

A key's `--spend-ceiling USD` bounds what that key can spend in total; `none` removes it. `workspaces switch` registers this device for another workspace without a browser.

## Tab completion

Add `eval "$(evalrouter completion zsh)"` to `~/.zshrc` (after `compinit`) or `eval "$(evalrouter completion bash)"` to your Bash startup file, then open a new terminal. For fish, run `evalrouter completion fish > ~/.config/fish/completions/evalrouter.fish`.

Tab completes commands, flags and their choices, benchmark names (type `@` for versions), model names, ACP Registry agents for `--agent`, `remote:` connection names, profiles and recent run IDs. Commands, flags and agents come from the installed CLI; the other names come from a local cache, `~/.cache/evalrouter/complete.json`, kept per profile, API origin and workspace and cleared by `login`, `logout` and `workspaces switch`. A background process refreshes it with read-only list requests when it is more than five minutes old. Pressing Tab never waits for the API and works offline.

## Preview commands

`recommend`, `releases`, `traces`, `datasets`, `improvements` and `tune` are
preview: they are marked `(preview)` in `evalrouter --help`, are outside the
stability promise and may change or be removed in any release. Scripts should rely only on the
commands in the reference below.

## Command reference

<!-- cli-commands:start (generated from contracts/sdk-manifest.json) -->

| Command | What it does |
|---|---|
| `evalrouter api-keys create` | Create an API key; its secret is shown once. |
| `evalrouter api-keys delete KEY_ID` | Revoke an API key. |
| `evalrouter api-keys list` | List this workspace's API keys. |
| `evalrouter api-keys roll KEY_ID` | Replace an API key; the old one keeps working for the overlap. |
| `evalrouter api-keys update KEY_ID` | Change an API key's spend ceiling. |
| `evalrouter benchmarks environments get FAMILY` | Show one environment family. |
| `evalrouter benchmarks environments list` | List environment families. Pages with `--limit`, `--all` and `--cursor`. |
| `evalrouter benchmarks evals get REF` | Show one evaluation reference. |
| `evalrouter benchmarks evals list` | List evaluation references. Pages with `--limit`, `--all` and `--cursor`. |
| `evalrouter benchmarks get SLUG` | Show one benchmark and its versions. |
| `evalrouter benchmarks imports capabilities` | Show which imports this workspace can use. |
| `evalrouter benchmarks imports github` | Import a file from a GitHub repository as a private source. |
| `evalrouter benchmarks imports github-preview` | Preview the files of a GitHub repository revision. |
| `evalrouter benchmarks imports github-refs` | List the branches and tags of a GitHub repository. |
| `evalrouter benchmarks imports huggingface` | Import a Hugging Face dataset as a private source. |
| `evalrouter benchmarks list` | List maintained benchmarks. Pages with `--limit`, `--all` and `--cursor`. |
| `evalrouter benchmarks private archive PROFILE_ID` | Stop new runs of a private benchmark. |
| `evalrouter benchmarks private create` | Create a private benchmark from a data file and a small manifest. |
| `evalrouter benchmarks private get PROFILE_ID` | Show one private benchmark. |
| `evalrouter benchmarks private list` | List this workspace's private benchmarks. |
| `evalrouter benchmarks uploads archive UPLOAD_ID` | Archive an uploaded source to free a slot. |
| `evalrouter benchmarks uploads create` | Upload local data files as a source private to this workspace. |
| `evalrouter benchmarks uploads get UPLOAD_ID` | Show one uploaded source. |
| `evalrouter benchmarks uploads list` | List uploaded sources and the slots they hold. |
| `evalrouter comparisons create` | Compare exact result versions under a release policy. |
| `evalrouter comparisons get COMPARISON_ID` | Show a comparison. |
| `evalrouter comparisons list` | List comparisons. Pages with `--limit`, `--all` and `--cursor`. |
| `evalrouter connections check CONNECTION_ID` | Test a connection with one small request. |
| `evalrouter connections create` | Connect an OpenAI-compatible endpoint. |
| `evalrouter connections delete CONNECTION_ID` | Stop using a connection. |
| `evalrouter connections list` | List endpoint connections. Pages with `--limit`, `--all` and `--cursor`. |
| `evalrouter connections update CONNECTION_ID` | Change a connection or rotate its key. |
| `evalrouter datasets approve DATASET_ID` | Approve pending examples. |
| `evalrouter datasets benchmark DATASET_ID VERSION_ID` | Build a private benchmark from a dataset version. |
| `evalrouter datasets import-traces DATASET_ID` | Add recent production traces as held-out examples; running it confirms the workspace may evaluate on them. |
| `evalrouter datasets versions create DATASET_ID` | Freeze the approved examples as a new version. |
| `evalrouter datasets versions list DATASET_ID` | List a dataset's versions. |
| `evalrouter improvements get PROJECT_ID` | Show an agent project. |
| `evalrouter improvements list` | List agent projects. Pages with `--limit`, `--all` and `--cursor`. |
| `evalrouter improvements prepare PROJECT_ID` | Fetch the source and detect its runtime. |
| `evalrouter improvements reconcile-runtime-fees PROJECT_ID CYCLE_ID CANDIDATE_ID` | Owner or admin only: check completed work against an existing reservation; no model runs start. |
| `evalrouter models list` | List models and their prices. Pages with `--limit`, `--all` and `--cursor`. |
| `evalrouter quotes create` | Price a run and freeze its setup without starting it. |
| `evalrouter quotes get QUOTE_ID` | Show a quote. |
| `evalrouter result-groups create` | Group existing runs' results for reuse. |
| `evalrouter result-groups export GROUP_ID` | Save a result group export to a file. |
| `evalrouter result-groups get GROUP_ID` | Show a result group. |
| `evalrouter result-groups list` | List result groups. Pages with `--limit`, `--all` and `--cursor`. |
| `evalrouter runs attempts list RUN_ID` | List a run's attempts and their outcomes. Pages with `--limit`, `--all` and `--cursor`. |
| `evalrouter runs cancel RUN_ID` | Request cancellation of a run. |
| `evalrouter runs compare RUN_ID` | Compare a multi-model run's models on the tasks every model finished. |
| `evalrouter runs create` | Start a run from a quote under a spending cap (--max-cost). |
| `evalrouter runs events list RUN_ID` | Print a run's events as JSON lines. |
| `evalrouter runs export RUN_ID` | Save a run's versioned export to a file. |
| `evalrouter runs get RUN_ID` | Show one run. |
| `evalrouter runs list` | List recent runs. Pages with `--limit`, `--all` and `--cursor`. |
| `evalrouter runs receipt RUN_ID` | Show a run's frozen receipt. |
| `evalrouter runs results RUN_ID` | Show a run's scores, coverage and uncertainty. |
| `evalrouter runs samples RUN_ID` | List a run's per-task samples (--all pages through every one). Pages with `--limit`, `--all` and `--cursor`. |
| `evalrouter runs wait RUN_ID` | Wait until it ends; interrupting never cancels the server work. |
| `evalrouter traces connect` | Create a trace connection and print its ingest key once. |
| `evalrouter traces connections list` | List trace connections. |
| `evalrouter traces list` | List recent complete traces. Pages with `--limit`, `--all` and `--cursor`. |
| `evalrouter workspaces create` | Create a workspace. |
| `evalrouter workspaces delete WORKSPACE_ID` | Delete a workspace (confirm with its name). |
| `evalrouter workspaces get WORKSPACE_ID` | Show a workspace. |
| `evalrouter workspaces list` | List workspaces you belong to. |
| `evalrouter workspaces update WORKSPACE_ID` | Rename a workspace. |
<!-- cli-commands:end -->