# Run a model evaluation

Choose a benchmark and a model, review a quote, and start the run. An optional spending cap limits what it may charge; without one, your workspace credit bounds it. A model provider you connect may bill inference separately.

For a free introduction, open [onboarding](/onboarding). Its recorded comparison consumes no credits. You need workspace credit to start real work.

## Install and sign in

Install [uv](https://docs.astral.sh/uv/getting-started/installation/), then install the CLI:

```sh
uv tool install evalrouter
evalrouter --help
```

uv sets up the CLI and a compatible Python. If your shell cannot find `evalrouter`, run `uv tool update-shell` and restart your terminal. Update later with `evalrouter upgrade`.

Sign in through your browser:

```sh
evalrouter login
evalrouter whoami
```

The CLI selects a workspace after sign-in. See [CLI sign-in](/developers/docs/cli-login) for remote terminals and CI. Installing the package does not add credit. A new account gets one $5 welcome credit in its workspace.

## Run in one step

```sh
evalrouter run gpqa-diamond --model gpt-4o-mini --sample 25
```

The CLI quotes the run, shows its maximum charge, asks you to confirm, starts it and follows it to the results. In a script, pass `--yes`; without it nothing is quoted or charged (exit status 5). Add `--max-cost USD` to cap the run: it starts only if the quote's maximum charge fits under that cap.

## Quote, then start

Find benchmark profile and model IDs:

```sh
evalrouter benchmarks list --json
evalrouter models list --json
```

Save a quote request as `quote.json`, replacing the example IDs with catalog values. To cap the run, add `max_charge_microusd` in micro-USD (`"2000000"` is $2); without it the run is bounded by your workspace credit:

```json
{
  "model": {"kind": "managed", "route_id": "MODEL_ROUTE_ID"},
  "selection": {"profile_ids": ["BENCHMARK_PROFILE_ID"]},
  "coverage": {"mode": "sample", "sample_count": 3, "seed": 42}
}
```

Review the quote, then start it and wait for the result:

```sh
evalrouter quotes create --body quote.json --json
evalrouter runs create --quote QUOTE_ID --yes --wait --json
evalrouter runs results RUN_ID
evalrouter runs export RUN_ID --file-format html,json
```

A quote checks compatibility, coverage and the maximum charge without making model calls. Starting the same quote again returns the same run, so retrying after a dropped connection never starts a second one. See [CLI commands](/developers/docs/cli) for every command, output formats and exit statuses.

## Use HTTP

Create a [workspace API key](/developers/docs/authentication) with `eval:read` and `eval:write`. Send the same body to `POST /v1/quotes`, then start the returned quote with `POST /v1/runs`, restating its cap. Follow `GET /v1/runs/{run_id}` and read `GET /v1/runs/{run_id}/results`. The [public API schema](/developers/docs/openapi.json) lists all fields.