Run a model evaluation

The flow has five steps:

• Choose a benchmark and a model.
• Review the quote's maximum charge.
• Accept the quote. EvalRouter reserves that credit.
• Wait while the tasks run and EvalRouter grades them.
• Read the results and exports.

A model provider you connect may bill inference separately.

Install and sign in#

1. Install uv, then the CLI:

   Terminal

   uv tool install evalrouter

   If your shell can't find the command, run uv tool update-shell and restart the terminal. Upgrade later with evalrouter upgrade.

2. Sign in through your browser. The CLI picks a workspace after sign-in.

   Terminal

   evalrouter login
   evalrouter whoami

   For remote terminals and CI, see CLI sign-in.

Run in one step#

evalrouter run gpqa-diamond --model gpt-4o-mini --sample 25

The CLI quotes the run, shows its maximum charge, asks you to confirm, then follows it to the results. Add --max-cost USD to start only if the quote fits under that cap.

Quote, then start#

1. Find benchmark profile and model IDs:

   Terminal

   evalrouter benchmarks list --json
   evalrouter models list --json

2. Save a quote request as quote.json with your IDs. To cap the run, add max_charge_microusd in micro-USD (2,000,000 is $2). Without it, workspace credit bounds the run.

   JSON

   {
     "model": {"kind": "managed", "route_id": "MODEL_ROUTE_ID"},
     "selection": {"profile_ids": ["BENCHMARK_PROFILE_ID"]},
     "coverage": {"mode": "sample", "sample_count": 3, "seed": 42}
   }

3. Review the quote, start it and read the result:

   Terminal

   evalrouter quotes create --body quote.json --json
   evalrouter runs create --quote QUOTE_ID --yes --wait --json
   evalrouter runs results RUN_ID
   evalrouter runs export RUN_ID --file-format html,json

A quote checks compatibility, coverage and the maximum charge without calling a model. Starting the same quote again returns the same run, so a retry never starts a second one. CLI commands lists every command and exit status.

Use HTTP#

Create a workspace API key with the eval:read and eval:write scopes, then:

1. Send the same body to the quotes endpoint.

2. Start the quote with POST/v1/runs, restating its cap.

3. Follow the run with GET/v1/runs/{run_id}, then read its results.

The API schema lists every field.