CLI commands
We recommend uv to install the evalrouter command in its own Python environment:
uv tool install --python 3.12 kimpton-evalrouter-sdk
evalrouter --helpuv supplies Python 3.12 if needed. If your shell cannot find evalrouter, run uv tool update-shell and restart your terminal. Upgrade an existing installation with uv tool upgrade kimpton-evalrouter-sdk.
Create your account and key, then follow the model-evaluation quickstart.
| Command | Purpose |
|---|---|
catalog | List benchmarks; use --slug for a benchmark's profiles or --models for model routes. |
connections list/create/check/update/disable | Manage your own model endpoint connections. |
quote --config FILE | Review compatibility, coverage and a spending cap before running. |
quote --id ID | Retrieve a saved quote. |
run --quote ID --idempotency-key KEY | Submit the reviewed quote once. |
status | List runs, or inspect a supplied run ID. |
wait RUN_ID | Follow a run with a bounded local wait. |
cancel RUN_ID | Request cancellation of server work. |
results RUN_ID | Read a result and its version. |
export RUN_ID --format json --output FILE | Save a JSON, CSV or HTML report. |
Run a command with --help for its exact flags. JSON request input may use --config - for stdin. --json keeps machine-readable output on stdout and progress on stderr.
Progress
For wait and run --wait, choose --progress auto|plain|off. Auto shows elapsed time and status in an interactive terminal, and a percentage only when validated processed and planned counts are available. Plain, redirected stderr and --json use structured progress. Off suppresses progress. Processed percentage is not a score.
Connected models
Use evalrouter connections create --help to configure a supported model endpoint. Its credential is read through --key-env; never pass the credential itself as a flag. Test the connection before selecting its returned ID in a quote:
{"kind": "connection", "connection_id": "CHECKED_CONNECTION_ID"}Use that object as the quote's model. Your provider may bill model requests separately from EvalRouter's platform charge. Testing a connection may send a small model request.
Recovery and exit codes
Persist the request file, quote ID, operation key and run ID. Reuse the same quote and operation key after an uncertain submission. Reading status or restarting a local wait does not create another evaluation.
Exit status 0 means success, 1 an API/transport or failed-run error, 2 invalid input or a rejected request, and 4 a partial or cancelled waited run. Inspect the returned status and error code. A local wait timeout or interruption does not establish that server work stopped.