Run a model evaluation

1. Install and create your account

Install uv, then install the CLI in its own Python environment:

uv tool install --python 3.12 kimpton-evalrouter-sdk
evalrouter --help

uv supplies Python 3.12 if needed. If your shell cannot find evalrouter, run uv tool update-shell and restart your terminal. To update later, run uv tool upgrade kimpton-evalrouter-sdk.

Create an EvalRouter account, verify your email, and open workspace API keys. Create a key for evaluation reads and writes. Keep its value in your secret manager or private environment; do not put it in a command argument, request file, or source control.

Set EVALROUTER_API_KEY to that key and EVALROUTER_WORKSPACE_ID to the selected workspace ID. Set the service origin without a /v1 suffix:

export EVALROUTER_BASE_URL='https://api.evalrouter.ai'

Account registration and package installation do not provide evaluation credit. Paid runs require sufficient available credit; a quote alone starts no paid work.

2. Choose a benchmark and model

evalrouter catalog --status active --json
evalrouter catalog --models --json
evalrouter catalog --slug BENCHMARK_SLUG --json

Replace BENCHMARK_SLUG with a slug returned by the catalog. Choose an exact profile ID whose quote_availability.status is ready_for_quote, and a compatible managed model route ID from those responses. Active status alone does not mean the profile is runnable. Do not copy illustrative identifiers as if they were available targets.

3. Review a quote

Save this as quote.json, replacing the two uppercase values. It requests three sampled tasks under a $1 platform cap; that cap does not guarantee that a particular evaluation fits.

{
  "model": {"kind": "managed", "route_id": "MODEL_ROUTE_ID"},
  "selection": {"profile_ids": ["BENCHMARK_PROFILE_ID"]},
  "coverage": {"mode": "sample", "sample_count": 3, "seed": 42},
  "max_charge_microusd": "1000000"
}

evalrouter quote --config quote.json --json > quote-response.json

Read the returned compatibility, coverage, warnings, cost components and expiry. Creating a quote does not start an evaluation. Resolve an incompatible model or insufficient cap before submitting.

4. Submit once and follow progress

Save the returned quote ID and choose a durable operation key for this submission. Replace the uppercase values below. This command starts work and can incur charges:

evalrouter run --quote REVIEWED_QUOTE_ID --idempotency-key SAVED_OPERATION_KEY --json

Save the returned run ID, then monitor it:

evalrouter status RUN_ID --json
evalrouter wait RUN_ID --wait-timeout 3600 --json

If the submission response is lost, retry the same quote and operation key. A new key can start separate work. Interrupting a local wait does not cancel the evaluation; use evalrouter cancel RUN_ID when you intend to request cancellation.

5. Read and export

evalrouter results RUN_ID --json
evalrouter export RUN_ID --format json --output result.json
evalrouter export RUN_ID --format csv --output result.csv
evalrouter export RUN_ID --format html --output result.html

Check terminal status, actual coverage, errors and billing alongside the score. A sample is not a full-benchmark score. See results and exports for versioned reports.