# EvalRouter for agents EvalRouter runs model benchmarks from a terminal or over HTTP. Choose a benchmark and model, run the evaluation, and export versioned results. An optional spending cap limits EvalRouter charges, including its platform fee; without one, the workspace's credit bounds the run. A connected model provider bills inference separately. Rules for agents: 1. The human's request to run an evaluation is enough to start it: no separate approval and no spending cap are needed. Show them the quoted price. When they ask what a run would cost, quote it with `--dry-run`, which starts nothing. Pass `--max-cost` only with a cap the human gave; never invent one. 2. Never print, log, commit or pass a credential as a command argument. 3. Use only IDs returned by this workspace's catalog and connections. Do not invent them. 4. Report a sample as a sample, and partial, failed or cancelled runs as what they are. The full agent workflow is the skill at https://evalrouter.ai/skill.md. This page is the short version. ## Install ```sh uv tool install --python 3.12 evalrouter # recommended: the evalrouter command in its own environment pip install evalrouter # or into an existing Python 3.12/3.13 environment evalrouter --help ``` ## Authenticate A human creates the account and workspace key in the browser: https://evalrouter.ai/register, then https://evalrouter.ai/settings/api-keys. The CLI never asks for a password. ```sh export EVALROUTER_API_KEY=... # set privately, from a secret manager export EVALROUTER_WORKSPACE_ID=... export EVALROUTER_BASE_URL='https://api.evalrouter.ai' # no /v1 suffix evalrouter whoami ``` Interactive alternatives: `evalrouter login` (browser device sign-in, `--no-browser` on a remote machine) or `evalrouter login --with-api-key` (stores a key read from stdin or a hidden prompt). `EVALROUTER_API_KEY` always takes precedence. Check that a key is set without printing it: `test -n "$EVALROUTER_API_KEY" && echo set`. ## Core loop ```sh evalrouter benchmarks list --json # benchmarks evalrouter benchmarks get gpqa-diamond --json # one benchmark's profiles, versions and metrics evalrouter models list --json # model routes evalrouter run gpqa-diamond --model gpt-4o-mini --sample 25 --dry-run --json # the quote; nothing starts evalrouter run gpqa-diamond --model gpt-4o-mini --sample 25 --yes --json # the run the human asked for evalrouter run gpqa-diamond --model gpt-4o-mini --sample 25 --max-cost 5 --yes --json # only with a cap the human gave evalrouter runs list --json # recent runs evalrouter runs get RUN_ID --json # one run's status and progress evalrouter runs wait RUN_ID --json # reattach to a run in progress evalrouter runs results RUN_ID --json # scores, coverage, errors, billing state evalrouter runs results RUN_ID --diff --json # compare with the previous run of this benchmark and model evalrouter runs export RUN_ID --file-format html,json # versioned reports; never overwrites without --overwrite ``` Without a terminal, with `--json` or `--no-input`, in CI and in known agent environments, `run` never prompts: it refuses a paid start without `--yes` and makes no paid request. `--max-cost` is optional: pass it only with a cap the human gave. Without it the run is bounded by the workspace credit. Coverage defaults to a 25-task sample; `--full` grades every task. Repeating the same command resumes the same run instead of charging again; `--new` starts a separate one and `--detach` returns right away. `RUN_ID` may be a unique prefix; it is always required. Use an idempotency key for safe retries of an uncertain request. See the [model-evaluation quickstart](/developers/docs/quickstart) for an HTTP example. Exit codes: 0 success, 1 the API refused the request or another error, 2 invalid command line or input (nothing was sent), 3 the run ended without completing (failed, partial or cancelled), 4 not signed in or the credential was rejected, 5 a confirmation or choice was needed and not given, or the quote exceeds `--max-cost` (nothing started), 6 the API could not be reached, 124 `--wait-timeout` passed. A local wait timeout does not stop server work; `evalrouter runs cancel RUN_ID` does, and only when the human intends it. ## Output - `--json` writes one JSON result to stdout and progress to stderr; an error is one `{"error": {...}}` object on stderr with stdout empty. Prefer it whenever you parse output. - List commands also take `--jq EXPR`, `--columns a,b`, `--sort COL`, `--filter KEY=VALUE` and `--limit N`. - `evalrouter runs export RUN_ID --file-format json,csv,html,bundle` chooses report formats; `--output` names the file. - `evalrouter run --output-format` (and `quotes create --output-format`) chooses how a model is asked for a JSON answer on benchmark versions that offer a choice: `default`, `leaderboard`, `prompt_json`, `native_json` or `json_schema`. Results record the format, and scores from different formats are never ranked together. - `--progress auto|plain|json|off` controls progress on stderr. `json` writes `evalrouter.progress.v1` records (phase, processed/planned, percent, errors, spend, elapsed time) on every change and every 30 seconds. - `evalrouter runs events list RUN_ID --follow` prints the run's events as JSON lines until it ends. Use `--type` to keep categories such as `sample` or `capacity`, and `--after SEQUENCE` to resume. ## Reporting progress to the human Start the run detached with the browser open, then check it in short steps, so no command outlives your tool timeout: ```sh evalrouter run gpqa-diamond --model gpt-4o-mini --sample 25 --yes --detach --open --json evalrouter runs get RUN_ID --json # repeat about once a minute until status is terminal evalrouter runs results RUN_ID --json # when it ends ``` The `run` result carries `report_url` and `opened`. `--open` opens the live report page in the person's browser on their machine; when `opened` is false (CI, SSH, no display), give them `report_url` instead. `evalrouter open RUN_ID --json` does the same later. Report from `progress` on the run record, not from guesses: "37/120 tasks, 2 errors, $0.41 spent, 4 minutes in". Say "waiting for capacity" when `phase` is `waiting`. Do not promise a finish time; task durations, capacity waits and retries make time-left estimates unreliable. A run you stop checking keeps running on the server. If you can read stderr as it streams, `evalrouter runs wait RUN_ID --json --progress json` gives the same record on every change instead. ## Your own model endpoint (BYOK) The provider key is never a command argument. In a terminal the CLI asks for it with hidden input; scripts pipe it with `--key-stdin` or name an environment variable with `--key-env NAME`. ```sh evalrouter connections create --name NAME --endpoint URL --model MODEL # hidden prompt printf '%s' "$PROVIDER_KEY" | evalrouter connections create --name NAME --endpoint URL --model MODEL --key-stdin evalrouter connections create --name NAME --endpoint URL --model MODEL --key-env PROVIDER_KEY evalrouter connections check CONNECTION_ID # may send one small, provider-billed request evalrouter run gpqa-diamond --model connection:CONNECTION_ID --dry-run --json evalrouter connections update CONNECTION_ID --rotate-key ``` Without a terminal and without either option, the CLI reads `PROVIDER_API_KEY` if it is set and otherwise stops with an error. Your provider bills model requests separately, outside EvalRouter's charges and any cap. ## Costs Money in the API is a string of integer micro-USD: `"1000000"` is $1. A quote records: - `max_charge_microusd`: the cap, when the human gave one. The run can never charge more. Without a cap, the workspace credit bounds the run. - `plan`: coverage, total tasks, profiles, concurrency, rates (`plan.model.pricing`, `plan.pricing`), the minimum cap when one applies (`plan.pricing.minimum_cap_microusd`) and warnings. - `expires_at`, `price_version`, `retention` and the quote `id`. EvalRouter does not publish cost forecasts: charges follow actual usage at the quoted rates and never exceed a cap that was set. The optional legacy field `estimated_charge_microusd` (a conservative worst case, not expected spend) is omitted while forecasts are unpublished; treat it as absent, never as zero, and do not present it. Registration gives no credit; paid runs need available workspace credit. ## Errors API errors share one envelope: ```json {"error": {"code": "validation_error", "message": "Check the request fields.", "details": [], "request_id": "..."}} ``` Branch on `error.code`, never on `message`. Every response carries an `X-Request-ID` header with the same `request_id`. On 429, honour `Retry-After`. For an invalid or expired key, check the workspace and key status instead of creating an evaluation to diagnose it. To report a bug or missing feature, use `evalrouter feedback` (`POST /v1/feedback`) with the `request_id`, the command shape and the `error.code`, and no credentials or workspace identifiers. `evalrouter feedback status FEEDBACK_ID` reads a report back. The skill's [defect-reporting reference](https://evalrouter.ai/skill/report-defects.md) lists what to include. ## Tune your repository's agent and open a pull request Tune and improvement projects must be enabled on the deployment. Connecting GitHub is a one-time browser step at https://evalrouter.ai/settings/integrations; `evalrouter improvements create` fails with `github_repository_unreadable` and prints that link when the GitHub App cannot read the repository. Their request to tune is enough to start the sweep. Pass `--max-cost` only with a cap the person gave; never invent one. Without `--max-cost` the sweep is paid from wallet credit as it runs and stops when credit runs out. ```sh evalrouter improvements create --repo owner/name --ref main --json evalrouter improvements prepare PROJECT_ID --json evalrouter tune start --agent-project PROJECT_ID --dataset-version VERSION_ID --json evalrouter tune quote SESSION_ID --json # --max-cost only with the person's cap evalrouter tune start-sweep SESSION_ID --json evalrouter tune status SESSION_ID --json # repeat until complete evalrouter tune result SESSION_ID --json evalrouter tune export SESSION_ID --patch tune.patch --json ``` The patch applies at `source_commit_sha` in the export. When the result keeps the current agent, `export --patch` fails with `tune_no_change`. Open the pull request yourself: ```sh git switch -c evalrouter/tune-SESSION_ID SOURCE_COMMIT_SHA git apply --index tune.patch git commit -m "Apply EvalRouter Tune result SESSION_ID" git push -u origin evalrouter/tune-SESSION_ID gh pr create --title "Apply EvalRouter Tune result" --body-file tune-summary.md # summary from tune result ``` See the [Tune guide](https://evalrouter.ai/developers/docs/tune) for details. ## Turn your app's traces into a benchmark ```sh evalrouter traces connect --name my-app --env-file .env.local # the ingest key is written once, mode 0600 evalrouter traces list --connection CONNECTION_ID evalrouter datasets create --name "my-app production" evalrouter datasets import-traces DATASET_ID --connection CONNECTION_ID --limit 200 evalrouter datasets approve DATASET_ID --all evalrouter datasets versions create DATASET_ID evalrouter datasets benchmark DATASET_ID 1 --grader match-production --judge-route JUDGE_ROUTE evalrouter run BENCHMARK_ID --model MODEL_A --full --yes evalrouter run BENCHMARK_ID --model MODEL_B --full --yes ``` See [Save traces to a dataset](https://evalrouter.ai/developers/docs/trace-datasets) for each step. ## More - Agent skill: https://evalrouter.ai/skill.md - Index for LLMs: https://evalrouter.ai/llms.txt (everything in one file: https://evalrouter.ai/llms-full.txt) - Public API schema: https://api.evalrouter.ai/openapi.json (also https://evalrouter.ai/.well-known/openapi.json) - Every docs page is also Markdown: add `.md`, for example https://evalrouter.ai/docs/cli.md ## Docs - [EvalRouter](https://evalrouter.ai/developers/docs/overview) ([Markdown](https://evalrouter.ai/docs/overview.md)): Run model benchmarks and inspect their results. - [Run a model evaluation](https://evalrouter.ai/developers/docs/quickstart) ([Markdown](https://evalrouter.ai/docs/quickstart.md)): Install, sign in, choose a benchmark and model, run and export. - [Account and workspace key](https://evalrouter.ai/developers/docs/authentication) ([Markdown](https://evalrouter.ai/docs/authentication.md)): Create an account and configure a scoped workspace key for the CLI. - [Sign in from the CLI](https://evalrouter.ai/developers/docs/cli-login) ([Markdown](https://evalrouter.ai/docs/cli-login.md)): Sign the CLI in with your EvalRouter account, switch workspaces and revoke devices. - [CLI commands](https://evalrouter.ai/developers/docs/cli) ([Markdown](https://evalrouter.ai/docs/cli.md)): Discover models and benchmarks, manage model connections, quote, run, monitor and export. - [Results and exports](https://evalrouter.ai/developers/docs/results) ([Markdown](https://evalrouter.ai/docs/results.md)): Read status, coverage and billing alongside native scores and versioned reports. - [Availability and limits](https://evalrouter.ai/developers/docs/availability) ([Markdown](https://evalrouter.ai/docs/availability.md)): Check runnable model/profile combinations and spending requirements. - [Model API reference](https://evalrouter.ai/developers/docs/model-api) ([Markdown](https://evalrouter.ai/docs/model-api.md)): Authentication, model discovery, quotes, bounded runs, status, results and exports. - [Bring your own benchmark](https://evalrouter.ai/developers/docs/bring-your-own-benchmark) ([Markdown](https://evalrouter.ai/docs/bring-your-own-benchmark.md)): Create a private benchmark with built-in scoring or request a maintained catalog addition. - [Benchmarks from Hugging Face datasets](https://evalrouter.ai/developers/docs/huggingface) ([Markdown](https://evalrouter.ai/docs/huggingface.md)): Pin a Hugging Face dataset to a commit; we check its hashes and license and keep our own copy. - [Report bugs and missing features](https://evalrouter.ai/developers/docs/feedback) ([Markdown](https://evalrouter.ai/docs/feedback.md)): Report a defect, missing capability or docs gap from the API or CLI, then check its outcome. - [Private benchmarks from your own data](https://evalrouter.ai/developers/docs/byob) ([Markdown](https://evalrouter.ai/docs/byob.md)): Upload a dataset or import from Hugging Face, define scoring, and run a benchmark private to your workspace. - [Models, environments, and compatibility](https://evalrouter.ai/developers/docs/selection) ([Markdown](https://evalrouter.ai/docs/selection.md)): Select exact admitted identities and inspect compatibility before spending. - [Quotes, budgets, and billing](https://evalrouter.ai/developers/docs/quotes-budgets) ([Markdown](https://evalrouter.ai/docs/quotes-budgets.md)): A quote bounds platform spending and makes external charges explicit. - [Import a workload](https://evalrouter.ai/developers/docs/workload-imports) ([Markdown](https://evalrouter.ai/docs/workload-imports.md)): Connect provider traces and datasets, review tasks, and freeze grading evidence. - [Tune models and agents](https://evalrouter.ai/developers/docs/tune) ([Markdown](https://evalrouter.ai/docs/tune.md)): Compare configurations on a reviewed workload and export the tested change. - [Save traces to a dataset](https://evalrouter.ai/developers/docs/trace-datasets) ([Markdown](https://evalrouter.ai/docs/trace-datasets.md)): Save retained traces as evidence, or turn recent traces into a private benchmark. - [Changelog](https://evalrouter.ai/developers/docs/changelog) ([Markdown](https://evalrouter.ai/docs/changelog.md)): Released app, CLI, and SDK updates, with explicit alpha and stable channels.