EvalRouter v2 clients: several models in one run
evalrouter and evalrouter-sdk on PyPI · @kimpton-ai/evalrouter on npm
Changed
- The CLI, Python SDK and npm package 0.4.x no longer work with the API. Every request from them is refused with HTTP 426
client_upgrade_requiredand the upgrade command, and is never charged. Upgrade to 0.5.0 or later:uv tool upgrade evalrouter(orpipx upgrade evalrouter) for the CLI,pip install --upgrade evalrouterin your project's environment for the Python SDK,npm install @kimpton-ai/evalrouter@latestfor npm. - From 0.5.0 on, the CLI prints the exact upgrade command for your installation, and the SDKs raise
ClientUpgradeRequiredwith that command.
Added
- Run two to eight models on the same tasks in one run under one spending cap:
evalrouter run gsm8k --model A --model B, orclient.runs.create(benchmark=..., subjects=[...]). evalrouter runs compare RUN(run.compare()) shows each model's rank, score and 95% range on the tasks every model finished (1=for a tie), and the difference when there are two models.- A run stopped by its spending cap, your credit or a cancellation shows as interrupted.
evalrouter runs continue RUN(run.continue_(), npmrun.continue()) prices only its unfinished tasks and finishes it without charging the finished ones again. client.wallet.get()reads your credit balance andclient.agents.list()lists the ACP Registry agents you can run, asevalrouter creditsandevalrouter agents listdo.
Removed
- Preview SDK groups for development-only features:
campaign_templates,environment_drafts,protocols, and the agent attempt protocolclient_runs(agents on your machine run throughruns.create(agent=...)andevalrouter.agents).