TypeScript SDK

npm install @kimpton-ai/evalrouter

import { EvalRouter } from "@kimpton-ai/evalrouter";

const client = new EvalRouter(); // EVALROUTER_API_KEY, optional EVALROUTER_BASE_URL and EVALROUTER_WORKSPACE_ID
const run = await client.runs.create({ benchmark: "gsm8k", model: "gpt-5", maxCost: 5, wait: true });
console.log(await run.score());

Pin the version you test against, and keep the API and SDK release identifiers with your lockfile.

Run a benchmark#

• benchmark and model take a slug or an unambiguous name ("gsm8k@2" pins a version). Exact IDs work too.
• maxCost is in USD and is the most the run may charge. If the quote is higher, EvalRouter refuses the run before it starts. Without it, your wallet balance bounds the run.
• wait: true resolves with the finished run. Without it, await run.wait() later. Repeating a create never starts a second run.
• dryRun: true resolves with the quote (maximum charge and plan) and starts nothing. Start it later with its quoteId. To build the request yourself, the quotes client takes the same body as the quotes endpoint.
• subjects runs two to eight models on the same tasks under one cap, or your wallet. Comparing the run ranks them on the tasks every model finished. Continuing an interrupted run prices only its unfinished tasks and starts the run that completes it.

Money fields in API records are strings of integer micro-USD. Don't convert them to floating-point dollars for accounting. UUIDs and timestamps are strings. Pages contain data and an opaque next_cursor. Keep the same filters when continuing.

Evaluate an agent#

await client.runs.create({ benchmark: "gsm8k", model: "gpt-5", agent: "opencode" }); // ACP Registry agent
await client.runs.create({ benchmark: "gsm8k", model: "gpt-5", agent: "local:my-agent --acp" });
await client.runs.create({ benchmark: "gsm8k", model: "gpt-5", agent: "remote:my-a2a-agent" }); // a connection's name

Registry and local agents, and agents written in TypeScript (defineAgent({ name, run }) from @kimpton-ai/evalrouter/agents), run on your machine. EvalRouter delivers each task and meters every model call through the run's gateway. It grades the submission with the benchmark's own grader. ACP process agents need the optional peer packages listed in the package README.

Names, progress and reports#

The SDK resolves the same names the command line accepts, without prompting:

const bench = await client.benchmarks.resolveBenchmark("gpqa-diamond");
const model = await client.benchmarks.resolveModel("ADMITTED_MODEL_NAME", { benchmark: bench });
for await (const event of client.runs.follow(run.id)) console.log(event.sequence, event.type);
console.log(client.runs.reportUrl(run.id));

• Several matches throw an EvalRouterError coded ambiguous_benchmark or ambiguous_model. No match throws benchmark_not_found or model_not_found.
• Following a run is an async iterator of its events in order until the run ends. It only reads. Pass the last sequence as after to resume, and ignore event types you don't recognise.
• The report URL is the web report link, or null when the web address is unknown.
• Charts are drawn by the web app and the command line. The SDK returns their data and the report link.

Requests, retries and upgrades#

• Give each application job a persisted operation key and keep it across network retries and restarts. A user pressing submit again should recover the existing submission, not create a second key.
• Honor retry headers and stop local polling with the client's request controls. Stopping the local request doesn't prove the run stopped. Use the cancellation endpoint and keep checking the run's final status and cost state.
• A client release the API no longer serves gets HTTP 426 as ClientUpgradeRequired, whose upgrade is the command to run. The SDK warns once per process about a Deprecation or Sunset response header.
• For raw HTTP, use fetch with JSON bodies and the headers in authentication. Keep tokens out of URLs, telemetry and client-side error reports.