# TypeScript SDK

Run benchmarks and read results from a trusted Node.js 22+ server (ESM). The package is `@kimpton-ai/evalrouter`. It shares the API contract with the Python SDK. The command line is a separate Python package.

```sh
npm install @kimpton-ai/evalrouter
```

```typescript
import { EvalRouter } from "@kimpton-ai/evalrouter";

const client = new EvalRouter(); // EVALROUTER_API_KEY, optional EVALROUTER_BASE_URL and EVALROUTER_WORKSPACE_ID
const run = await client.runs.create({ benchmark: "gsm8k", model: "gpt-5", maxCost: 5, wait: true });
console.log(await run.score());
```

> [!WARNING]
> Keep credentials in server-side code. A browser application should call its own authenticated server rather than bundle a workspace key.

Pin the version you test against, and keep the API and SDK release identifiers with your lockfile.

## Run a benchmark

- `benchmark` and `model` take a slug or an unambiguous name (`"gsm8k@2"` pins a version). Exact IDs work too.
- `maxCost` is in USD and is the most the run may charge. If the quote is higher, EvalRouter refuses the run before it starts. Without it, your wallet balance bounds the run.
- `wait: true` resolves with the finished run. Without it, `await run.wait()` later. Repeating a create never starts a second run.
- `dryRun: true` resolves with the quote (maximum charge and plan) and starts nothing. Start it later with its `quoteId`. To build the request yourself, the quotes client takes the same body as the quotes endpoint.
- `subjects` runs two to eight models on the same tasks under one cap, or your wallet. Comparing the run ranks them on the tasks every model finished. Continuing an interrupted run prices only its unfinished tasks and starts the run that completes it.

Money fields in API records are strings of integer micro-USD. Don't convert them to floating-point dollars for accounting. UUIDs and timestamps are strings. Pages contain `data` and an opaque `next_cursor`. Keep the same filters when continuing.

## Evaluate an agent

```typescript
await client.runs.create({ benchmark: "gsm8k", model: "gpt-5", agent: "opencode" }); // ACP Registry agent
await client.runs.create({ benchmark: "gsm8k", model: "gpt-5", agent: "local:my-agent --acp" });
await client.runs.create({ benchmark: "gsm8k", model: "gpt-5", agent: "remote:my-a2a-agent" }); // a connection's name
```

Registry and local agents, and agents written in TypeScript (`defineAgent({ name, run })` from `@kimpton-ai/evalrouter/agents`), run on your machine. EvalRouter delivers each task and meters every model call through the run's gateway. It grades the submission with the benchmark's own grader. ACP process agents need the optional peer packages listed in the package README.

## Names, progress and reports

The SDK resolves the same names the command line accepts, without prompting:

```typescript
const bench = await client.benchmarks.resolveBenchmark("gpqa-diamond");
const model = await client.benchmarks.resolveModel("ADMITTED_MODEL_NAME", { benchmark: bench });
for await (const event of client.runs.follow(run.id)) console.log(event.sequence, event.type);
console.log(client.runs.reportUrl(run.id));
```

- Several matches throw an `EvalRouterError` coded `ambiguous_benchmark` or `ambiguous_model`. No match throws `benchmark_not_found` or `model_not_found`.
- Following a run is an async iterator of its events in order until the run ends. It only reads. Pass the last sequence as `after` to resume, and ignore event types you don't recognise.
- The report URL is the web report link, or null when the web address is unknown.
- Charts are drawn by the web app and the command line. The SDK returns their data and the report link.

## Requests, retries and upgrades

- Give each application job a persisted operation key and keep it across network retries and restarts. A user pressing submit again should recover the existing submission, not create a second key.
- Honor retry headers and stop local polling with the client's request controls. Stopping the local request doesn't prove the run stopped. Use the cancellation endpoint and keep checking the run's final status and cost state.
- A client release the API no longer serves gets HTTP 426 as `ClientUpgradeRequired`, whose `upgrade` is the command to run. The SDK warns once per process about a `Deprecation` or `Sunset` response header.
- For raw HTTP, use `fetch` with JSON bodies and the headers in [authentication](/developers/docs/authentication). Keep tokens out of URLs, telemetry and client-side error reports.

<details>
<summary>Stability tiers</summary>

Stable resources keep their behaviour within a major version. Preview resources ship in the same package but are outside that promise. Any release may change or remove them. Each preview method says so in its doc comment.

| Tier | Groups |
| --- | --- |
| Stable | `benchmarks` (with `uploads`, `imports`, `private`, `evals` and `environments`), `models`, `connections`, `quotes`, `runs` (with `events`, `attempts` and `samples`), `comparisons`, `result_groups`, `api_keys`, `workspaces`, `account`, `wallet`, `agents` (the ACP Registry agents you can run, read offline), `feedback` ([report bugs and missing features](/developers/docs/feedback)) |
| Preview | `recommendations`, `releases`, `suites`, `traces`, `trace_integrations`, `trace_discovery`, `datasets`, `schedules`, `tune`, `tune_monitors`, `improvements`, `remote_scorers`, `workload_evaluators`, and individually marked methods on `runs` |

</details>