# Python SDK

Run benchmarks, compare models and read results from Python 3.12 or 3.13. The SDK is the `evalrouter-sdk` package. It depends only on httpx and pydantic and has no terminal interface. The command line is a separate package that installs the same SDK version.

```bash
python -m pip install evalrouter-sdk
```

```python
from evalrouter import EvalRouter

client = EvalRouter()  # EVALROUTER_API_KEY and EVALROUTER_BASE_URL from the environment
run = client.runs.create(benchmark="gsm8k", model="gpt-5", max_cost=5, wait=True)
print(run.score)
```

The async client, `AsyncEvalRouter`, has the same calls, awaited. The client's benchmark and model lists read the catalog.

## Upgrade from 0.4

From 0.5.0 the SDK is `evalrouter-sdk` and the `evalrouter` package is the command line only. The API refuses 0.4.x clients with HTTP 426 and the upgrade command.

| Where the SDK is installed | Upgrade |
| --- | --- |
| An environment with pip | `pip install --upgrade evalrouter && pip install --force-reinstall --no-deps evalrouter-sdk` |
| A uv-created environment (no pip) | `uv pip install --upgrade evalrouter` |
| A uv project | `uv remove evalrouter && uv add evalrouter-sdk` |
| The CLI as a uv tool | `uv tool upgrade evalrouter` |

Then depend on `evalrouter-sdk` in your requirements. Upgrading only `evalrouter-sdk` in place leaves the old 0.4 command line behind, broken.

## Run a benchmark

- **Names, not IDs.** The benchmark and model take a slug or an unambiguous name, and `"gsm8k@2"` pins a version. Exact IDs work too.
- **`max_cost`** is in USD and is the most the run may charge. If the quote is higher, EvalRouter refuses the run before it starts. Without it, your wallet balance bounds the run.
- **`wait=True`** returns the finished run. Without it, wait later with `run.wait()`.
- **Idempotent.** Repeating a create never starts a second run.

## Several models in one run

```python
run = client.runs.create(benchmark="gsm8k", subjects=["gpt-5", "claude-sonnet-5"], wait=True)
ranking = run.compare()
```

The `subjects` list runs two to eight models on the same tasks under one spending cap, or your wallet. Each name resolves like a single model. The comparison ranks the models on the tasks every model finished:

| Field | Meaning |
| --- | --- |
| `profile_id`, `metric`, `tasks_compared` | What was compared |
| Per model | Rank (`1=` for a tie), score and 95% interval |
| `summary` | The top two as one line |
| `pairs` | Every pairwise difference |

A run that stopped early (cap reached, out of credit or cancelled) carries the reason in `run.interruption`. Continuing it with `run.continue_()` prices only its unfinished tasks and starts the run that completes it, reusing finished ones at no charge. A dry run returns that quote first.

## Quote, then start

Use a quote when a person or application reviews the cost first:

```python
quote = client.runs.create(benchmark="gsm8k", model="gpt-5", max_cost=5, dry_run=True)
# Review the quote's maximum charge and plan. Nothing has started.
run = client.runs.create(quote_id=quote.id, max_cost=5)
run = client.runs.wait(run.id, timeout=3600)
result = client.runs.results(run.id)
```

To build the request yourself, `client.quotes.create(body)` takes the same body as the quotes endpoint. Its `max_charge_microusd` is in micro-USD (1 USD = 1,000,000), so `"1000000"` is $1. The convenience parameters `max_cost` and `budget_usd` take USD. Waiting is polling. A local timeout raises `WaitTimeout` and never cancels the server work, so read the run, or cancel it explicitly, before starting another.

## Evaluate an agent on your machine

The agents module runs your agent locally. EvalRouter does four things for each task:

- It delivers the task.
- It meters every model call through the run's gateway (managed models only).
- It grades the submission with the benchmark's own grader.
- It records the trajectory.

The agent can be:

- an ACP Registry agent, `registry_agent("opencode")`
- a local ACP command, `local_agent("my-agent --acp")`, which needs the `evalrouter-sdk[agents]` extra
- an agent written in Python:

```python
from evalrouter import EvalRouter, agents


class Solver(agents.Agent):
    name = "my solver"

    async def run(self, context):
        task = await context.task.get()
        reply = await context.gateway.chat(messages=task["inputs"]["messages"])
        await context.task.submit(reply["choices"][0]["message"]["content"])
        await context.task.done()


client = EvalRouter()
evaluation = agents.quote(
    client,
    agent=Solver(),
    model="ADMITTED_ROUTE_ID",
    tasks=agents.benchmark("eval://PROVIDER/NAME/VERSION"),
    coverage={"mode": "sample", "sample_count": 3, "seed": 42},
    budget_usd="2",
)
# Review evaluation.quote before starting; nothing is charged until start().
run = evaluation.start()
for event in run.events():  # or: async for event in run.events_async()
    print(event.type, event.task_ref, event.failure_code or "")
print(run.wait()["status"], run.results())
```

One call, `agents.evaluate(...)`, does all of this. Its result reports whether the run succeeded. A failed, partial or abandoned run comes back marked unsuccessful, as a wait returns it, and is not raised. The CLI's agent runs use the same functions and exit 0 only for a completed run (1 when it failed, 4 otherwise).

<details>
<summary>Tasks, context and resuming</summary>

| Source of tasks | Call |
| --- | --- |
| A benchmark | `agents.benchmark(REF)` |
| An environment version | `agents.environment(REF, split="test")` |
| A private benchmark | `agents.private_benchmark(ID)` |
| A suite | A saved suite version |

| Context | What it does |
| --- | --- |
| `context.task.get()` | Read the task |
| `context.task.submit(...)` | Submit the answer text, or nothing for file and repository tasks, whose files or diff are collected from `context.workspace` |
| `context.task.done()` | Finish the task |
| `context.tools.list()`, `context.tools.call(...)` | List and call every task tool |
| `context.gateway` | OpenAI chat-compatible. `gateway.openai()` returns an `AsyncOpenAI` client when `openai` is installed. |

| Run control | What it does |
| --- | --- |
| Leaving the event loop early | Finishes the attempt in flight as an interrupted controller |
| `agents.AgentRun.resume(client, RUN_ID, agent)` | Continues the run with the same agent |
| `run.cancel()` | Stops the run |
| `run.costs()`, `run.export("csv")`, `run.trajectories()` | Read the finalized results |
| `agents.compare(client, [run_a, run_b])` | Compare finalized runs |

</details>

<details>
<summary>Agent traces</summary>

Each attempt's trace is one span tree: the agent, the model turns and the tool calls each turn requested. The SDK sends it while the attempt runs, about every half second. The SDK records gateway calls and task tool calls automatically.

| Add to the trace | Call |
| --- | --- |
| Your own spans | `with context.trace.span("plan"):` |
| Checkpoints | `context.trace.checkpoint("after planning")` |
| Test results | `context.trace.test_results(passed=41, failed=1, total=42)`, optionally split SWE-bench style with `groups={"fail_to_pass": {"passed": 2, "failed": 0}, ...}` |
| Command results | `context.trace.command_result(...)` |
| An OpenTelemetry framework (LangGraph or the OpenAI Agents SDK, through OpenInference, OpenLLMetry or the GenAI conventions) | `provider.add_span_processor(context.trace.otel_processor())` |
| LangGraph checkpoints | `context.trace.langgraph_checkpointer(saver)` |

Choose what the trace records with a `trace=` argument when quoting or evaluating. It takes a level and the kinds to include:

| Level | Records |
| --- | --- |
| `off` | Nothing |
| `metadata` | Agent spans with their counts and results only |
| `spans` (default) | The tree, metadata only |
| `content` | Also tool inputs and outputs, agent messages and file diffs, with secrets redacted and each field bounded to 8 KiB, in the SDK and again on the server |

By default the trace includes every kind:

- `model_turns`, `tool_calls`, `command_output`
- `file_diffs`, `agent_messages`

Where the platform turns content off, the API refuses a content-level quote with `client_trace_level_unavailable`. Add your own redaction patterns with `trace_redact=[r"acct-\d+"]` when executing. They never leave your machine. The CLI takes the same choices as trace level, include and redact flags.

</details>

## Names, progress and reports

The SDK resolves the same names the command line accepts, without prompting:

```python
bench = client.benchmarks.resolve_benchmark("gpqa-diamond")
model = client.benchmarks.resolve_model("ADMITTED_MODEL_NAME", benchmark=bench)
```

- The benchmark resolver returns the one quotable version for a slug, display name, `slug@version`, eval reference or private benchmark ID.
- The model resolver returns the quote's model body for a route ID, model name or connection, matching only models that benchmark version can run.
- Several matches raise `ambiguous_benchmark` or `ambiguous_model`, and the message lists them. Use the exact slug or route ID. No match raises `benchmark_not_found` or `model_not_found`. A model the benchmark refuses raises the API's refusal code.

Following a run yields its events in order until the run ends. It retries transient API failures and only reads, so stopping the loop never cancels the run. Pass `after=` the last sequence you processed to resume without gaps, and ignore event types you don't recognise.

```python
for event in client.runs.follow(run_id):
    print(event["sequence"], event["type"])
print(client.runs.report_url(run_id))
```

The report link comes from the `web_url=` argument, the `EVALROUTER_WEB_URL` variable or the API address, and is empty when unknown. Charts are drawn by the web app and the command line. The SDK returns the data behind them (results, comparisons and exports), so plot it with your own library if you need a picture in code.

<details>
<summary>Stability tiers</summary>

Stable resources keep their behaviour within a major version. Preview resources ship in the same package but are outside that promise. Any release may change or remove them. Each preview method says so in its docstring.

| Tier | Groups |
| --- | --- |
| Stable | `benchmarks` (with `uploads`, `imports`, `private`, `evals` and `environments`), `models`, `connections`, `quotes`, `runs` (with `events`, `attempts` and `samples`), `comparisons`, `result_groups`, `api_keys`, `workspaces`, `account`, `wallet`, `agents` (the ACP Registry agents you can run, read offline), `feedback` ([report bugs and missing features](/developers/docs/feedback)) |
| Preview | `recommendations`, `releases`, `suites`, `traces`, `trace_integrations`, `trace_discovery`, `datasets`, `schedules`, `tune`, `tune_monitors`, `improvements`, `remote_scorers`, `workload_evaluators`, and individually marked methods on `runs` |

</details>

## Errors and retries

Read an error's code and status, not its human-readable message. Every error derives from `EvalRouterError`.

| Error | When |
| --- | --- |
| `APIStatusError` subclasses | HTTP failures: `BadRequestError`, `AuthenticationError`, `PaymentRequiredError`, `PermissionDeniedError`, `NotFoundError`, `ConflictError`, `UnprocessableEntityError`, `RateLimitError`, `InternalServerError` |
| `APIConnectionError`, `APITimeoutError` | Transport failures |
| `ClientUpgradeRequired` | HTTP 426 for a release the API no longer serves. Its `upgrade` is the command to run. |

When a response marks an operation as deprecated (with a Deprecation or Sunset header), the SDK warns once per process. It retries supported reads and idempotent submission and cancellation requests. It never blindly repeats quote creation, connection verification or other side effects.

Use the preview traces group for capture settings and scoring. Use the types shipped with your pinned release and the [OpenAPI schema](/developers/docs/openapi.json) for exact payload limits. See [Save traces to a dataset](/developers/docs/trace-datasets) for capture.