# Models, environments, and compatibility

For automated release checks, the [CI integration guide](https://github.com/kimpton-ai/kimpton-evalrouter/tree/main/examples/ci)
includes a Python adapter and GitHub Actions workflow. It runs a pinned task suite
against candidate and baseline models, then fails CI when evidence is incomplete
or a configured score or regression threshold is violated.

Choose a versioned evaluation and a model that can run it. Check the pair before you spend.

`GET /v1/evals` lists evaluations from admitted providers. The public catalogs show native benchmarks and environments. Your workspace may also have private environments, which need workspace context to discover.

## Model routes and connections

A managed route states its model, capabilities, rates, and limits. Choose the exact catalog entry. A familiar model name alone does not tell you whether the route supports the tools, reasoning, or output format a task needs.

A workspace connection points to your provider endpoint, which bills you separately. Checking it makes a small provider request and may cost money outside EvalRouter.

## Task identity

Use `eval://provider/name/version` to name the exact evaluation. Native aliases such as `name@version` resolve through the registry. Save the resolved identity with reports; a display name cannot identify the version later.

The quote checks model capabilities, input and output limits, task availability, permissions, and resources. An unsupported pair fails before the run starts. Removing a requirement changes the evaluation protocol; it is not a compatibility fix.

## Coverage

A sample freezes its tasks and seed. A full run targets the admitted task collection. Report completed, missing, failed, and excluded tasks separately. Scores from different subsets, prompts, graders, or splits may not be comparable, even under the same benchmark name.

Use development results to iterate. Keep held-out tests separate. Remember that prior benchmark exposure or pretraining contamination can limit what a score means.

## Choose specific tasks

To choose tasks yourself, open **New evaluation → Benchmarks → Select tasks**. Search task IDs and metadata, check tasks for each source, then review the exact IDs before starting. Required dependencies are included when available. The saved setup and repeated runs retain exact task selection.

In the CLI, select **Specific tasks** during interactive `evalrouter run`, or repeat `--task` with IDs from that benchmark version:

```sh
evalrouter run BENCHMARK --model ROUTE_ID --task TASK_ID_1 --task TASK_ID_2 --dry-run
```

`--task` cannot be combined with `--sample` or `--full`. `GET /v1/catalog/tasks?source_ref=EXACT_REF` lists safe task metadata for an accessible source, with `q`, `limit`, and `offset` for search and pagination. The quote checks whether the selected tasks can execute with your model. An unavailable task selection does not fall back to sampling.

## Recommend tasks for an objective

Recommendations are a preview. Save a task selection with the CLI:

```bash
evalrouter recommend --objective "math and reasoning" --model ROUTE_ID --tasks 25 --json > recommendation.json
```

The API equivalent is `POST /v1/recommendations` with `objective`, `models`, and `task_limit`. Review its selections, uncovered capabilities, exclusions, and warnings. Recommendation itself starts no work and creates no quote. To run the selected tasks of one benchmark, pass their IDs to `evalrouter run` with `--task`, as above.

Tasks can come from multiple benchmarks and environments. Selection preserves their source versions and explicit task IDs. Declared task metadata determines relevance where available; source labels are a disclosed fallback. Required dependencies count toward the task limit. Results on a selected subset are not full benchmark scores.

Optional `--budget` is historical-cost selection guidance only. Cost evidence can be missing or based on a source mean. It is not a guaranteed price or execution spending cap. Launch uses available workspace credit and actual usage billing.

## Reusable exact-task suites

In New evaluation, choose tasks on the Benchmarks step and select **Save selection as suite**. A suite saves the exact source versions, splits and task IDs across benchmarks and environments. It does not save a model or start an evaluation. Open **Suites** to configure an evaluation from a saved version or create its next version. Earlier versions remain available.

The API accepts `task_selections` in `POST /v1/suites`. Each selection contains `source_ref`, `split` and `task_ids`. Pass `previous_version_id` to create the next immutable version. Sources must be available to the workspace and selected tasks must include their dependencies.

Run `evalrouter suites create` in a terminal to select sources and exact tasks, name the suite and save without a JSON file or a model run. Use `evalrouter suites version --id SUITE_VERSION_ID` to edit a saved selection into its next version.

Use `evalrouter run --suite SUITE_VERSION_ID --model MODEL` to reuse a saved version. Add `--dry-run` to inspect preflight without launching. Exact-task suites use evaluation jobs; the legacy suite quote path rejects them. Suite operations are preview SDK capabilities. CLI availability requires a package containing this feature.