Models, environments, and compatibility

For automated release checks, the CI integration guide includes a Python adapter and GitHub Actions workflow. It runs a pinned task suite against candidate and baseline models, then fails CI when evidence is incomplete or a configured score or regression threshold is violated.

Choose a versioned evaluation and a model that can run it. Check the pair before you spend.

GET /v1/evals lists evaluations from admitted providers. The public catalogs show native benchmarks and environments. Your workspace may also have private environments, which need workspace context to discover.

Model routes and connections#

A managed route states its model, capabilities, rates, and limits. Choose the exact catalog entry. A familiar model name alone does not tell you whether the route supports the tools, reasoning, or output format a task needs.

A workspace connection points to your provider endpoint, which bills you separately. Checking it makes a small provider request and may cost money outside EvalRouter.

Task identity#

Use eval://provider/name/version to name the exact evaluation. Native aliases such as name@version resolve through the registry. Save the resolved identity with reports; a display name cannot identify the version later.

The quote checks model capabilities, input and output limits, task availability, permissions, and resources. An unsupported pair fails before the run starts. Removing a requirement changes the evaluation protocol; it is not a compatibility fix.

Coverage#

A sample freezes its tasks and seed. A full run targets the admitted task collection. Report completed, missing, failed, and excluded tasks separately. Scores from different subsets, prompts, graders, or splits may not be comparable, even under the same benchmark name.

Use development results to iterate. Keep held-out tests separate. Remember that prior benchmark exposure or pretraining contamination can limit what a score means.

Choose specific tasks#

To choose tasks yourself, open New evaluation → Benchmarks → Select tasks. Search task IDs and metadata, check tasks for each source, then review the exact IDs before starting. Required dependencies are included when available. The saved setup and repeated runs retain exact task selection.

In the CLI, select Specific tasks during interactive evalrouter run, or repeat --task with IDs from that benchmark version:

evalrouter run BENCHMARK --model ROUTE_ID --task TASK_ID_1 --task TASK_ID_2 --dry-run

--task cannot be combined with --sample or --full. GET /v1/catalog/tasks?source_ref=EXACT_REF lists safe task metadata for an accessible source, with q, limit, and offset for search and pagination. The quote checks whether the selected tasks can execute with your model. An unavailable task selection does not fall back to sampling.

Recommend tasks for an objective#

Recommendations are a preview. Save a task selection with the CLI:

evalrouter recommend --objective "math and reasoning" --model ROUTE_ID --tasks 25 --json > recommendation.json

The API equivalent is POST /v1/recommendations with objective, models, and task_limit. Review its selections, uncovered capabilities, exclusions, and warnings. Recommendation itself starts no work and creates no quote. To run the selected tasks of one benchmark, pass their IDs to evalrouter run with --task, as above.

Tasks can come from multiple benchmarks and environments. Selection preserves their source versions and explicit task IDs. Declared task metadata determines relevance where available; source labels are a disclosed fallback. Required dependencies count toward the task limit. Results on a selected subset are not full benchmark scores.

Optional --budget is historical-cost selection guidance only. Cost evidence can be missing or based on a source mean. It is not a guaranteed price or execution spending cap. Launch uses available workspace credit and actual usage billing.

Reusable exact-task suites#

In New evaluation, choose tasks on the Benchmarks step and select Save selection as suite. A suite saves the exact source versions, splits and task IDs across benchmarks and environments. It does not save a model or start an evaluation. Open Suites to configure an evaluation from a saved version or create its next version. Earlier versions remain available.

The API accepts task_selections in POST /v1/suites. Each selection contains source_ref, split and task_ids. Pass previous_version_id to create the next immutable version. Sources must be available to the workspace and selected tasks must include their dependencies.

Run evalrouter suites create in a terminal to select sources and exact tasks, name the suite and save without a JSON file or a model run. Use evalrouter suites version --id SUITE_VERSION_ID to edit a saved selection into its next version.

Use evalrouter run --suite SUITE_VERSION_ID --model MODEL to reuse a saved version. Add --dry-run to inspect preflight without launching. Exact-task suites use evaluation jobs; the legacy suite quote path rejects them. Suite operations are preview SDK capabilities. CLI availability requires a package containing this feature.