Tune

Tune is a preview: it is off unless it is enabled on the deployment, and it is not covered by the stability promise.

Tune compares model or agent configurations on a reviewed workload. It records the source, task version, environment, grading configuration, model routes, and spending cap, if any, used in the comparison.

Open Tune. For a complete agent, select a prepared repository and an approved dataset version containing development and held-out tasks. Choose the task grading rules or a saved calibration for that version. Set the objective, quality tolerance, and candidate limit. Review the quote before starting.

Candidates run on development tasks. Tune selects one finalist for comparison with the current agent on held-out tasks. Incomplete execution cannot establish a successful replacement. Inspect the result's quality, cost, latency, and execution evidence before adopting a change.

Export the tested configuration from the result. The export includes its source and workload commitments and the selected policy. A repository change can be reviewed in the linked improvement project and submitted as a pull request. Opening a pull request does not merge or deploy it.

The CLI supports the same flow. See Tune your repository's agent and open a pull request.

To compare admitted ACP versions, select ACP agent versions and enter the environment, baseline, alternatives, and two declared disjoint splits. Each alternative needs its own valid admission. Tune exports the winning admission and native run evidence; it does not modify an admitted image. The CLI accepts --acp-environment, --baseline-agent, and repeated --candidate-agent arguments.

For a single model call, choose that mode in the interface or run evalrouter tune start --repo owner/repository --max-draft 2. Review the generated task cases and expected answers before approving the workload. Repository drafting has its own explicit cap. Rubric approval requires a managed judge route (--judge-route in the CLI), whose inference is included in the native quote.

Remote evaluator configurations show a maximum call count and declared provider charge estimate before an agent search. Those charges are billed by the scorer provider separately from EvalRouter.

Recurring verification requires separate standing authority with a per-run cap and a total cap. It checks the selected configuration against fresh, approved validation tasks. It does not repeat the search or reuse held-out tasks. A scheduled check with no fresh workload records a skipped occurrence without starting a paid job. Queue a reviewed dataset version for a scheduled check, or submit it as an event for an event-triggered check. Versions that use a calibrated evaluator need a fresh calibration of that same evaluator. For ACP results, submit a new approved environment version and its declared split. The check runs the fixed baseline and selected admission on fresh tasks under the occurrence cap. You can pause the authority.

Tune must be enabled on the deployment. Production activation requires qualification of the selected application and runtime plus independent approval. Local implementation and passing tests do not establish that activation. See availability and budgets.

Repositories can declare model roles in evalrouter-model-policy.json:

{
  "schema_version": "evalrouter.model-policy.v1",
  "policies": {
    "default": { "planner": "MODEL_ID", "writer": "MODEL_ID" }
  }
}

The application must read those roles with the runtime helper: from evalrouter_model_policy import get_model in Python, or require('evalrouter-model-policy.cjs').getModel in Node. Pass the role name to select its model. Tune changes only declared roles, pins managed routes, and rejects unquoted model calls. The declaration alone does not change applications that use hard-coded model names.

Tune your repository's agent and open a pull request#

A coding agent can run this flow from a local clone of a Python uv and LangGraph repository. EvalRouter tests the candidates and returns the selected change as a patch. The agent opens the pull request with local git and the GitHub CLI.

1. Connect GitHub once in the browser at Integrations and grant the GitHub App access to the repository. This is the only browser step. When the app cannot read the repository, evalrouter improvements create stops with the error code github_repository_unreadable and prints this link.

2. Create and prepare the improvement project. create pins the commit of --ref and uses the workspace's only active trace connection unless you pass --trace-connection. prepare fetches that commit and finds the LangGraph entrypoint from langgraph.json and uv.lock.

   sh

   Copy

   evalrouter improvements create --repo owner/name --ref main --json
   evalrouter improvements prepare PROJECT_ID --json
   evalrouter improvements get PROJECT_ID --json

3. Start Tune with an approved dataset version that has development and held-out tasks, then quote the search. Without --max-cost the sweep is paid from wallet credit as it runs and stops when credit runs out. An agent passes --max-cost only with a cap the person gave. Show the quote; the person's request to tune is enough to start the sweep.

   sh

   Copy

   evalrouter tune start --agent-project PROJECT_ID --dataset-version VERSION_ID --json
   evalrouter tune quote SESSION_ID --json
   evalrouter tune start-sweep SESSION_ID --json
   evalrouter tune status SESSION_ID --json
   evalrouter tune result SESSION_ID --json

   Repeat status until the session is complete.

4. Export the result as a patch. The patch applies at source_commit_sha in the export. When the result keeps the current agent, the export has no patch, the command stops with the error code tune_no_change and there is nothing to open.

   sh

   Copy

   evalrouter tune export SESSION_ID --patch tune.patch --json

5. Open the pull request from the clone. Write the description from tune result: the held-out quality and cost of the current agent and the candidate, and the session ID.

   sh

   Copy

   git fetch origin
   git switch -c evalrouter/tune-SESSION_ID SOURCE_COMMIT_SHA
   git apply --index tune.patch
   git commit -m "Apply EvalRouter Tune result SESSION_ID"
   git push -u origin evalrouter/tune-SESSION_ID
   gh pr create --title "Apply EvalRouter Tune result" --body-file tune-summary.md

Opening a pull request does not merge or deploy it.