Repository agents with the CLI

Prepare a GitHub agent program for a supported environment, then run a separate evaluation from the CLI. ACP means Agent Client Protocol, the interface the agent program implements. A repository of benchmark tasks alone is not an agent program.

Account, installation and availability

Create an account, verify your email and create a workspace key. Paid preparation and evaluations require available workspace credit in Billing. Store EVALROUTER_API_KEY and EVALROUTER_WORKSPACE_ID privately; set EVALROUTER_BASE_URL to the service origin without /v1.

Repository commands require CLI 0.2.0 or later, Python 3.12 or 3.13, and a service that enables repository qualification for your workspace:

uv tool install --python 3.12 'kimpton-evalrouter-sdk>=0.2.0'
evalrouter agent-builds options --json
evalrouter catalog --environment ENVIRONMENT_FAMILY_PATH --json

For an existing installation, use uv tool upgrade kimpton-evalrouter-sdk. Installing the package does not enable the service. Stop if qualification is disabled or your exact target/model is absent. The catalog lookup uses the family path before @; preparation requires the exact versioned environment returned by options. An active catalog entry alone does not establish readiness.

Prepare your Node/npm program

The supported runtime uses Node 22 with a committed package.json, package-lock.json and evalrouter-agent.json. Custom images, arbitrary install commands and Python agent runtimes are not supported. Commit a manifest like this, adapting the ID and entrypoint:

{
  "schema_version": 1,
  "id": "example-agent",
  "protocol": "acp",
  "runtime": {
    "kind": "node-npm",
    "project": ".",
    "lockfile": "package-lock.json",
    "entrypoint": ["node", "agent.js"]
  },
  "inference": "managed-openai-chat"
}

This manifest is configuration, not an implementation: agent.js must implement the supported Agent Client Protocol interface. The project path is relative to the repository root; the entrypoint is relative to that project. Keep provider keys, GitHub tokens, custom endpoints and secrets out of the source URL and manifest.

Save build.json, replacing the URL, environment and model with your actual choices:

{
  "repository_url": "YOUR_PUBLIC_GITHUB_REPOSITORY_URL",
  "ref": "main",
  "manifest_path": "evalrouter-agent.json",
  "qualification": {
    "environment_ref": "EXACT_ENVIRONMENT_FROM_OPTIONS",
    "model": {"kind": "managed", "route_id": "MODEL_FROM_OPTIONS"}
  },
  "max_cost_microusd": "1000000"
}

The example cap is $1, not a price guarantee. Use the preview to assess whether the selected scope fits your cap and available credit.

Private repository access

New private-repository connections are not available in this docs-only website. Use a public repository for the setup above. Existing workspace repository grants are unchanged; this page does not create, select or authorize a repository grant. Returning from GitHub App installation alone does not grant workspace access.

Preview, create and recover preparation

evalrouter agent-builds preview --config build.json --json

Review the resolved immutable commit, manifest, qualification scope, cap, costs and expiry. Preview creates no job or credit hold and starts no sandbox or model work. Save its quote ID and a durable operation key before creating the job:

evalrouter agent-builds create --quote REVIEWED_BUILD_QUOTE --idempotency-key SAVED_BUILD_KEY --json
evalrouter agent-builds status BUILD_JOB_ID --json
evalrouter agent-builds list --quote REVIEWED_BUILD_QUOTE --json

Creation reserves the quoted cap and may incur the stated costs. Recover an uncertain submission with the same quote and key or the quote-filtered list; a new key may create separate work. To stop work, use evalrouter agent-builds cancel BUILD_JOB_ID. A cancelling or reconciling status remains unresolved. Wait for terminal cleanup and zero held funds.

Ready requires status: ready, cleanup_confirmed: true, zero held funds and an agent.ref. It proves only the returned compatibility scope, not benchmark quality or full coverage.

Quote and evaluate the ready revision

Use the ready record's exact qualification.environment_ref, qualification.split and entire qualification.evaluation_coverage object in a new quote.json:

{
  "agent": "READY_AGENT_REF",
  "selection": {"environment": "RETURNED_ENVIRONMENT_REF", "split": "RETURNED_SPLIT"},
  "coverage": "REPLACE_WITH_RETURNED_EVALUATION_COVERAGE_OBJECT",
  "max_charge_microusd": "1000000"
}

Replace the coverage placeholder with the returned object; do not guess tasks, counts or seeds. Preparation and evaluation have separate caps and charges. Ready does not automatically submit an evaluation.

evalrouter quote --config quote.json --json
evalrouter run --quote REVIEWED_EVALUATION_QUOTE --idempotency-key SAVED_EVALUATION_KEY --json
evalrouter wait RUN_ID --wait-timeout 3600 --json
evalrouter results RUN_ID --json
evalrouter export RUN_ID --format json --output result.json
evalrouter export RUN_ID --format csv --output result.csv
evalrouter export RUN_ID --format html --output result.html

Review the evaluation quote before running. Save its ID and a separate durable operation key. Inspect terminal status, completed coverage, errors and billing alongside scores. A local wait timeout does not cancel server work; use evalrouter cancel RUN_ID when cancellation is intended. See CLI recovery and versioned results.