# Save traces to a dataset

Keep useful or failed traces together in a workspace-private dataset. Saving a trace copies the content that is still available at that moment. It does not change the source trace or generate a correct answer.

## Create a dataset

Open **Datasets**, choose **New dataset**, enter a name, and create it. Members, admins and owners can create datasets and save traces; viewers can read retained snapshots.

## Add one trace

Open a trace and choose **Add to dataset**. Search for an existing dataset or create one in the same dialog, then save. The saved snapshot includes available span inputs, outputs, errors, events, attributes and source provenance. Missing content stays marked as missing.

The picker includes empty datasets and trace collections. Datasets containing workload uploads, curated examples or frozen versions stay separate so saving traces does not interrupt their review or versioning flow.

## Add several traces

On **Traces**, select rows, then choose **Add to dataset**. You can select up to 50 traces at once. Search the dataset picker or create a dataset and save the selection. If part of the selection fails, retry the failed traces without saving the successful copies again.

You can also open a dataset and choose **Add traces**. Filter for failed traces, select the ones to keep, and save them.

## Inspect saved evidence

Open a dataset to see saved traces, recorded outcomes and input/output coverage. Choose **View snapshot** to inspect a copy without leaving the dataset.

A dataset has its own retention period. Its saved copies remain available until that period ends, even if the original trace expires. Only content permitted by the source policy can be copied; redaction and workspace access still apply. This initial release supports up to 100 spans and 256 KiB of saved evidence per trace. If the trace changes during capture, retry the save to capture one consistent revision.

These snapshots are observations. They do not become reference answers, exportable training data or runnable benchmarks automatically. Dataset imports, review workflows, version exports and benchmark generation are outside this initial Production UI. The existing Dev workload review and versioning screens remain available independently.

## Turn your app's traces into a benchmark

A coding agent can run this whole flow from the CLI with a workspace API key or a CLI sign-in. It is available on Dev, where private benchmarks and workload evaluators are enabled. Add `--json` to any command for machine-readable output.

1. Create a trace connection. An API key needs the `eval:write` and `traces:write` scopes; a CLI sign-in needs the admin or owner role. The ingest key is shown once, so write it to an ignored env file and never commit it.

   ```sh
   evalrouter traces connect --name my-app --env-file .env.local
   ```

2. Capture traces with the SDK using `EVALROUTER_TRACE_CONNECTION_ID` and `EVALROUTER_TRACE_KEY`. Set the root span input to the user's request and the root span output to your app's final answer. Plain text works best. `evalrouter traces list --connection CONNECTION_ID` shows what has arrived, and `evalrouter traces connections list` lists the connections.

3. Create a dataset and import the most recent completed traces. Each trace becomes one held-out example: the root span input is the task and the root span output is the reference answer. Traces without both, traces whose root span failed and oversized traces are skipped and counted in `skipped`. Importing confirms that your workspace may evaluate models on this data.

   ```sh
   evalrouter datasets create --name "my-app production"
   evalrouter datasets import-traces DATASET_ID --connection CONNECTION_ID --limit 200
   ```

4. Review the examples, then approve them and freeze a version. `--all` approves every pending example at its current draft. Use `--example ID` to approve a reviewed subset instead.

   ```sh
   evalrouter datasets approve DATASET_ID --all
   evalrouter datasets versions create DATASET_ID
   ```

5. Build a private benchmark from the version. The `match-production` grader asks a managed judge model whether each answer is at least as correct, complete and helpful as your app's answer for the same input. Each grade is a metered judge call. Use `--grader exact-match` for exact string matching instead. An API key needs the `benchmarks:submit` scope for this step. The command returns `benchmark_id`.

   ```sh
   evalrouter datasets benchmark DATASET_ID 1 --grader match-production --judge-route JUDGE_ROUTE
   ```

6. Run each candidate model on the benchmark, group the runs, and rank them:

   ```sh
   evalrouter run BENCHMARK_ID --model MODEL_A --full --yes
   evalrouter run BENCHMARK_ID --model MODEL_B --full --yes
   evalrouter result-groups create --name "my-app models" --run RUN_A --run RUN_B
   evalrouter result-groups get GROUP_ID --rank
   ```

   `--rank` orders models by their headline score, then by cost, and marks the best model and the cheapest model within one standard error of it.

The cheapest model within one standard error of the best score is the usual switch candidate. A judge that cannot decide leaves that case ungraded, so check coverage before you switch.