Save traces to a dataset
Early accessKeep useful or failed traces together in a workspace-private dataset. Saving copies the content still available at that moment. It doesn't change the source trace or generate a correct answer.
This feature is rolling out to workspaces. Ask us to turn it on.
On this page
Save traces#
- Open Datasets › New dataset and name it. Members, admins and owners can create datasets and save traces. Viewers can read retained snapshots.
- Open a trace and choose Add to dataset, or select up to 50 rows on Traces and choose Add to dataset. You can also open a dataset, choose Add traces and filter for failed ones.
- Search for a dataset, or create one in the same dialog, and save. If part of a selection fails, retry the failed traces. Successful copies aren't saved twice.
A snapshot includes available span inputs, outputs, errors, events, attributes and source provenance. Missing content stays marked as missing. The picker lists empty datasets and trace collections. Datasets with workload uploads, curated examples or frozen versions stay separate so saving traces doesn't interrupt their review or versioning.
Tell infrastructure failures from model failures#
A model call can fail because the provider, EvalRouter or the run's own limits ended it, not because of the model's answer. Before you save failed traces as evidence about a model, check which kind of failure each one is. Each model call reports these fields:
| Field | Meaning |
|---|---|
failure_class | Empty when the call went through. Otherwise one of the classes below. New values can be added. |
failure_class_source | exact for EvalRouter's own gateway calls, from the gateway's record. reported for imported calls and a remote provider's own spans, read from what the span says. |
upstream_http_status | The provider's HTTP status, when known |
attempt_count, retried_count | The attempts made and how many were retries. Known for EvalRouter's gateway calls only. |
model_outcome | The model's finish reason (length, for example) or completed; empty when the call didn't complete |
| Class | Meaning |
|---|---|
provider_fault | The provider or endpoint failed: a 5xx, unreachable, an invalid answer |
request_error | The provider refused the request as invalid, or refused the credential of your own connection |
throttled | Rate limited |
timeout | No answer before the deadline |
client_cancel | The calling application cancelled the call |
platform_fault | EvalRouter's own path failed, including a managed route's credential |
stopped | The run's own terms ended it: its budget cap, reversed credit, a withdrawn route or connection |
unclassified | It failed, and the cause can't be read |
These fields describe the outcome after retries. A call that the gateway retried until it succeeded isn't a failure; its retried_count shows the retries. A failure is never the model's: whether the model achieved the task is its grade. A trace's failure_class is its dominant class: the class with the most failed calls, a tie going to the class that failed last, then to the later class name, or empty when no call failed.
What an imported span carries depends on where it came from:
| Source | What classifies a failure |
|---|---|
| OpenTelemetry GenAI spans | error.type, when it names a rate limit, timeout or cancellation; gen_ai. gives the finish reason |
A span with http.response.status_code | 429 is throttled, 408 and 504 timeout, 499 client_cancel, other 5xx provider_fault, other 4xx request_error |
| LangSmith, Braintrust, Langfuse, OpenInference | Nothing yet, so their failures are unclassified |
GET/v1 returns all five fields on span rows, and the dominant failure_class on trace rows. Three counts of calls come from GET/v1, in its totals and in each series bucket, and from a single trace's failure_counts in GET/v1: failures_by_class, and failures_by_source and model_calls_by_source, which count failed calls and calls for each source. The source counts leave out stopped calls, so failures_by_source. is EvalRouter's own failure rate, and a budget cap never reads as unreliability.
failure_class and failure_class_source are filters and group_by dimensions on spans and traces, for example {"field": . On traces they use the dominant class and the source of that class's last failed call, and each group counts traces, not calls.
Compare endpoints#
endpoint_connection_id on a span is the connection whose endpoint answered an EvalRouter model call. It differs from connection_id, which is where the trace came from: every span of a run shares the run's connection. It is empty for a managed route, which you identify by model and model_provider, and for imported spans. Filter or group spans by it, for example:
{"version": 1, "source": "spans", "filters": [{"field": "is_leaf_model", "op": "eq", "value": 1}], "group_by": ["endpoint_connection_id"]}Each breakdown row of GET/v1 has the same measures as its totals, including count, error_count, error_rate, the latency percentiles and duration_ms_max, failures_by_class, token and cost totals, and output_tokens_per_second. Match connection IDs to names with GET/v1.
- Per attempt. EvalRouter retries some failed calls, and each retry is its own span.
count,error_rateand the latency percentiles count attempts.failures_by_classcounts calls that failed after their retries. output_tokens_per_secondis the sum of output tokens divided by the sum of call durations, over the calls that report both. A call's duration runs from the request to the complete answer, including queueing and prompt processing, because EvalRouter doesn't stream. It is empty for trace breakdowns, because a trace's duration includes work other than its model calls.- Not like for like. This breakdown covers whatever each endpoint happened to answer. Endpoints that answered different tasks, prompts or runs can't be compared fairly this way. To compare endpoints on the same tasks, run a comparison.
What the agent did#
Each tool call is one span with its final status, so you can count an agent's calls and see which tools fail. A span's tool_name names the tool. Its tool_origin says whose tool it was: agent for the agent's own tools, task for the task's tools through EvalRouter, or environment for an environment's tools. Grading, an environment's own setup and teardown, and model requests the task refused carry a span_class (grader, infrastructure or refused_request). They are not the agent's calls.
Trace detail retains EvalRouter's metered execution fee with public kind=tool; its raw evalrouter. attribute identifies it as platform execution rather than an agent tool.
To see failures per tool, query spans grouped by tool_name, with the filters {"field": and {"field": . Each group's top_error_type is its most common error. Trace rows carry tool_call_count and tool_error_count. Analytics on traces return tool_calls_per_trace_p50, tool_calls_per_trace_p90 and tool_calls_per_trace_max, and the same for model calls.
Filter by whether the task passed#
Each trace, and each of its spans, carries whether its task passed, separate from the failure fields above:
| Field | Meaning |
|---|---|
verdict | passed, failed, inconclusive or not_graded. New values can be added. Empty while the task runs, when its grade is a score only, or while the grade is withheld. |
verdict_judged_by | Who decided: benchmark_grader for a trace from an EvalRouter run, trace_evaluator for any other trace. New values can be added. |
verdict_score | From 0 to 1 |
verdict_evaluator_key | The evaluator's key when exactly one evaluator judged the trace; empty when several did |
Any other trace takes the most severe of its evaluators' results: failed if any evaluator failed it, otherwise inconclusive if any was inconclusive, otherwise passed. Each evaluator counts with its most recent whole-trace result. Opening a trace (GET/v1) lists each evaluator's verdict in trace.evaluator_verdicts.
A trace from an EvalRouter run shows its task's verdict, the same as the run's task list. failed only ever means that the model's answer was wrong: a task whose model calls failed is not_graded. A client or remote agent run shows verdicts once the run ends, and a benchmark that shows only aggregate results never shows them.
Filter or group by these fields in GET/v1 and GET/v1, for example {"field": . GET/v1 also returns, in its totals and each time bucket, verdicts (the number of traces per verdict) and pass_rate: rate, its 95% interval lower and upper, and n. The rate is passed traces divided by passed and failed traces, the same calculation as a run's pass rate. Inconclusive traces aren't in it. On spans, this selects the calls made in failed traces.
Break traces down by agent, Tune configuration or your own metadata#
Traces from EvalRouter runs carry what ran them:
| Field | Meaning |
|---|---|
environment_version | The environment version the run used, such as acme/support@3. environment_id is its ID. |
agent | The agent: its registry ID, your local agent's name, or your registered agent's name |
agent_version | The agent's version. A local agent that states no version shows a short fingerprint of what ran, such as sha256:3f2a9c1b7e40. |
tune_session_id | The Tune session the run belonged to |
tune_configuration | The model and reasoning level the run tried in that session, as route or route:level |
Any trace, including traces you send or import, can also carry your own metadata keys. An admin chooses up to 10 keys in Settings › Integrations › Metadata keys, or with PUT/v1, for example {"keys": . EvalRouter then keeps those keys, and no other attributes, for analytics. A key's value comes from the span attribute of that name, or from that key in a metadata object (imports from LangSmith and others keep it there). For a trace from an EvalRouter run, it can also come from the run's metadata. A trace takes its root span's value, otherwise the earliest span's that has the key.
Filter or group traces by metadata.<key>, for example {"group_by": . Sessions can be filtered by it. Spans can't, because a child span seldom repeats its trace's metadata. Declaring a key applies to traces received afterwards; earlier traces are not updated. Removing a key stops keeping it, but values already kept stay until their traces are deleted.
All these fields work as filters and group-by fields in GET/v1 and GET/v1. GET/v1 lists them. Span rows include the run fields; trace rows don't, so trace lists stay fast. Group or filter by them instead.
Chart a measure over time, by group#
GET/v1 returns the same measures in its totals, each time bucket, each series_by bucket and each breakdown row. The exception is a breakdown by a behavior field, such as behavior_status, which counts findings instead (finding_count, observed_count). The measures are:
failure_rate: failed model calls, after retries, over the calls that ended, withby_classper failure class. Calls the run's own spending cap stopped aren't failures. This isn'terror_rate, which counts traces (or spans) with an error.pass_rateandverdicts, as above.cost_per_case: what EvalRouter charged per trace. Empty for imported traces.reported_cost_per_case: the cost your provider reported, per trace.tool_calls_per_trace_p50,_p90and_max, and the same for model calls (traces only).
Both GET/v1 and GET/v1 return retention: days, your workspace's trace window, and kept_since, the time before which no trace that EvalRouter received is kept. A trace you import keeps its own start time, so it can start before kept_since.
Every explicit request carries ISO from_ts and to_ts values and can cover at most 366 days, so a leap-year calendar selection fits. The web offers 24-hour, 7-day, 30-day, 3-month, 6-month and 9-month presets plus a custom range. GET/v1 returns earliest_started_at (or null when unknown), which callers can use to disable unavailable earlier days. The response's window gives the resolved start and end plus the chart's bucket and bucket_seconds. EvalRouter uses minute buckets through 6 hours, five-minute buckets through 24 hours, hourly buckets through 7 days, daily buckets through 90 days, and Monday-based UTC weekly buckets beyond that. When an analytics range is too large for its selected data path, the 422 response gives the largest fitting whole-day range in suggested_window_seconds and suggested_from_ts.
Trace Logs scan newest-first in bounded slices. A selective search over a long range can return fewer rows than the requested limit while has_more is true; follow next_cursor to continue the scan. This keeps Load more bounded without skipping or repeating traces.
Add "series_by": "agent" to get one series per value of a field. series_by works on the traces and spans sources, not sessions. It takes one field from that source's group_by_fields in GET/v1, including metadata.<key> on traces. Behavior fields such as behavior_status are refused with a 422, as is anything that isn't a field name.
You get the most frequent values first: 5 by default, or set series_limit from 1 to 10. An empty value counts as a value of its own. One group marked other holds the rest, and it appears only when there is a rest. Each group has exactly the time buckets and measures of series, and the groups add up to it.
Inspect saved evidence#
Open a dataset to see saved traces, recorded outcomes and input/output coverage. Choose View snapshot to inspect a copy without leaving the dataset.
A dataset is kept until you delete it. Its saved copies stay when the original traces are deleted. A dataset copies only the content that the source policy permits. Redaction and workspace access still apply. This initial release supports up to 100 spans and 256 KiB of saved evidence per trace. If the trace changes during capture, retry the save to capture one consistent revision.
These snapshots are observations. They do not become reference answers, exportable training data or runnable benchmarks automatically. These screens don't include dataset imports, review workflows, version exports or benchmark generation.
Turn your app's traces into a benchmark#
A coding agent can run this flow from the CLI with a workspace API key or a CLI sign-in. It needs a deployment with private benchmarks. Add --json to any command.
-
Create a trace connection. An API key needs the
eval:writeandtraces:writescopes. A CLI sign-in needs the admin or owner role. EvalRouter shows the ingest key once. Write it to an ignored env file, and never commit it.Terminalevalrouter traces connect --name my-app --env-file .env.local -
Capture traces with the SDK using
EVALROUTER_TRACE_CONNECTION_IDandEVALROUTER_TRACE_KEY. Set the root span input to the user's request and the root span output to your app's final answer. Plain text works best. The traces list command shows what has arrived, and the connections list command lists the connections. -
Create a dataset and import the most recent completed traces. Each trace becomes one held-out example: the root span input is the task and the root span output is the reference answer. The import skips a trace without both, a trace whose root span failed and an oversized trace. It counts them in
skipped. Importing confirms that your workspace may evaluate models on this data.Terminalevalrouter datasets create --name "my-app production" evalrouter datasets import-traces DATASET_ID --connection CONNECTION_ID --limit 200 -
Review the examples, then approve them and freeze a version.
--allapproves every pending example at its current draft. Use--example IDto approve a reviewed subset instead.Terminalevalrouter datasets approve DATASET_ID --all evalrouter datasets versions create DATASET_ID -
Build a private benchmark from the version. The command compares each answer with your app's answer as exact text. An API key needs the benchmark submit scope for this step. The command returns the benchmark ID.
Terminalevalrouter datasets benchmark DATASET_ID 1 -
Run each candidate model on the benchmark, group the runs, and rank them:
Terminalevalrouter run BENCHMARK_ID --model MODEL_A --full --yes evalrouter run BENCHMARK_ID --model MODEL_B --full --yes evalrouter result-groups create --name "my-app models" --run RUN_A --run RUN_B evalrouter result-groups get GROUP_ID --rank--rankorders models by their headline score, then by cost, and marks the best model and the cheapest model within one standard error of it.
The cheapest model within one standard error of the best score is the usual switch candidate.