Import a workload
Early accessBring production traces, datasets and experiments from Braintrust, Langfuse or LangSmith into review, then turn approved tasks into a reviewed workload.
This feature is rolling out to workspaces. Ask us to turn it on.
Connect a source#
From New source, a workspace admin supplies the provider credentials and selects the project and region. EvalRouter encrypts the credentials and syncs traces in the background. The connection shows progress and errors, with controls to pause, resume, import history or update credentials.
A workspace can connect each source project once, and at most 20 projects in total. EvalRouter checks provider keys at most 30 times in 10 minutes for each workspace and each person. When you reach that limit, wait and try again. EvalRouter skips any trace or dataset row whose ID contains a credential.
For Braintrust and Langfuse datasets or experiments, open a dataset and choose Import workload data. Select the connected project and resource, declare usage rights and assign a split, then import pages until the import completes. Source identifiers and content hashes keep provenance across retries.
Review before you rely on it#
- Historical model outputs and provider scores stay source evidence. They never become correct answers or pass labels.
- Imported inputs seed a task draft. Review its instruction, inputs, tool behavior, reference answer and grading rule before approving it, and freeze an approved dataset version before you use it.
- Choose configurations on development tasks and keep separate held-out tasks for the final comparison. A copied task, or a paraphrase of one, stays in its source's split.
Grading#
Grade each task with a reviewed deterministic rule. Imported historical scores never become a grader, and these imports never run uploaded scorer code.
Provider connections depend on deployment availability. See availability.