Upload a file

Use Import a benchmark in the browser, or a small manifest with the CLI. You need member, admin or owner access. Viewers can read benchmarks but not import them.

Import in the browser#

1. Open Import a benchmark while signed in, and put a file on the source area or choose it. (For a dataset, see Hugging Face or GitHub.)
2. Map the columns, write the prompt with {question} (and {choices} for multiple choice), and choose a grader.
3. Confirm your data rights, then validate. Review the first rows exactly as the model receives them, and create the benchmark.

The page uses your account and workspace, so you don't need an API key. Your settings stay for 24 hours after a refresh in the same tab. The file contents and tokens don't stay, so select the file and confirm your data rights again. If the browser can't save a draft, keep the tab open until the import completes.

Use the CLI#

1. Sign in with evalrouter login. A device signed in before importing existed must sign in once more. Device grants are never silently expanded. For CI, use a workspace key with the benchmarks:submit scope, plus the evaluation read and write scopes to quote and run. The write scope alone can't import.

2. Write a manifest next to your data file, in JSON. YAML also works when you install PyYAML with the CLI.

   JSON

   {
     "name": "Capitals (synthetic example)",
     "data": "capitals.jsonl",
     "task": "exact_match",
     "columns": {"id": "id", "question": "country", "answer": "capital"},
     "prompt_template": "What is the capital of {question}? Answer with the city name only.",
     "grader_options": {"extract": "last_line"},
     "max_output_tokens": 16,
     "data_license": "LicenseRef-Workspace-Private",
     "rights_attestation": true
   }

3. Check every row without creating anything. The CLI uploads and verifies the data, and EvalRouter charges nothing.

   Terminal

   evalrouter benchmarks private create manifest.json --dry-run

   The response shows the row count, what the model receives, the first three rows as delivered, and warnings. Answers aren't shown, because they're never delivered. EvalRouter refuses a problem and names its row and reason:

   • a missing column
   • a duplicate ID
   • an answer that isn't a number where the grader needs one
   • a mapped column that carries the answer (byob_answer_in_input)

4. Create it, then run it by the printed ID like any benchmark:

   Terminal

   evalrouter benchmarks private create manifest.json
   evalrouter run byob-<uuid>@1 --model openai/gpt-4.1-mini-2025-04-14 --full --max-cost 1

   --max-cost 1 caps the run at $1. Add --dry-run to run to see the quote. Its warnings repeat what the model receives and the grader's match rule. Read and export results as in results.

What the model receives#

The dry run, the created benchmark (delivery in the API response) and every quote state this contract for your benchmark.

• Delivered per row, and nothing else:

  1. your system_prompt, verbatim, if you set one
  2. your prompt_template as the user message, with each placeholder replaced by that row's mapped value and {choices} by lettered lines (A. text)

  EvalRouter adds no instructions, format hints or examples. A mapped value must be text or a number. EvalRouter refuses a list or an object.

• Never delivered: answer, id and group. The grader alone reads the answer, and the template can't contain {answer}. Unmapped columns are never read and stay only in the stored copy of your file.

• Answer guard: if a row's delivered messages contain its own answer, EvalRouter refuses the benchmark with byob_answer_in_input and names the column. The check uses Unicode NFKC, case folding and collapsed whitespace, and covers answers of 20 or more characters. Shorter answers occur by chance and aren't checked, so make sure no mapped column carries them. Multiple-choice answers are letters and are never checked.

• How each row runs: 0 shots, one greedy attempt at temperature 0, no automatic retry, no tools, each answer cut at max_output_tokens. Validation refuses a rendered user message over 100,000 characters.

• How each row scores: one grader decides 1 or 0 from the text output alone. The benchmark records its options, including defaults, and they can't change.

Graders#

Every grader is built in, deterministic and scores each row 1 or 0. They never execute uploaded code or arbitrary lm-eval or Inspect repositories. Results report accuracy with its standard error. Their label names an EvalRouter built-in grader, and you can compare them only with other runs of the same benchmark ID. Maintained catalog benchmarks can use their own specialized graders.

normalize defaults to Unicode NFKC, case-folded and whitespace-collapsed comparison. Normalization keeps punctuation unless you set strip_punctuation, which turns punctuation into spaces. An unparseable or unmatched answer scores 0 with a recorded reason. Timeouts and provider errors are errors, never scores.

Manage private benchmarks#

evalrouter benchmarks private list lists them. Archiving one with benchmarks private archive stops new runs and keeps earlier runs and results. In the catalog, Source › Created by me shows the ones you created in this workspace. Imports with the same name have separate IDs, and the creation date tells them apart. Members, admins and owners can choose Delete, which asks you to check the ID and date. Delete then does three things:

• It archives that benchmark.
• It removes the benchmark from the active catalog.
• It frees the benchmark slot.

Delete keeps the uploaded source and its source slot.

A benchmark version never changes. To change the data, template or grader, create a new benchmark.

evalrouter benchmarks uploads list --status completed
evalrouter benchmarks uploads archive <upload_id>

From a GitHub data file#

GitHub imports are available only to enabled workspaces. Elsewhere the page hides the GitHub source, and a saved draft or direct link offers file upload or Hugging Face instead.

1. In Import a benchmark, choose GitHub. Use Public for a repository URL or owner/repository, or Private to select one connected private repository. Owners and admins connect or disconnect accounts in Settings → Integrations. Set repository permissions on GitHub. Connecting an account makes its granted repositories available in the importer. You don't need a personal access token.
2. Files load automatically from the default branch. Expand folders in the file tree to select one CSV, JSONL or Parquet file, or filter by typing a path. You can also paste an exact file path. Use the arrow keys to move through files and expand or collapse folders, and Enter or Space to select a file. To use another revision, choose a branch, tag or recent commit from the dropdown. You can also enter a reference. Files can be up to 16 MiB and within your workspace limits. The import keeps the commit it shows, even if the branch later moves.
3. Map the columns, choose one of the built-in graders, confirm your data rights, then validate and create the benchmark. The verified file and benchmark stay private to your workspace, including data from a public repository.

This imports data for your chosen prompts and scoring rules. It does not run repository code, install dependencies, or reproduce an author's custom evaluation protocol. JSON arrays, Git LFS pointers, symlinks and submodules are not supported. Upload the underlying supported data file instead.

evalrouter benchmarks imports capabilities, or the capability endpoint below, shows whether GitHub imports are available to your workspace. Where they are, the GitHub import commands (refs, preview and import) follow the same steps as the browser.

Privacy, rights and limits#

• A private benchmark and its upload are visible only inside your workspace. Anyone else gets "not found", never "forbidden". It never enters the shared catalog, and you can't share its results through a public link.
• EvalRouter stores your file once per upload in your workspace's private storage. It reads the file back by its SHA-256 before every run. It never deduplicates or compares identical files in two workspaces.
• Each run you start sends the rows to its model provider, like any benchmark prompt.
• EvalRouter treats uploaded content as untrusted data and executes nothing in it. Previews and errors replace strings that look like credentials with [redacted] and warn you. The run still sends the file itself to the model, so remove real credentials.

Rights and terms#

In the browser, choose the dataset’s existing license. Use Private or proprietary for data governed by your own private terms, a named license when it applies, or Other license to enter its identifier. This records the existing terms. It does not change them or make the benchmark public. The CLI records this choice in data_license. Hugging Face imports preserve the verified source license.

By setting rights_attestation: true you confirm: "This workspace has the right to use this data for evaluation and to send its rows to the model providers selected for each run. EvalRouter keeps it private to this workspace and never publishes, shares or reuses it." The benchmark records the statement and your data_license, and every result's notices show them. Don't upload data that you may not process this way. Don't upload personal data that you have no basis to send to model providers.

Limits#

The 64 MiB limit applies to the file that you build a benchmark from. An upload or a Hugging Face import can be larger, up to your workspace's upload limit. EvalRouter refuses to create a private benchmark from a file over 64 MiB, with private_benchmark_too_large.

Uploads and imported GitHub and Hugging Face files share your workspace's storage allowance. A GitHub import reserves its bytes before it writes to storage. Retrying the same import after a storage failure reuses that reservation. Archiving a benchmark or completed upload does not free those stored bytes.

Not supported yet: model-graded (judge) scoring, partial credit, few-shot examples, more than one attempt per row, and custom grader code.