Private benchmarks from your own data

Make a benchmark from your own data, private to your workspace. Upload a file, choose prompts and a built-in grader, and validate the rows. EvalRouter stores a verified copy and gives you a benchmark ID. Run it on a compatible chat model using available workspace credit.

Use Import a benchmark in the browser, or put the same settings in a small manifest for the CLI. File uploads and Hugging Face imports stay private to your workspace where imports are enabled.

What you need#

• A data file: JSON Lines (.jsonl), CSV (.csv) or Parquet (.parquet), one question per row, at most 64 MiB and 100,000 rows.
• Column mappings, a prompt template and one built-in grader. Set them in the browser, or use a JSON manifest with the CLI (YAML is also accepted when PyYAML is installed alongside the CLI).
• Member, admin or owner access to the workspace. Viewers can read benchmarks but cannot import them.

Import in the browser#

Open Import a benchmark while signed in. Choose a file or Hugging Face dataset, or GitHub when enabled for your workspace. Map columns, set scoring rules, and review a sample before creating the benchmark. The page uses your account and workspace. No API key is needed.

Drop a file onto the source area or choose it from your computer. The page previews a small local sample and shows upload and verification progress.

Your settings survive a refresh in the same tab for 24 hours. The draft does not save file contents or Hugging Face tokens. Reselect the file or re-enter the token when asked, then confirm data rights again. If the browser cannot save your draft, keep the tab open until the import is complete.

In the CLI, use evalrouter login. If your device was signed in before benchmark importing was added, sign in again once to grant that permission. Existing device grants are never silently expanded.

For CI and automation, create a workspace API key with benchmarks:submit, plus eval:read and eval:write to quote and run. Create it in API key settings and set EVALROUTER_API_KEY. eval:write alone does not grant permission to import.

Write a manifest for the CLI#

{
  "name": "Capitals (synthetic example)",
  "data": "capitals.jsonl",
  "task": "exact_match",
  "columns": {"id": "id", "question": "country", "answer": "capital"},
  "prompt_template": "What is the capital of {question}? Answer with the city name only.",
  "grader_options": {"extract": "last_line"},
  "max_output_tokens": 16,
  "data_license": "LicenseRef-Workspace-Private",
  "rights_attestation": true
}

How we interpret your dataset#

This is the whole contract between your file and the model. The dry run, the created benchmark (delivery in the API response) and every quote for it state the same thing for your benchmark.

What the model receives, per row: one or two messages, nothing else.

1. Your system_prompt, verbatim, as a system message, if you set one.
2. Your prompt_template as the user message, with each {role} replaced by that row's value from the column you mapped to the role, and {choices} (multiple choice only) replaced by lettered lines A. text, B. text, and so on.

EvalRouter adds no instructions, no answer-format hint, no examples and no other text. A mapped prompt value must be text or a number; a list or object is refused.

What the model never receives.

• answer: read to grade the row, never delivered. The template cannot contain {answer}.
• id and group: used to identify and sample rows, never delivered.
• Every column you did not map: not read at all and never delivered. It stays only in the stored copy of your file.

Answer guard. A mapped prompt column is delivered, whatever its name. If a row's delivered messages contain that row's own answer, the benchmark is refused with byob_answer_in_input, naming the column (for example a solution or explanation column mapped as {hint}). The comparison uses Unicode NFKC, case folding and collapsed whitespace, and covers answers of at least 20 characters after that. Shorter answers, such as numbers, letters or "yes", occur in prompts by chance and are not checked, so make sure no mapped column carries them. A multiple-choice answer is a letter and is never checked: delivering the correct choice's text is the format.

How each row is run. No few-shot examples (0 shots). One attempt per row, greedy, at temperature 0, with no automatic retry. Each answer is cut at max_output_tokens. No tools are offered. A rendered user message over 100,000 characters is refused at validation.

How each row is scored. One grader decides 1 or 0 from the model's text output alone. Its options, including the defaults you did not set, are recorded with the benchmark and cannot change.

Graders#

Self-service uploads use the five graders below. They support datasets with reference answers. They never execute uploaded Python code or arbitrary lm-eval or Inspect repositories. Maintained catalog benchmarks can use their own specialized grading implementations.

Every grader is built into EvalRouter, deterministic and scores each row 1 or 0. Results report accuracy with its standard error and are labeled as scored by an EvalRouter built-in grader, not by any benchmark author's own grader. They are comparable only to other runs of the same benchmark ID.

normalize defaults to Unicode NFKC, case-folded and whitespace-collapsed comparison. Punctuation is kept unless you set strip_punctuation (punctuation becomes spaces). An unparseable or unmatched answer scores 0 with a recorded reason. Timeouts and provider errors are errors, never scores.

Tune rubric grading. The API also accepts "grader_protocol": "tune_rubric_v1" with "task": "exact_match", an evaluator_id (a rubric version in your workspace) and a managed judge_route_id. Delivery to the model is unchanged. Instead of the table above, a separately metered managed judge reads the rubric, the delivered user message, the reference answer and the output, and returns pass or fail; an abstention leaves the row inconclusive.

Create and run it#

1. Check the manifest against every row without creating anything. The data is uploaded and verified. Nothing is charged:

   sh

   Copy

   evalrouter benchmarks private create manifest.json --dry-run

   The response shows the row count, how the dataset is interpreted (delivered and never-delivered columns, grader and match rule, examples, attempts and output limit), the first three rows exactly as the model receives them (system and user messages, untruncated), and any warnings. Answers are not shown, because they are never delivered. A mapping problem is refused with the row and reason, for example a missing column, a duplicate ID, an answer that is not a number, or a mapped column that carries the answer (byob_answer_in_input).

2. Create the benchmark:

   sh

   Copy

   evalrouter benchmarks private create manifest.json

   It prints the benchmark ID, byob-<uuid>@1.

3. Run it like any benchmark, by its ID (or through the quickstart request, replacing BENCHMARK_PROFILE_ID with it):

   sh

   Copy

   evalrouter run byob-<uuid>@1 --model openai/gpt-4.1-mini-2025-04-14 --full --max-cost 1

   Use --dry-run to see the quote without starting. For a private benchmark, the quote's warnings repeat what the model receives (system prompt, template and mapped columns), what it never receives, and the grader's match rule. Read the scores with evalrouter runs results RUN and export them with evalrouter runs export RUN, as described in results.

4. List your private benchmarks with evalrouter benchmarks private list, and stop new runs of one with evalrouter benchmarks private archive byob-<uuid>@1. Earlier runs and results are kept.

In the catalog, choose Source → Created by me to see the private benchmarks you created in the current workspace. Imports with the same name have separate benchmark IDs. Their creation dates help distinguish them.

Members, admins and owners can choose Delete on a private benchmark. Check its ID and creation date in the confirmation, then choose Delete benchmark. This archives only that benchmark, removes it from the active catalog and frees its benchmark slot. Existing runs and results remain available. The uploaded source is not erased and its source slot is not freed.

Free an uploaded source slot#

Every completed upload is an active source until you archive it, whether or not a benchmark uses it, and a workspace has 1,000 slots. Each benchmarks private create with a local data file (including --dry-run) uploads a new source. When they are all in use, a new upload is refused with private_benchmark_limit and "limit_kind": "upload_sources". Archiving a private benchmark does not free a source slot.

evalrouter benchmarks uploads list --status completed
evalrouter benchmarks uploads archive <upload_id>

benchmarks uploads list lists each upload's ID, status, size, rows, date and files, and how many of the 1,000 slots are used. benchmarks uploads archive accepts the upload ID or its workspace://<upload_id>/<file> source. Archiving is permanent: the source can no longer be named in a new manifest. Nothing is deleted: its files stay stored (and still count toward storage), and private benchmarks already created from it, and their runs, keep working. To reuse a source you still need instead of uploading it again, name it in the manifest as "source": "workspace://<upload_id>/<file>" in place of data.

A benchmark version never changes. To change the data, template or grader, create a new benchmark.

From a Hugging Face dataset#

Instead of uploading a file, you can build a private benchmark from a Hugging Face dataset your workspace has imported (see Benchmarks from Hugging Face datasets). This needs Hugging Face imports to be enabled where you run it. Otherwise an hf:// source is refused. In the browser, enter the dataset page URL (https://huggingface.co/datasets/<org>/<dataset>), its ID (<org>/<dataset>) or hf://<org>/<dataset>@<revision>. A URL or ID uses the main branch. The CLI and API take hf://<org>/<dataset>@<revision>. That page also explains private datasets and read tokens.

1. Import it. The import pins the revision to a commit, checks each file's hash and the dataset's license, and keeps a copy private to your workspace:

   sh

   Copy

   evalrouter benchmarks imports huggingface hf://<org>/<dataset>@main --split test --json

   The response lists each file as hf://<org>/<dataset>@<commit>/<path> and its sha256 and bytes in the receipt. With a mapping (--grader, --prompt-field, --answer-field), it also returns a declaration. Its default template is {prompt}. For multiple_choice it is {prompt}, a blank line, {choices}, a blank line, then Answer with one letter as "ANSWER: <letter>". Whatever template you put in the manifest is delivered verbatim, including that line if you keep it.

2. Name that file in the manifest instead of data:

   json

   Copy

   {
     "name": "Imported QA (example)",
     "source": "hf://<org>/<dataset>@<commit>/data/test-00000-of-00001.jsonl",
     "source_sha256": "<sha256 from the receipt>",
     "source_bytes": 12345,
     "task": "exact_match",
     "columns": {"question": "question", "answer": "answer"},
     "prompt_template": "{question}\nAnswer briefly.",
     "rights_attestation": true
   }

3. Create and run it exactly as above, with evalrouter benchmarks private create manifest.json.

Only a full 40-character commit is accepted, and only a file your own workspace imported: another workspace's import, a different sha256 or size, or a tag or branch name is refused. The same size, row, rights and privacy rules apply as for uploads.

From a GitHub data file#

GitHub imports are available only to enabled workspaces. When unavailable, the GitHub source is hidden. A saved draft or direct link keeps your settings and offers file upload or Hugging Face instead. Publishing this guide does not enable access.

1. In Import a benchmark, choose GitHub. Use Public for a repository URL or owner/repository, or Private to select one connected private repository. Owners and admins connect or disconnect accounts in Settings → Integrations. Set repository permissions on GitHub. Connecting an account makes its granted repositories available in the importer. No personal access token is needed.
2. Files load automatically from the default branch. Expand folders in the file tree to select one CSV, JSONL or Parquet file, or filter by typing a path. You can also paste an exact file path. Use the arrow keys to move through files and expand or collapse folders, and Enter or Space to select a file. To use another revision, choose a branch, tag or recent commit from the dropdown. You can also enter a reference. Files can be up to 16 MiB and within your workspace limits. The displayed commit is frozen for the import, even if the branch later moves.
3. Map the columns, choose one of the built-in graders, confirm your data rights, then validate and create the benchmark. The verified file and benchmark stay private to your workspace, including data from a public repository.

This imports data for your chosen prompts and scoring rules. It does not run repository code, install dependencies, or reproduce an author's custom evaluation protocol. JSON arrays, Git LFS pointers, symlinks and submodules are not supported. Upload the underlying supported data file instead.

evalrouter benchmarks imports capabilities (or the capability endpoint below) shows whether GitHub imports are available to your workspace. Where they are, evalrouter benchmarks imports github-refs, github-preview and github follow the same steps as the browser.

Privacy and isolation#

• A private benchmark and its upload are visible only inside your workspace. Anyone else asking for its ID or upload gets "not found", never "forbidden", so an ID reveals nothing.
• It never appears in the shared catalog, cannot be added to it, and its results cannot be shared through a public link.
• Your file is stored once per upload under your workspace's private storage and read back by its SHA-256 before every run. Identical files in two workspaces are never deduplicated or compared.
• Rows are sent to the model provider of each run you start, like any benchmark prompt.

Rights and terms#

In the browser, choose the dataset’s existing license. Use Private or proprietary for data governed by your own private terms, a named license when it applies, or Other license to enter its identifier. This records the existing terms. It does not change them or make the benchmark public. The CLI records this choice in data_license. Hugging Face imports preserve the verified source license.

By setting rights_attestation: true you confirm: "This workspace has the right to use this data for evaluation and to send its rows to the model providers selected for each run. EvalRouter keeps it private to this workspace and never publishes, shares or reuses it." The statement and your data_license are recorded with the benchmark and shown in every result's notices. Do not upload data you are not allowed to process this way, and do not upload personal data you have no basis to send to model providers.

Limits and safety#

The 64 MiB limit applies to the file a benchmark is built from. An upload or a Hugging Face import can be larger, up to your workspace's upload limit, but creating a private benchmark from a file over 64 MiB is refused with private_benchmark_too_large.

Uploads and imported GitHub and Hugging Face files share your workspace's storage allowance. A GitHub import reserves its bytes before storage is written. Retrying the same import after a storage failure reuses that reservation. Archiving a benchmark or completed upload does not free those stored bytes.

Uploaded content is treated as untrusted data. Nothing in it is executed: prompts are rendered with plain placeholders and answers are scored by the built-in graders only. Previews and error messages replace credential-shaped strings, such as API keys, tokens and private keys, with [redacted] and warn you. They are still part of your data and are still sent to the model, so remove real credentials from the file.

HTTP API#

Not supported yet#

• Model-graded (judge) scoring other than Tune rubric grading, partial credit, few-shot examples and more than one attempt per row.
• Custom grader code.