Benchmarks from Hugging Face datasets

Import a Hugging Face dataset as a benchmark private to your workspace. EvalRouter pins the dataset to an exact commit and checks its files and license. Runs read EvalRouter's own copy, so later Hub changes can't alter an earlier version.

On this page

Name the dataset and file
Import in the browser
Create the benchmark with the CLI
Gated and private datasets
Licenses
What can be evaluated
Two checks: source, then rows
After you create it
Limits

"Public" and "private" below describe who can read the dataset on Hugging Face. A benchmark you import is always private to your workspace. An import keeps the data, not the author's evaluation: you choose prompts, answers and a built-in grader. For an author's established protocol, use a maintained catalog benchmark or request an addition.

Name the dataset and file#

Form	Example	Where
Dataset page URL	`https://huggingface.co/datasets/cais/mmlu`	Browser
Dataset ID	`cais/mmlu`	Browser
Pinned reference	`hf://cais/mmlu@main` or `hf://cais/mmlu@<40-character commit>`	Browser, CLI and API

Only a dataset page is accepted. A model page, a single-file link, a viewer link or a URL with a query or fragment is refused before anything is fetched. A URL or ID uses the main branch. The CLI and API always need a revision after @: a commit, tag or branch. EvalRouter resolves it to one commit, and stored files and receipts name only that commit. Importing a moved branch later creates a new source.

Then choose a config and split (--config abstract_algebra --split test. We use the files the dataset card assigns, or the Hub's usual layout), or explicit file paths (* matches within one folder). Only Parquet, JSON Lines and CSV are accepted. Nothing else is downloaded and nothing is executed: no loading script, custom code or repository grader.

Import in the browser#

Open Import a benchmark and choose Hugging Face. Search by dataset or publisher, or enter an ID or page URL, and select a public dataset to pin its commit.
Browse its folders and choose one JSON Lines, CSV or Parquet file. Preview selected file shows CSV or JSONL columns and sample rows. For Parquet, enter column names in Scoring and review the server's sample at validation. Source and access options holds config, split, exact file and a read token for private or gated data.
Map the prompt and answer columns, choose a grader and click Import and validate. Review the returned rows exactly as the model receives them, and which columns are never sent, then Create benchmark.

No EvalRouter API key or manifest is needed. The server checks access, file integrity, license and every row when you validate.

Example: 100 MMLU questions

Setting	Value
Dataset	`https://huggingface.co/datasets/cais/mmlu`, `cais/mmlu`, or `hf://cais/mmlu@main`
Config and split	`abstract_algebra`, `test`
File	`abstract_algebra/test-00000-of-00001.parquet`
Columns	prompt `question`, choices `choices`, answer `answer`
Grader	`multiple_choice`, answers are zero-based indexes

For the exact data, enter hf://cais/mmlu@c30699e8356da336a370243923dbaf21066bb9fe (100 rows). The built-in multiple-choice grader scores it. This doesn't reproduce the original MMLU protocol.

Create the benchmark with the CLI#

Import the dataset. The import runs on our side and the CLI uploads nothing. It pins the commit, checks hashes and license, and keeps a private copy:
Terminal
```
evalrouter benchmarks imports huggingface hf://<org>/<dataset>@main --config <config> --split test \
  --prompt-field question --answer-field answer --grader exact_match
```
The receipt lists each file as hf://<org>/<dataset>@<commit>/<path> with its sha256 and size. With the mapping flags (prompt, answer and grader, plus choices for multiple choice) it also validates the rows and returns a declaration:

Grader	Default template
Any	`{prompt}`
`multiple_choice`	`{prompt}`, a blank line, `{choices}`, a blank line, then `Answer with one letter as "ANSWER: <letter>".`

Whatever template your manifest holds is delivered verbatim.

Name that file in a manifest instead of data:

JSON

{
  "name": "Imported QA (example)",
  "source": "hf://<org>/<dataset>@<commit>/data/test-00000-of-00001.jsonl",
  "source_sha256": "<sha256 from the receipt>",
  "source_bytes": 12345,
  "task": "exact_match",
  "columns": {"question": "question", "answer": "answer"},
  "prompt_template": "{question}\nAnswer briefly.",
  "rights_attestation": true
}

Create and run it as for an uploaded file: evalrouter benchmarks private create manifest.json.

Only a full 40-character commit and a file your own workspace imported are accepted: another workspace's import, a different sha256 or size, or a tag or branch name is refused. Hugging Face imports must be enabled where you run, or an hf:// source is refused. The same size, row, rights and privacy rules apply as for uploads.

Gated and private datasets#

A gated or private dataset needs a Hugging Face token that can read it. For a gated dataset, first accept its terms on huggingface.co with the token's account.

Browser: paste the token when asked. It's sent with that import only and isn't saved in your draft.

CLI: keep it off the command line:

Terminal

evalrouter benchmarks imports huggingface hf://<org>/<private-dataset>@main --split test --hf-token-env HF_TOKEN
printf '%s\n' "$HF_TOKEN" | evalrouter benchmarks imports huggingface hf://<org>/<private-dataset>@main --split test --hf-token-stdin

EvalRouter uses the token only to read that dataset for that one import. It is never stored and never appears in the receipt, the benchmark, logs or error messages.

Case	Refusal
A gated dataset, no token	`token_required` (403)
A private dataset, no token	`access_refused` (404), exactly like a dataset that doesn't exist
A token that can't read it, for example terms not accepted	`access_refused` (404)

Put your own data in a private dataset, with a minimal token

On huggingface.co, create a dataset repository (New Dataset) and set it to Private.
Upload one CSV, JSON Lines or Parquet file with one row per question and a reference-answer column: Files and versions › Add file › Upload files, or hf upload <owner>/<name> ./test.jsonl test.jsonl --repo-type dataset.
Add an accepted license to the YAML header of the dataset card, for example license: cc-by-4.0. A private dataset needs one too.
In Settings › Access Tokens, create a fine-grained token with read access to this repository only: no write permission and no access to other repositories, models, Spaces or organization settings. Revoke it when you no longer need it.

Import it by its page URL, its ID or hf://<owner>/<name>@main, naming the uploaded file's path (for example test.jsonl).

Licenses#

The dataset card must report a license at the resolved commit, and every license it reports must be MIT, Apache-2.0, BSD-2-Clause, BSD-3-Clause, CC-BY-4.0, CC-BY-SA-4.0 or CC0-1.0. No license, an unknown or "other" license, or a non-commercial or no-derivatives license is refused with license_refused, listing what is accepted. This applies to private datasets too: a token grants access, not a license. The catalog accepts a few other permissive licenses, and "other" only after a separate review of those exact files.

What can be evaluated#

A file becomes a benchmark only if its rows fit a built-in grader. Each row needs one prompt from text columns, one reference answer and one model reply. Mapped prompt columns must hold text or numbers in every row. EvalRouter refuses a list of chat messages or a nested object. A dotted path such as reward_model.ground_truth doesn't reach inside one.

Not supported: images, audio, multi-turn conversations, tool use, agent environments, model-graded scoring, and a dataset's own graders, reward functions, verifiers or loading script. EvalRouter judges a dataset by its rows, not its name. An agent-environment dataset (chat-message prompts, a reward-model block, custom verifiers) passes the source import. Row validation then refuses its rows, for example row 1 column 'prompt' must be text or a number, not list. Send it as a catalog request instead.

Two checks: source, then rows#

Source import. Resolve the revision, select files, check access and license, then download and verify every file. These refusals happen before EvalRouter reads any row. Their codes are in Details below.
Row validation checks every row, not just the preview, against your mapping and grader. A refusal answers 422 and names the row and the reason:
- a missing column
- an empty reference
- a duplicate ID
- an answer that isn't a number where the grader needs one
- a choice list that isn't a list
- a prompt column that holds a list
The refusal codes are in Details below.

EvalRouter creates nothing and calls no model until both checks pass and you create the benchmark.

Refusal codes

Code	Status	Meaning
`invalid_source`, `invalid_selection`	422	The dataset reference, config, split or file path is malformed.
`revision_unresolved`	422	The branch, tag or commit does not exist.
`config_required`, `unknown_config`, `split_required`, `unknown_split`, `incomplete_split`	422	The config or split is missing, unknown, or its files are incomplete.
`file_not_found`, `unsupported_format`	422	The named file does not exist, is not Parquet, JSON Lines or CSV, or the selection mixes formats.
`file_refused`	422	The Hub lists no size or hash for a selected file, so it cannot be verified.
`too_many_files`, `import_too_many_files`	422	The selection has more files than one import or your workspace allows.
`license_refused`	422	The card's license is missing or not accepted.
`token_required`	403	A gated dataset (or one of its files) needs a token and none was given.
`access_refused`	404	The dataset does not exist, is private and no token was given, the token cannot read it, or it is disabled on the Hub.
`too_large`	413	The selection exceeds your workspace's upload limits.
`hash_mismatch`, `hub_unavailable`	502	A file did not match the Hub's hash, or huggingface.co could not be reached. Retry later.
`invalid_mapping`	422	Row validation during a Hugging Face import with a mapping.
`invalid_private_benchmark`	422	Row validation in a private benchmark dry run (CLI, API or Import and validate).
`import_row_limit`, `upload_row_limit`	422	The file has more rows than allowed.
`private_benchmark_too_large`	422	The file is over 64 MiB.

After you create it#

What we check and keep at import

The commit. The Hub must resolve the revision to a commit. A full hash must resolve to itself.
Every file's hash. We download only the selected files and refuse any whose size or sha256 (Git LFS) differs from the Hub's listing. We check a small file stored in Git against its blob ID and record its sha256.
The license and access, as above.

We store each file under its sha256 and never overwrite a stored copy. The fetch receipt records these facts:

the dataset, the requested revision and the resolved commit
each file's size and sha256
the card's license
whether the dataset is gated or private

Importing the same selection again downloads nothing and returns the same receipt.

After you create it, the benchmark appears in the catalog and the new-evaluation list, marked as your workspace's. Only members can see or run it, even when the dataset is public. Its version keeps the commit, file hash, mapping and grader forever. To change any of them, create a new benchmark. Results name an EvalRouter built-in grader, and you can compare them only with runs of the same benchmark ID. Manage and delete it as described in Upload a file.

Limits#

One import holds at most your workspace's upload limits (files, bytes and rows).
You can build a private benchmark only from a file of 64 MiB or less. A larger imported file stays in your workspace, but EvalRouter refuses a benchmark from it with private_benchmark_too_large.
A built-in grader reads exactly one file. Name it with --file when a split has several.
Datasets that need a loading script, archives and other formats aren't supported.