Benchmarks from Hugging Face datasets

Use a Hugging Face dataset as benchmark data. EvalRouter resolves the dataset to an exact commit, checks the files and license, and stores a copy. Runs read that copy, so later Hub changes cannot alter an earlier benchmark version.

"Public" and "private" below describe who can read the dataset on Hugging Face. They do not change who can see the benchmark: a benchmark you import is always private to your workspace.

Status#

An import does not reproduce the dataset author's benchmark by itself. You choose prompts, reference answers, and a built-in grader. For the author's established protocol, use a maintained catalog benchmark.

Naming a dataset#

Only a dataset page is accepted. A model page, a link to one file, a dataset viewer link or a URL with a query or # fragment is refused before anything is fetched.

A pinned reference names a revision: a full commit, a tag or a branch. A URL or ID has no revision, so the browser uses the dataset's main branch. The CLI and the API always need @<revision>. Whatever you name, EvalRouter resolves it to one commit when it fetches the data, and stored files and receipts name only that commit. Importing a moved branch or tag later creates a new source instead of changing the old one.

You then choose which file to use:

• a config (when the dataset has several) and a split, for example --config abstract_algebra --split test. We use the files the dataset card assigns to that split, or the Hub's usual layout (<config>/<split>-00000-of-00001.parquet, data/<split>-...);
• or explicit file paths in the repository. A * matches within one folder.

Only Parquet, JSON Lines and CSV files are accepted. Nothing else in the repository is downloaded, and nothing from it is executed: no dataset loading script, no custom code and no grader from the repository.

Import in the browser#

Open Import a benchmark and choose Hugging Face. Search by dataset or publisher name, or enter a dataset ID or page URL in the same input. Select the public dataset to pin its commit and browse its folders. Choose one .jsonl, .csv or .parquet file; Next and Previous move through large folders, and the breadcrumbs return to parent folders. The filter searches the current page. Typing a folder path ending in / opens that folder.

Use Preview selected file to inspect CSV or JSONL columns and sample rows at the displayed commit. For Parquet, enter column names in Scoring and review the server's sample during validation. Source and access options contains optional config, split and exact-file inputs, plus a read token for private or gated data. Public discovery does not use that token; private or gated sources use the manual source and file path.

Map the prompt and answer columns, choose a grader, then click Import and validate. Review the returned rows, shown exactly as the model receives them, and which columns are never sent, before Create benchmark. The browser needs no EvalRouter API key or manifest. Public browsing and samples help choose a file; dataset access, file integrity, license and every row are checked by the server when you validate (see Two checks).

Example: 100 MMLU questions#

Use this public dataset to try a small multiple-choice benchmark:

For the exact data used in this example, enter hf://cais/mmlu@c30699e8356da336a370243923dbaf21066bb9fe. This split contains 100 rows. EvalRouter's built-in multiple-choice grader scores the answers; this setup does not reproduce the original MMLU evaluation protocol. For an established protocol, choose a maintained catalog benchmark that matches the evaluation you need.

Gated and private datasets#

A gated or private dataset needs a Hugging Face token that can read it. For a gated dataset, first accept its terms on huggingface.co with the account that owns the token.

Put your own data in a private dataset#

1. On huggingface.co, create a new dataset repository (New Dataset) under your account or organization and set its visibility to Private.

2. Upload one .csv, .jsonl or .parquet file, with one row per question and a column for the reference answer: Files and versions → Add file → Upload files, or hf upload <owner>/<name> ./test.jsonl test.jsonl --repo-type dataset.

3. Add a license to the dataset card (README.md), in its YAML header, for example:

   yaml

   Copy

   ---
   license: cc-by-4.0
   ---

   It must be one of the accepted licenses. A private dataset needs one too.

Then import it as https://huggingface.co/datasets/<owner>/<name>, <owner>/<name> or hf://<owner>/<name>@main, and name the uploaded file's path (for example test.jsonl).

Create a read token for that repository#

Use the smallest token that works. In your Hugging Face Settings → Access Tokens, create a fine-grained token. Under repository permissions, select only this dataset repository and give it read access to its contents. It needs no write permission, and no access to other repositories, models, Spaces or your organization settings. Revoke it on Hugging Face when you no longer need it.

Use the token#

• Browser: paste the token when asked. It is sent with that import only. It is not saved in your draft, so enter it again if you reload the page.

• CLI: pass it without putting it on the command line:

  bash

  Copy

  evalrouter benchmarks imports huggingface hf://<org>/<private-dataset>@main --split test --hf-token-env HF_TOKEN
  printf '%s\n' "$HF_TOKEN" | evalrouter benchmarks imports huggingface hf://<org>/<private-dataset>@main --split test --hf-token-stdin

EvalRouter uses the token only to read that dataset from huggingface.co, for that one import. We never store it, and it never appears in the receipt, the benchmark, our logs or an error message.

Without a token, a gated dataset is refused with token_required (403). A private dataset is invisible to the Hub without one, so it is refused as not found, access_refused (404), exactly like a dataset that does not exist. A token that cannot read the dataset, for example because its terms were not accepted, also gets access_refused.

Licenses#

The dataset card must report a license at the resolved commit, and every license it reports must be one of MIT, Apache-2.0, BSD-2-Clause, BSD-3-Clause, CC-BY-4.0, CC-BY-SA-4.0 or CC0-1.0. A card that reports no license, other, unknown, or a non-commercial or no-derivatives license is refused with license_refused, and the message lists what is accepted. This applies to private datasets too: a token grants access, not a license. The catalog also accepts a few other permissive licenses, and it can accept a card that reports other only after a separate license review for those exact files.

What can be evaluated#

An imported file becomes a benchmark only if its rows fit a built-in grader: one prompt from text columns, one reference answer, one model reply per row, scored by exact_match, multiple_choice, numeric, regex_match or json_field_match.

A generic import keeps the data, not the author's evaluation. It does not preserve a reinforcement-learning or agent environment, the dataset's original or custom graders, reward functions or verifiers, a loading script, or any other code in the repository. Every imported benchmark is scored by one of the five built-in graders.

Mapped prompt columns must hold text or numbers in every row. A column holding a list of chat messages or a nested object is refused, and a dotted path such as reward_model.ground_truth does not reach inside one. Images, audio, multi-turn conversations, tool use, agent environments, model-graded scoring and a dataset's own reward functions or verifiers are not supported by self-service import.

Datasets are judged by their rows and the evaluation they need, never by their name or a tag. Some reinforcement-learning datasets store text questions with plain reference answers and import as usual. Others are built for agent environments: their rows hold chat-message prompts, a reward-model block and environment metadata, and they are scored by the author's custom verifiers. Such a file can pass the source import, but its rows are refused at validation, for example row 1 column 'prompt' must be text or a number, not list. Flattening it into a question and an answer would not reproduce the author's evaluation, so it cannot be run as that benchmark here. Describe it in a catalog request instead.

Two checks: source, then rows#

An import is not yet a runnable benchmark. It passes two separate checks.

1. Source import. Resolve the revision, select the files, check access and the license, then download and verify every file. These refusals happen here, before any row is read:

2. Row validation. EvalRouter checks every row, not just the preview, against your column mapping and grader. With a mapping, evalrouter benchmarks imports huggingface refuses a problem as invalid_mapping. evalrouter benchmarks private create --dry-run, POST /v1/private-benchmarks with "dry_run": true and the browser's Import and validate refuse it as invalid_private_benchmark. Both are 422 and name the row and the reason. Examples: a missing column, an empty reference, a duplicate ID, a numeric answer that is not a number, a choice list that is not a list, or a prompt column holding a list. A file with more rows than allowed is refused with import_row_limit or upload_row_limit, and a file over 64 MiB with private_benchmark_too_large.

Nothing is created and no model is called until both checks pass and you create the benchmark. Once created, it runs like any benchmark, using available workspace credit.

After you create it#

Create benchmark adds the benchmark to your workspace's private catalog. It appears in the catalog and in the benchmark list for new evaluations, marked as belonging to your workspace, and only members of this workspace can see or run it. It is never published, listed for other workspaces or shared with them, even when the Hugging Face dataset is public.

The page shows the benchmark ID and row count. Choose Configure an evaluation to run it on a compatible chat model. The benchmark version never changes: it keeps the pinned commit, file hash, column mapping and grader. To change any of them, create a new benchmark. Results are labeled as scored by an EvalRouter built-in grader and are comparable only to other runs of the same benchmark ID.

Choose Source → Created by me in the catalog to find your imports in the current workspace. Same-name imports remain separate benchmarks. Delete asks you to confirm the exact benchmark ID, then archives it and frees its benchmark slot while keeping earlier runs and results. See managing private benchmarks.

In the catalog#

A catalog manifest can name the dataset directly:

source:
  uri: hf://cais/mmlu@c30699e8356da336a370243923dbaf21066bb9fe
  config: abstract_algebra
  split: test
  files:
    - path: abstract_algebra/test-00000-of-00001.parquet
      sha256: 92338cdcdb43d2871691ef9a0e125e633ca57335ed635a635fe6dead94f224c2
      bytes: 9964

The manifest must pin the full commit and list every file of the split with its sha256 and size, so the review sees exactly what will run. Before anything is downloaded, we check the list against the Hub at that commit: each file must match, and a file of that split that the manifest leaves out is refused. The rest of the manifest (task, grader, license, qualification) is the same as for any catalog benchmark.

Into your workspace with the CLI#

evalrouter benchmarks imports huggingface hf://<org>/<dataset>@main --config <config> --split test \
  --prompt-field question --answer-field answer --grader exact_match

The import runs on our side: the CLI uploads nothing. It prints the commit the revision resolved to, the license and each stored file's sha256. With --prompt-field, --answer-field and --grader (and --choices-field for multiple_choice), we also run the row validation above. Then name the stored file in a manifest to create the benchmark; see From a Hugging Face dataset. Results are labeled as scored by an EvalRouter built-in grader, not the benchmark author's. Imported data is private to your workspace and is never shared with, or deduplicated against, another workspace.

What we check and keep#

1. The commit. The Hub must resolve the revision to a commit. A full commit hash must resolve to itself.
2. Every file's hash. The Hub lists each file's size and, for files stored with Git LFS, its sha256 at that commit. We download only the selected files and refuse any whose size or sha256 differs. A small file stored directly in Git is checked against its Git blob ID, and we record its sha256.
3. The license and access, as described above.

We then store each file under its sha256, never overwriting a stored copy, and keep a fetch receipt: the dataset, the revision you asked for and the commit it resolved to, each file with its size and sha256, the card's license, and whether the dataset was gated or private. Importing the same selection again downloads nothing and returns the same receipt.

Limits#

• One import holds at most your workspace's upload limits (files, bytes and rows).
• A private benchmark is built only from a file of at most 64 MiB. A larger imported file stays in your workspace, but creating a benchmark from it is refused with private_benchmark_too_large.
• A built-in grader reads exactly one file; name it with --file when a split has several.
• Datasets that need a loading script, archives and other formats are not supported.