# Bring your own benchmark

You can run your own data as a private benchmark or ask us to add a maintained benchmark to the public catalog. The choice depends on whose grading rules you need.

## Options and status

| What you want to do | Availability | Start here |
| --- | --- | --- |
| Run a maintained benchmark | Available | Choose a compatible model in the [catalog](/catalog), or run `evalrouter benchmarks list`. |
| Upload your own dataset | Available | [Upload a file](/catalog/import?source=file): JSON Lines, CSV or Parquet. Define prompts and scoring; no manifest is needed in the browser. |
| Import a Hugging Face dataset | Available for supported files and licenses | [Import from Hugging Face](/catalog/import). Map the columns and choose a built-in grader. [Requirements](/developers/docs/huggingface). |
| Import a gated or private Hugging Face dataset | Requires authorized source access and a supported license | Provide a Hugging Face token with access to the dataset. The imported benchmark stays private to your workspace. |
| Keep a benchmark private to your workspace | Available | All [self-service imports](/developers/docs/byob) are workspace-private. |
| Use built-in scoring | Available | Choose exact match, multiple choice, numeric, regular expression or JSON fields. See [graders](#built-in-graders). |
| Add a public benchmark to the maintained catalog | Reviewed by request | [Send a catalog request](#request-an-addition) with the source, evaluation definition and license. |
| Import a GitHub data file | Enabled workspaces only | [Import a CSV, JSONL or Parquet file](/developers/docs/byob#from-a-github-data-file), up to 16 MiB, from a public repository or a selected private GitHub App repository. |
| Import directly from another HTTPS URL | Catalog request only | Include the source URL in your request. For a supported data file you already have, use file upload. |
| Run custom grader code or an arbitrary evaluation repository | Not available through self-service import | Describe the required grader in a catalog request. Uploads do not execute code. |
| Supply data for one run without retaining a benchmark | Not available | Create a private benchmark, then archive it to stop future runs. Archiving is not immediate deletion. |

A catalog addition is visible to every workspace. A private import stays in yours and does not change your data's license.

## What a public catalog addition needs

- **Public data pinned exactly.** The data can be in one of these places:
  - a Hugging Face Hub dataset, identified by its full 40-character commit hash;
  - a GitHub repository, identified by its full 40-character commit hash. Files stored with Git LFS are fine;
  - a GitHub release asset, named by the release tag and the file name. The tag only finds the file; its sha256 is what we check;
  - a public HTTPS URL for a single file. It must not need a login or a signed link, and it cannot point to a private or internal network address.

  For Hugging Face and GitHub repositories, branch names, tags, `main` and `latest` are not accepted, because their contents can change. If you only know a tag or branch, send it and we will pin the commit it points to at review time. Whatever the host, we download each file once and keep our own copy. If the host changes the file later, the catalog version keeps the copy we checked.
- **Every file listed with its sha256 and size.** That includes companion files the task loader needs, such as other splits. We refuse a file whose bytes do not match.
- **An existing task.** The benchmark must match a task in [lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness) or [inspect_evals](https://github.com/UKGovernmentBEIS/inspect_evals) at the revision EvalRouter's runners pin. We run the author's task code. If that code must change to use your data, the benchmark cannot be added yet.

  For data on GitHub or at an HTTPS URL, we give the framework's file loader our verified copy of your files. It supports Parquet, JSON Lines, JSON, or CSV; Inspect's Hugging Face reader supports JSON Lines, JSON, or CSV only. Prompts, answer extraction, and metrics stay as the author wrote them. The new benchmark version records the changed data location and every file's sha256.

  This path works for one lm-evaluation-harness task that reads one plain Hugging Face dataset, or an Inspect task already in our catalog that uses Inspect's Hugging Face reader. Name each file for its split, such as `train.jsonl` or `test-00000-of-00001.parquet`; the main file is the scored split. We decline task groups, loading scripts, custom dataset code, row shuffling on read, and tasks that require their exact Hub dataset. Those datasets need to be on the Hub.
- **The author's grader and metrics.** Scoring uses the task's native metric at that same pinned revision. You cannot supply your own scoring code.
- **An allowed data license.** One of MIT, Apache-2.0, BSD-2-Clause, BSD-3-Clause, CC-BY-4.0, CC-BY-SA-4.0 or CC0-1.0. The license the Hugging Face Hub or GitHub reports at the pinned commit or release must agree with it. A plain HTTPS URL reports no license, so send a link to where the license is granted; we review it and keep a copy of the notice. Anything else needs a separate license review and may be declined.

## Example manifest

We record each catalog benchmark in a manifest. This is a trimmed version of the one for COPA:

```yaml
slug: copa
name: "COPA"
category: Commonsense
description: All 100 SuperGLUE validation questions, scored with native causal-continuation likelihoods.

source:
  uri: hf://aps/super_glue@3de24cf8022e94f4ee4b9d55a6f539891524d646/copa/validation-00000-of-00001.parquet
  sha256: 247a7f586c5cc706eb18a15ad20f1a5b543b9c7af0df4161df138a25edc95d67
  bytes: 11975

adapter:
  runner: lm-eval
  entrypoint: super_glue/copa/default.yaml
  split: validation
  capability: continuation_loglikelihood

grader:
  uri: github://EleutherAI/lm-evaluation-harness@b954108c9baaaa934b4ad842033b31a97ee30816/lm_eval/tasks/super_glue/copa/default.yaml
  kind: author-native
  metrics: [acc]

sampling:
  generation: greedy
  max_output_tokens: 0
  attempts_per_task: 1

license:
  data: BSD-2-Clause
  code: MIT
  url: https://asgordon.github.io/copa.html

qualification:
  parity: {mode: full, rows: 100}
  failure_checks: [invalid_output, timeout, unsupported_endpoint, grader_error]
```

| Block | Meaning |
| --- | --- |
| `source` | The data file, pinned to a commit (or a GitHub release asset or HTTPS URL), with its sha256 and size. |
| `adapter` | The runner (`lm-eval` or Inspect), the existing task, the split that is scored and the kind of model call it needs. |
| `grader` | The task author's grader, pinned to a commit, and the metrics it reports. |
| `sampling` | How the model is called: generation mode, output token limit and attempts per task. |
| `license` | The data and code licenses and where the grant is published. |
| `qualification` | How much of the benchmark must reproduce the native task's results, and which failure cases are exercised. |

You do not write a version number. We assign one. If any of the data, task, grader, sampling or license changes, the benchmark gets a new version and the previous version keeps working, so earlier results stay comparable to what they measured.

## Built-in graders

Self-service [private benchmarks](/developers/docs/byob) use the five graders below. Maintained catalog benchmarks can use specialized graders from their evaluation definitions, so this list is not a limit on every benchmark in EvalRouter.

Use built-in graders for a JSON Lines, CSV, or Parquet file with one question per row. Map the question, choices, answer, and optional row ID. In the browser, write a prompt with `{question}` and, for multiple choice, `{choices}`. CLI manifests can map more fields. Validation refuses missing or mistyped columns, empty rows, and duplicate IDs.

| Grader | What counts as correct |
| --- | --- |
| `multiple_choice` | The model's letter (or the exact text of one choice) matches the answer, given as a letter, an index or the choice text. |
| `exact_match` | The model's answer equals a reference answer after the normalization you declare: letter case, whitespace and punctuation. |
| `regex_match` | Your declared pattern matches the model's answer, anchored to the whole answer by default, and the captured text equals the reference. |
| `numeric` | The number extracted from the answer equals the reference, within the absolute or relative tolerance you declare. |
| `json_field_match` | The answer parses as JSON and each field you name equals the reference. |

Each row scores 1 or 0. Results show accuracy and standard error. The label names EvalRouter's built-in grader, not an author's grader. Compare only runs with the same grader, mapping, and prompt.

## What happens after you ask

1. We review the request: pins, license and whether an existing task expresses it.
2. We fetch the pinned files and refuse any whose size or sha256 does not match.
3. We store the verified files. Runs read those stored copies, not the live source.
4. We qualify it: our run must reproduce the native task's results on the stated rows, and invalid output, timeouts, unsupported model endpoints and grader errors must be recorded as failures rather than scores.
5. Once qualified and published, it appears as a versioned benchmark in the catalog. Only compatible models can run it.

A request can be declined at any step. We tell you which requirement it did not meet.

## Request an addition

Email [info@kimpton.ai](mailto:info@kimpton.ai) with:

- where the data is (Hugging Face dataset, GitHub repository or release, or HTTPS URL) and the commit, tag or URL you want;
- the lm-eval or Inspect task it matches, and the split to score;
- the data license and a link to where it is granted.

Once it is listed, run it like any other benchmark from the [CLI quickstart](/developers/docs/quickstart).