Bring your own benchmark

Turn your own data into a benchmark that stays private to your workspace, or ask us to add a maintained benchmark to the public catalog.

On this page

Choose a path
Request an addition

A private benchmark takes four steps:

Upload a file or import a dataset.
Map the prompt, the answer and the grader.
Validate every row.
Run the verified private copy by its benchmark ID.

Choose a path#

What you want	Availability	Start here
Run a maintained benchmark	Available	The catalog, or `evalrouter benchmarks list`.
Upload your own dataset	Available	Upload a file: JSON Lines, CSV or Parquet. No manifest needed in the browser.
Import a Hugging Face dataset	Supported files and licenses	Hugging Face datasets. Gated or private datasets need a token with access.
Import a GitHub data file	Enabled workspaces only	A CSV, JSONL or Parquet file up to 16 MiB, from a public repository or a connected private one.
Score with built-in rules	Available	Exact match, multiple choice, numeric, regular expression or JSON fields. See graders.
Add a public benchmark to the catalog	Reviewed by request	Request an addition with the source, task and license.
Import from another HTTPS URL	Catalog request only	Include the URL in your request, or upload a supported file you already have.
Run custom grader code or an evaluation repository	Not available	Describe the grader in a catalog request. Uploads never execute code.
Use data for one run without keeping a benchmark	Not available	Create a private benchmark, then archive it to stop future runs. Archiving is not immediate deletion.

Every self-service import is private to your workspace and doesn't change your data's license. A catalog addition is visible to every workspace.

Request an addition#

Email info@kimpton.ai with:

where the data is (Hugging Face dataset, GitHub repository or release, or HTTPS URL) and the commit, tag or URL
the lm-eval or Inspect task it matches, and the split to score
the data license and a link to the page that grants it.

We review each request. An accepted benchmark appears as a versioned catalog benchmark that compatible models can run. If we decline a request, we say which requirement it didn't meet. Any change to the data, task, grader, sampling or license creates a new version. Earlier versions keep working, so earlier results stay comparable.

What a catalog addition needs

Public data pinned exactly, in one of three forms. The first is a Hugging Face dataset or GitHub repository at a full 40-character commit (Git LFS files are fine). The second is a GitHub release asset by tag and file name (we check its sha256). The third is a public HTTPS URL for one file. The URL must need no login or signed link, and it can't be on a private or internal network. For repositories, we don't accept branches, tags, main or latest. If you send one, we pin its commit at review. We download each file once and keep our copy, so later host changes don't alter the catalog version.
Every file listed with its sha256 and size, including companion files the task loader needs, such as other splits. We refuse a file whose bytes don't match.
An existing task in lm-evaluation-harness or inspect_evals at the revision EvalRouter's runners pin. We run the author's task code unchanged. For data on GitHub or at an HTTPS URL, the framework's loader reads our verified copy in Parquet, JSON Lines, JSON or CSV. Inspect's Hugging Face reader takes JSON Lines, JSON or CSV. The version records the new location and each file's sha256. This works for one lm-eval task that reads one plain Hugging Face dataset. It also works for an Inspect task in the catalog that uses Inspect's Hugging Face reader. Name each file for its split, such as train.jsonl or test-00000-of-00001.parquet. The main file is the scored split. We decline task groups, loading scripts, custom dataset code, row shuffling on read and tasks that require their exact Hub dataset.
The author's grader and metrics at the same pinned revision. You can't supply scoring code.
An allowed data license: MIT, Apache-2.0, BSD-2-Clause, BSD-3-Clause, CC-BY-4.0, CC-BY-SA-4.0 or CC0-1.0, agreeing with what the Hub or GitHub reports at the pin. For a plain HTTPS URL, link to the page that grants the license. We review it and keep a copy of the notice. Any other license needs a separate review, and we may decline it.

Choose a path#

What you want	Availability	Start here
Run a maintained benchmark	Available	The catalog, or `evalrouter benchmarks list`.
Upload your own dataset	Available	Upload a file: JSON Lines, CSV or Parquet. No manifest needed in the browser.
Import a Hugging Face dataset	Supported files and licenses	Hugging Face datasets. Gated or private datasets need a token with access.
Import a GitHub data file	Enabled workspaces only	A CSV, JSONL or Parquet file up to 16 MiB, from a public repository or a connected private one.
Score with built-in rules	Available	Exact match, multiple choice, numeric, regular expression or JSON fields. See graders.
Add a public benchmark to the catalog	Reviewed by request	Request an addition with the source, task and license.
Import from another HTTPS URL	Catalog request only	Include the URL in your request, or upload a supported file you already have.
Run custom grader code or an evaluation repository	Not available	Describe the grader in a catalog request. Uploads never execute code.
Use data for one run without keeping a benchmark	Not available	Create a private benchmark, then archive it to stop future runs. Archiving is not immediate deletion.

Every self-service import is private to your workspace and doesn't change your data's license. A catalog addition is visible to every workspace.

Request an addition#

where the data is (Hugging Face dataset, GitHub repository or release, or HTTPS URL) and the commit, tag or URL

the lm-eval or Inspect task it matches, and the split to score

the data license and a link to the page that grants it.

What a catalog addition needs

Public data pinned exactly, in one of three forms. The first is a Hugging Face dataset or GitHub repository at a full 40-character commit (Git LFS files are fine). The second is a GitHub release asset by tag and file name (we check its sha256). The third is a public HTTPS URL for one file. The URL must need no login or signed link, and it can't be on a private or internal network. For repositories, we don't accept branches, tags, main or latest. If you send one, we pin its commit at review. We download each file once and keep our copy, so later host changes don't alter the catalog version.
Every file listed with its sha256 and size, including companion files the task loader needs, such as other splits. We refuse a file whose bytes don't match.
An existing task in lm-evaluation-harness or inspect_evals at the revision EvalRouter's runners pin. We run the author's task code unchanged. For data on GitHub or at an HTTPS URL, the framework's loader reads our verified copy in Parquet, JSON Lines, JSON or CSV. Inspect's Hugging Face reader takes JSON Lines, JSON or CSV. The version records the new location and each file's sha256. This works for one lm-eval task that reads one plain Hugging Face dataset. It also works for an Inspect task in the catalog that uses Inspect's Hugging Face reader. Name each file for its split, such as train.jsonl or test-00000-of-00001.parquet. The main file is the scored split. We decline task groups, loading scripts, custom dataset code, row shuffling on read and tasks that require their exact Hub dataset.
The author's grader and metrics at the same pinned revision. You can't supply scoring code.
An allowed data license: MIT, Apache-2.0, BSD-2-Clause, BSD-3-Clause, CC-BY-4.0, CC-BY-SA-4.0 or CC0-1.0, agreeing with what the Hub or GitHub reports at the pin. For a plain HTTPS URL, link to the page that grants the license. We review it and keep a copy of the notice. Any other license needs a separate review, and we may decline it.