Bring your own benchmark

EvalRouter runs benchmarks from a reviewed catalog. You can ask us to add a public benchmark to that catalog. This page explains what a benchmark needs, what we check, and which options are available today.

Options and status

A catalog addition is visible to every workspace. Do not request one for data you cannot share publicly.

What a benchmark needs

• Public data pinned exactly. The data can be in one of these places:

  • a Hugging Face Hub dataset, identified by its full 40-character commit hash;
  • a GitHub repository, identified by its full 40-character commit hash. Files stored with Git LFS are fine;
  • a GitHub release asset, named by the release tag and the file name. The tag only finds the file; its sha256 is what we check;
  • a public HTTPS URL for a single file. It must not need a login or a signed link, and it cannot point to a private or internal network address.

  For Hugging Face and GitHub repositories, branch names, tags, main and latest are not accepted, because their contents can change. If you only know a tag or branch, send it and we will pin the commit it points to at review time. Whatever the host, we download each file once and keep our own copy. If the host changes the file later, the catalog version keeps the copy we checked.

• Every file listed with its sha256 and size. That includes companion files the task loader needs, such as other splits. We refuse a file whose bytes do not match.

• An existing task. The benchmark must match a task that already exists in lm-evaluation-harness or inspect_evals at the revision EvalRouter's runners pin. We call the task author's code directly. If the task needs changes to run your data, it cannot be added yet. Data hosted on GitHub or at an HTTPS URL works only with a task that loads its data from a file URL. Tasks that read a Hugging Face dataset, which includes lm-evaluation-harness tasks, need their data on the Hub.

• The author's grader and metrics. Scoring uses the task's native metric at that same pinned revision. You cannot supply your own scoring code.

• An allowed data license. One of MIT, Apache-2.0, BSD-2-Clause, BSD-3-Clause, CC-BY-4.0, CC-BY-SA-4.0 or CC0-1.0. The license the Hugging Face Hub or GitHub reports at the pinned commit or release must agree with it. A plain HTTPS URL reports no license, so send a link to where the license is granted; we review it and keep a copy of the notice. Anything else needs a separate license review and may be declined.

Example manifest

We record each catalog benchmark in a manifest. This is a trimmed version of the one for COPA:

slug: copa
name: "COPA: SuperGLUE validation (complete 100)"
category: Commonsense
description: All 100 SuperGLUE validation questions, scored with native causal-continuation likelihoods.

source:
  uri: hf://aps/super_glue@3de24cf8022e94f4ee4b9d55a6f539891524d646/copa/validation-00000-of-00001.parquet
  sha256: 247a7f586c5cc706eb18a15ad20f1a5b543b9c7af0df4161df138a25edc95d67
  bytes: 11975

adapter:
  runner: lm-eval
  entrypoint: super_glue/copa/default.yaml
  split: validation
  capability: continuation_loglikelihood

grader:
  uri: github://EleutherAI/lm-evaluation-harness@b954108c9baaaa934b4ad842033b31a97ee30816/lm_eval/tasks/super_glue/copa/default.yaml
  kind: author-native
  metrics: [acc]

sampling:
  generation: greedy
  max_output_tokens: 0
  attempts_per_task: 1

license:
  data: BSD-2-Clause
  code: MIT
  url: https://asgordon.github.io/copa.html

qualification:
  parity: {mode: full, rows: 100}
  failure_checks: [invalid_output, timeout, unsupported_endpoint, grader_error]

You do not write a version number. We assign one. If any of the data, task, grader, sampling or license changes, the benchmark gets a new version and the previous version keeps working, so earlier results stay comparable to what they measured.

What happens after you ask

1. We review the request: pins, license and whether an existing task expresses it.
2. We fetch the pinned files and refuse any whose size or sha256 does not match.
3. We store the verified files. Runs read those stored copies, not the live source.
4. We qualify it: our run must reproduce the native task's results on the stated rows, and invalid output, timeouts, unsupported model endpoints and grader errors must be recorded as failures rather than scores.
5. If it qualifies, it is added to the catalog as a new benchmark version. It is tested in our staging environment first, and production availability follows separately.

A request can be declined at any step. We tell you which requirement it did not meet.

Request an addition

Email info@kimpton.ai with:

• where the data is (Hugging Face dataset, GitHub repository or release, or HTTPS URL) and the commit, tag or URL you want;
• the lm-eval or Inspect task it matches, and the split to score;
• the data license and a link to where it is granted.

Once it is listed, run it like any other benchmark from the CLI quickstart.

Coming next

• Private benchmarks in your workspace, visible only to you, with the same pinning, license check and sandboxed qualification.
• Bring your own data per run. You send the data or a link with its sha256 for one run. We verify it, run it in a sandbox and delete it when the run ends; the result records only the data's hash.
• Local files, as part of bring your own data.
• Suggested graders. When your data names a known task or has a standard shape, such as multiple choice or a single reference answer, we propose a grader for you to confirm.

None of these are available yet.