# Private benchmarks from your own data Make a benchmark from your own data, private to your workspace. Upload a file, choose prompts and a built-in grader, and validate the rows. EvalRouter stores a verified copy and gives you a benchmark ID. Run it on a compatible chat model using available workspace credit. Use [Import a benchmark](/catalog/import) in the browser, or put the same settings in a small manifest for the CLI. File uploads and Hugging Face imports stay private to your workspace where imports are enabled. ## What you need - A data file: JSON Lines (`.jsonl`), CSV (`.csv`) or Parquet (`.parquet`), one question per row, at most 64 MiB and 100,000 rows. - Column mappings, a prompt template and one built-in grader. Set them in the browser, or use a JSON manifest with the CLI (YAML is also accepted when PyYAML is installed alongside the CLI). - Member, admin or owner access to the workspace. Viewers can read benchmarks but cannot import them. ## Import in the browser Open [Import a benchmark](/catalog/import) while signed in. Choose a file or Hugging Face dataset, or [GitHub](#from-a-github-data-file) when enabled for your workspace. Map columns, set scoring rules, and review a sample before creating the benchmark. The page uses your account and workspace. No API key is needed. Drop a file onto the source area or choose it from your computer. The page previews a small local sample and shows upload and verification progress. Your settings survive a refresh in the same tab for 24 hours. The draft does not save file contents or Hugging Face tokens. Reselect the file or re-enter the token when asked, then confirm data rights again. If the browser cannot save your draft, keep the tab open until the import is complete. In the CLI, use `evalrouter login`. If your device was signed in before benchmark importing was added, sign in again once to grant that permission. Existing device grants are never silently expanded. For CI and automation, create a workspace API key with `benchmarks:submit`, plus `eval:read` and `eval:write` to quote and run. Create it in [API key settings](/settings/api-keys) and set `EVALROUTER_API_KEY`. `eval:write` alone does not grant permission to import. ## Write a manifest for the CLI ```json { "name": "Capitals (synthetic example)", "data": "capitals.jsonl", "task": "exact_match", "columns": {"id": "id", "question": "country", "answer": "capital"}, "prompt_template": "What is the capital of {question}? Answer with the city name only.", "grader_options": {"extract": "last_line"}, "max_output_tokens": 16, "data_license": "LicenseRef-Workspace-Private", "rights_attestation": true } ``` | Key | Meaning | | --- | --- | | `name`, `description` | What people in your workspace see. | | `data` or `source` | `data` is a local file, relative to the manifest; the CLI uploads it first. `source` is an existing upload, `workspace:///`, from `evalrouter benchmarks uploads create`, or a Hugging Face file your workspace imported, `hf:///@/` (see [From a Hugging Face dataset](#from-a-hugging-face-dataset)). | | `source_sha256`, `source_bytes` | Required with an `hf://` source: the file's sha256 and size from the import receipt. | | `format` | Optional: `jsonl`, `csv` or `parquet`. Inferred from the file name. | | `task` | The grader: `multiple_choice`, `exact_match`, `regex_match`, `numeric` or `json_field_match`. | | `columns` | Maps roles to your column names. `answer` is required. `id` (a unique row ID) and `group` (a sampling stratum) are optional. `choices` is required for `multiple_choice` only. Every other role is a prompt placeholder. | | `prompt_template` | The user message, with `{role}` placeholders. Every mapped prompt column must appear; `id`, `answer` and `group` never can. `{choices}` renders lettered lines. `{{` and `}}` are literal braces. | | `system_prompt` | Optional system message. | | `grader_options` | Options for the chosen grader (below). Omitted options take their defaults, which are recorded in the benchmark. | | `max_output_tokens` | Output limit per answer, 1 to 4096 (default 256). | | `split` | Optional label for the scored rows (default `test`). | | `data_license` | A license identifier for your data, recorded with results (default `LicenseRef-Workspace-Private`). | | `rights_attestation` | Must be `true`. See [Rights and terms](#rights-and-terms). | ## How we interpret your dataset This is the whole contract between your file and the model. The dry run, the created benchmark (`delivery` in the API response) and every quote for it state the same thing for your benchmark. **What the model receives, per row: one or two messages, nothing else.** 1. Your `system_prompt`, verbatim, as a system message, if you set one. 2. Your `prompt_template` as the user message, with each `{role}` replaced by that row's value from the column you mapped to the role, and `{choices}` (multiple choice only) replaced by lettered lines `A. text`, `B. text`, and so on. EvalRouter adds no instructions, no answer-format hint, no examples and no other text. A mapped prompt value must be text or a number; a list or object is refused. **What the model never receives.** - `answer`: read to grade the row, never delivered. The template cannot contain `{answer}`. - `id` and `group`: used to identify and sample rows, never delivered. - Every column you did not map: not read at all and never delivered. It stays only in the stored copy of your file. **Answer guard.** A mapped prompt column is delivered, whatever its name. If a row's delivered messages contain that row's own answer, the benchmark is refused with `byob_answer_in_input`, naming the column (for example a `solution` or `explanation` column mapped as `{hint}`). The comparison uses Unicode NFKC, case folding and collapsed whitespace, and covers answers of at least 20 characters after that. Shorter answers, such as numbers, letters or "yes", occur in prompts by chance and are not checked, so make sure no mapped column carries them. A multiple-choice answer is a letter and is never checked: delivering the correct choice's text is the format. **How each row is run.** No few-shot examples (0 shots). One attempt per row, greedy, at temperature 0, with no automatic retry. Each answer is cut at `max_output_tokens`. No tools are offered. A rendered user message over 100,000 characters is refused at validation. **How each row is scored.** One [grader](#graders) decides 1 or 0 from the model's text output alone. Its options, including the defaults you did not set, are recorded with the benchmark and cannot change. ## Graders Self-service uploads use the five graders below. They support datasets with reference answers. They never execute uploaded Python code or arbitrary lm-eval or Inspect repositories. Maintained catalog benchmarks can use their own specialized grading implementations. Every grader is built into EvalRouter, deterministic and scores each row 1 or 0. Results report accuracy with its standard error and are labeled as scored by an EvalRouter built-in grader, not by any benchmark author's own grader. They are comparable only to other runs of the same benchmark ID. | Grader | Options (defaults) | How it extracts and matches | | --- | --- | --- | | `multiple_choice` | `answer_format`: `letter`, `index` or `text` (`letter`); `index_base`: `0` or `1` (`0`) | In order: the whole output is one letter, optionally in parentheses and followed by `.`, `)` or `:` (any case); otherwise the last `answer`, optional `is`, optional `:` or `-`, then an **uppercase** letter (`answer: B`, `The answer is (C)`; `answer is b` is not read); otherwise the one choice whose normalized text equals the whole output. Correct when that letter is the reference. A letter outside the row's choices scores 0. | | `exact_match` | `extract`: `full`, `last_line` or `answer_prefix` (`full`); `prefix` (`ANSWER:`); `normalize` | `full` takes the whole output, `last_line` the last non-blank line, `answer_prefix` the text after the last `prefix` (any case) up to the end of that line. Correct when it equals a reference, both normalized. `answer` may be a list of accepted answers (in CSV, a JSON list of strings). | | `regex_match` | `pattern` (required, up to 512 characters); `anchored` (`true`); `ignore_case`, `dot_all` (`false`); `normalize` | The pattern must match the whole stripped output (`anchored`) or anywhere in it. It takes the group named `answer`, else the only group, else the whole match. Correct when it equals a reference, both normalized. | | `numeric` | `extract`: `last_number`, `first_number`, `full` or `answer_prefix` (`last_number`); `prefix` (`ANSWER:`, for `answer_prefix`); `abs_tol`, `rel_tol` (`0`) | `last_number`/`first_number` take the last or first number in the output. `full` takes the whole output and `answer_prefix` the text after the last `prefix`; both then drop trailing periods and one trailing `%` and must be a single number. Numbers may have a sign, comma thousands separators, decimals and an exponent. Correct when equal as decimals, or within `abs_tol`/`rel_tol` when set. | | `json_field_match` | `fields` (required dotted paths); `extract`: `strict` or `first_object` (`strict`); `normalize_strings` (`false`) | `strict` parses the whole output; `first_object` the first fenced `json` block, else the first decodable JSON object or array. Correct when every named field equals the reference's value. Booleans never equal numbers. Strings are compared exactly unless `normalize_strings`. | `normalize` defaults to Unicode NFKC, case-folded and whitespace-collapsed comparison. Punctuation is kept unless you set `strip_punctuation` (punctuation becomes spaces). An unparseable or unmatched answer scores 0 with a recorded reason. Timeouts and provider errors are errors, never scores. **Tune rubric grading.** The API also accepts `"grader_protocol": "tune_rubric_v1"` with `"task": "exact_match"`, an `evaluator_id` (a rubric version in your workspace) and a managed `judge_route_id`. Delivery to the model is unchanged. Instead of the table above, a separately metered managed judge reads the rubric, the delivered user message, the reference answer and the output, and returns pass or fail; an abstention leaves the row inconclusive. ## Create and run it 1. Check the manifest against every row without creating anything. The data is uploaded and verified. Nothing is charged: ```sh evalrouter benchmarks private create manifest.json --dry-run ``` The response shows the row count, [how the dataset is interpreted](#how-we-interpret-your-dataset) (delivered and never-delivered columns, grader and match rule, examples, attempts and output limit), the first three rows exactly as the model receives them (system and user messages, untruncated), and any warnings. Answers are not shown, because they are never delivered. A mapping problem is refused with the row and reason, for example a missing column, a duplicate ID, an answer that is not a number, or a mapped column that carries the answer (`byob_answer_in_input`). 2. Create the benchmark: ```sh evalrouter benchmarks private create manifest.json ``` It prints the benchmark ID, `byob-@1`. 3. Run it like any benchmark, by its ID (or through the [quickstart request](/developers/docs/quickstart), replacing `BENCHMARK_PROFILE_ID` with it): ```sh evalrouter run byob-@1 --model openai/gpt-4.1-mini-2025-04-14 --full --max-cost 1 ``` Use `--dry-run` to see the quote without starting. For a private benchmark, the quote's warnings repeat what the model receives (system prompt, template and mapped columns), what it never receives, and the grader's match rule. Read the scores with `evalrouter runs results RUN` and export them with `evalrouter runs export RUN`, as described in [results](/developers/docs/results). 4. List your private benchmarks with `evalrouter benchmarks private list`, and stop new runs of one with `evalrouter benchmarks private archive byob-@1`. Earlier runs and results are kept. In [the catalog](/catalog), choose **Source → Created by me** to see the private benchmarks you created in the current workspace. Imports with the same name have separate benchmark IDs. Their creation dates help distinguish them. Members, admins and owners can choose **Delete** on a private benchmark. Check its ID and creation date in the confirmation, then choose **Delete benchmark**. This archives only that benchmark, removes it from the active catalog and frees its benchmark slot. Existing runs and results remain available. The uploaded source is not erased and its source slot is not freed. ### Free an uploaded source slot Every completed upload is an active source until you archive it, whether or not a benchmark uses it, and a workspace has 1,000 slots. Each `benchmarks private create` with a local `data` file (including `--dry-run`) uploads a new source. When they are all in use, a new upload is refused with `private_benchmark_limit` and `"limit_kind": "upload_sources"`. Archiving a private benchmark does not free a source slot. ```sh evalrouter benchmarks uploads list --status completed evalrouter benchmarks uploads archive ``` `benchmarks uploads list` lists each upload's ID, status, size, rows, date and files, and how many of the 1,000 slots are used. `benchmarks uploads archive` accepts the upload ID or its `workspace:///` source. Archiving is permanent: the source can no longer be named in a new manifest. Nothing is deleted: its files stay stored (and still count toward storage), and private benchmarks already created from it, and their runs, keep working. To reuse a source you still need instead of uploading it again, name it in the manifest as `"source": "workspace:///"` in place of `data`. A benchmark version never changes. To change the data, template or grader, create a new benchmark. ## From a Hugging Face dataset Instead of uploading a file, you can build a private benchmark from a Hugging Face dataset your workspace has imported (see [Benchmarks from Hugging Face datasets](/developers/docs/huggingface)). This needs Hugging Face imports to be enabled where you run it. Otherwise an `hf://` source is refused. In the browser, enter the dataset page URL (`https://huggingface.co/datasets//`), its ID (`/`) or `hf:///@`. A URL or ID uses the `main` branch. The CLI and API take `hf:///@`. That page also explains private datasets and read tokens. 1. Import it. The import pins the revision to a commit, checks each file's hash and the dataset's license, and keeps a copy private to your workspace: ```sh evalrouter benchmarks imports huggingface hf:///@main --split test --json ``` The response lists each file as `hf:///@/` and its `sha256` and `bytes` in the receipt. With a mapping (`--grader`, `--prompt-field`, `--answer-field`), it also returns a `declaration`. Its default template is `{prompt}`. For `multiple_choice` it is `{prompt}`, a blank line, `{choices}`, a blank line, then `Answer with one letter as "ANSWER: ".` Whatever template you put in the manifest is delivered verbatim, including that line if you keep it. 2. Name that file in the manifest instead of `data`: ```json { "name": "Imported QA (example)", "source": "hf:///@/data/test-00000-of-00001.jsonl", "source_sha256": "", "source_bytes": 12345, "task": "exact_match", "columns": {"question": "question", "answer": "answer"}, "prompt_template": "{question}\nAnswer briefly.", "rights_attestation": true } ``` 3. Create and run it exactly as above, with `evalrouter benchmarks private create manifest.json`. Only a full 40-character commit is accepted, and only a file your own workspace imported: another workspace's import, a different sha256 or size, or a tag or branch name is refused. The same size, row, rights and privacy rules apply as for uploads. ## From a GitHub data file GitHub imports are available only to enabled workspaces. When unavailable, the GitHub source is hidden. A saved draft or direct link keeps your settings and offers file upload or Hugging Face instead. Publishing this guide does not enable access. 1. In [Import a benchmark](/catalog/import?source=github), choose **GitHub**. Use **Public** for a repository URL or `owner/repository`, or **Private** to select one connected private repository. Owners and admins connect or disconnect accounts in [Settings → Integrations](/settings/integrations). Set repository permissions on GitHub. Connecting an account makes its granted repositories available in the importer. No personal access token is needed. 2. Files load automatically from the default branch. Expand folders in the file tree to select one CSV, JSONL or Parquet file, or filter by typing a path. You can also paste an exact file path. Use the arrow keys to move through files and expand or collapse folders, and Enter or Space to select a file. To use another revision, choose a branch, tag or recent commit from the dropdown. You can also enter a reference. Files can be up to **16 MiB** and within your workspace limits. The displayed commit is frozen for the import, even if the branch later moves. 3. Map the columns, choose one of the [built-in graders](#graders), confirm your data rights, then validate and create the benchmark. The verified file and benchmark stay private to your workspace, including data from a public repository. This imports data for your chosen prompts and scoring rules. It does not run repository code, install dependencies, or reproduce an author's custom evaluation protocol. JSON arrays, Git LFS pointers, symlinks and submodules are not supported. Upload the underlying supported data file instead. `evalrouter benchmarks imports capabilities` (or the capability endpoint below) shows whether GitHub imports are available to your workspace. Where they are, `evalrouter benchmarks imports github-refs`, `github-preview` and `github` follow the same steps as the browser. ## Privacy and isolation - A private benchmark and its upload are visible only inside your workspace. Anyone else asking for its ID or upload gets "not found", never "forbidden", so an ID reveals nothing. - It never appears in the shared catalog, cannot be added to it, and its results cannot be shared through a public link. - Your file is stored once per upload under your workspace's private storage and read back by its SHA-256 before every run. Identical files in two workspaces are never deduplicated or compared. - Rows are sent to the model provider of each run you start, like any benchmark prompt. ## Rights and terms In the browser, choose the dataset’s existing license. Use **Private or proprietary** for data governed by your own private terms, a named license when it applies, or **Other license** to enter its identifier. This records the existing terms. It does not change them or make the benchmark public. The CLI records this choice in `data_license`. Hugging Face imports preserve the verified source license. By setting `rights_attestation: true` you confirm: "This workspace has the right to use this data for evaluation and to send its rows to the model providers selected for each run. EvalRouter keeps it private to this workspace and never publishes, shares or reuses it." The statement and your `data_license` are recorded with the benchmark and shown in every result's notices. Do not upload data you are not allowed to process this way, and do not upload personal data you have no basis to send to model providers. ## Limits and safety | Limit | Value | | --- | --- | | File size for a private benchmark | 64 MiB | | GitHub source file | 16 MiB | | Rows | 100,000 | | Rendered prompt | 100,000 characters per row | | Private benchmarks created per workspace per day | 10 | | Active private benchmarks per workspace | 1,000 (archive one to free a slot) | | Active uploaded sources per workspace | 1,000 (archive one with `evalrouter benchmarks uploads archive`) | | Uploads per workspace per day | 10, at most 2 open at a time | | Minimum available balance to upload | $5 | The 64 MiB limit applies to the file a benchmark is built from. An upload or a Hugging Face import can be larger, up to your workspace's upload limit, but creating a private benchmark from a file over 64 MiB is refused with `private_benchmark_too_large`. Uploads and imported GitHub and Hugging Face files share your workspace's storage allowance. A GitHub import reserves its bytes before storage is written. Retrying the same import after a storage failure reuses that reservation. Archiving a benchmark or completed upload does not free those stored bytes. Uploaded content is treated as untrusted data. Nothing in it is executed: prompts are rendered with plain placeholders and answers are scored by the built-in graders only. Previews and error messages replace credential-shaped strings, such as API keys, tokens and private keys, with `[redacted]` and warn you. They are still part of your data and are still sent to the model, so remove real credentials from the file. ## HTTP API | Operation | Purpose | | --- | --- | | `GET /v1/benchmark-imports/capabilities` | Return `{ "workspace_id": "", "github": true }` or `github: false` for the authorized workspace, with `Cache-Control: no-store`. Use the same authentication and workspace headers as importing; clients should hide GitHub actions on false or an unavailable endpoint. | | `POST /v1/benchmark-uploads` | Create an upload, then send each file to its returned upload URL with `PUT` (what `evalrouter benchmarks uploads create` does). | | `POST /v1/benchmark-uploads/{id}/complete` | Complete the upload and verify the data file. | | `GET /v1/benchmark-uploads` | List your uploads (`status`, `cursor`, `limit` up to 100; with `active_sources` and `active_source_limit`). | | `POST /v1/benchmark-uploads/{id}/archive` | Archive a completed upload to free a source slot. | | `POST /v1/benchmark-imports/huggingface` | Import a pinned Hugging Face dataset into your workspace (what `evalrouter benchmarks imports huggingface` does). | | `POST /v1/benchmark-imports/github/refs` | When enabled, list the default branch, up to 100 branches, 100 tags and 20 recent commits for a repository. Branches and tags include `name` and `commit_sha`; commits include `commit_sha`. `branches_limited` and `tags_limited` indicate lists that reached the limit. Private repositories require both workspace connection/repository identifiers. Responses use `Cache-Control: no-store`. | | `POST /v1/benchmark-imports/github/preview` | When enabled, resolve a repository and revision and preview a selected data file. | | `POST /v1/benchmark-imports/github` | Import the selected file using the preview's full `commit_sha` as `revision`. Private repositories also require the workspace's selected `github_connection_id` and `github_repository_id`. | | `POST /v1/private-benchmarks` | Validate (`"dry_run": true`) or create a private benchmark from `source` (with `source_sha256` and `source_bytes` for imported `hf://` or `github://` references) plus the manifest keys above. GitHub sources require current workspace availability and repository access. | | `GET /v1/private-benchmarks` | List your private benchmarks. | | `GET /v1/private-benchmarks/{id}` | Read a private benchmark. | | `GET /v1/evals?include_private=true&catalog_view=true&private_only=true` | Read the workspace catalog; uploaded private benchmark records include `created_by` (user ID, or `null` when unavailable) and `created_at`. | | `POST /v1/private-benchmarks/{id}/archive` | Stop new runs and free the benchmark slot while keeping earlier runs and results. The browser's **Delete** action uses this operation. | | `POST /v1/runs` | Run with `"selection": {"profile_ids": ["byob-@1"]}`. A spending cap is optional. | ## Not supported yet - Model-graded (judge) scoring other than [Tune rubric grading](#graders), partial credit, few-shot examples and more than one attempt per row. - Custom grader code.