# Upload a file

Make a private benchmark from a JSON Lines, CSV or Parquet file. You map the prompts and answers, choose a built-in grader and check every row. Then you run it with workspace credit.

Use [Import a benchmark](/catalog/import) in the browser, or a small manifest with the CLI. You need member, admin or owner access. Viewers can read benchmarks but not import them.

## Import in the browser

1. Open [Import a benchmark](/catalog/import) while signed in, and put a file on the source area or choose it. (For a dataset, see [Hugging Face](/developers/docs/huggingface) or [GitHub](#from-a-github-data-file).)
2. Map the columns, write the prompt with `{question}` (and `{choices}` for multiple choice), and choose a [grader](#graders).
3. Confirm your [data rights](#rights-and-terms), then validate. Review the first rows exactly as the model receives them, and create the benchmark.

The page uses your account and workspace, so you don't need an API key. Your settings stay for 24 hours after a refresh in the same tab. The file contents and tokens don't stay, so select the file and confirm your data rights again. If the browser can't save a draft, keep the tab open until the import completes.

## Use the CLI

1. Sign in with `evalrouter login`. A device signed in before importing existed must sign in once more. Device grants are never silently expanded. For CI, use a workspace key with the `benchmarks:submit` scope, plus the evaluation read and write scopes to quote and run. The write scope alone can't import.
2. Write a manifest next to your data file, in JSON. YAML also works when you install PyYAML with the CLI.

   ```json
   {
     "name": "Capitals (synthetic example)",
     "data": "capitals.jsonl",
     "task": "exact_match",
     "columns": {"id": "id", "question": "country", "answer": "capital"},
     "prompt_template": "What is the capital of {question}? Answer with the city name only.",
     "grader_options": {"extract": "last_line"},
     "max_output_tokens": 16,
     "data_license": "LicenseRef-Workspace-Private",
     "rights_attestation": true
   }
   ```

3. Check every row without creating anything. The CLI uploads and verifies the data, and EvalRouter charges nothing.

   ```sh
   evalrouter benchmarks private create manifest.json --dry-run
   ```

   The response shows the row count, [what the model receives](#what-the-model-receives), the first three rows as delivered, and warnings. Answers aren't shown, because they're never delivered. EvalRouter refuses a problem and names its row and reason:

   - a missing column
   - a duplicate ID
   - an answer that isn't a number where the grader needs one
   - a mapped column that carries the answer (`byob_answer_in_input`)

4. Create it, then run it by the printed ID like any benchmark:

   ```sh
   evalrouter benchmarks private create manifest.json
   evalrouter run byob-<uuid>@1 --model openai/gpt-4.1-mini-2025-04-14 --full --max-cost 1
   ```

   `--max-cost 1` sets a $1 spending cap: new work stops when it can't fund the next call, and calls already running finish and are charged. Add `--dry-run` to `run` to see the quote. Its warnings repeat what the model receives and the grader's match rule. Read and export results as in [results](/developers/docs/results).

<details>
<summary>Manifest keys</summary>

| Key | Meaning |
| --- | --- |
| `name`, `description` | What people in your workspace see. |
| `data` or `source` | `data` is a local file relative to the manifest. The CLI uploads it first. `source` is an existing upload, `workspace://<upload_id>/<file>`, or a Hugging Face file your workspace imported, `hf://<org>/<name>@<commit>/<path>` (see [Hugging Face](/developers/docs/huggingface#create-the-benchmark-with-the-cli)). |
| `source_sha256`, `source_bytes` | Required with an `hf://` source: the file's sha256 and size from the import receipt. |
| `format` | Optional `jsonl`, `csv` or `parquet`. Inferred from the file name. |
| `task` | The grader: `multiple_choice`, `exact_match`, `regex_match`, `numeric` or `json_field_match`. |
| `columns` | Maps roles to your column names. `answer` is required. `id` (unique row ID) and `group` (sampling stratum) are optional. Every other role is a prompt placeholder, except the multiple-choice roles. For multiple choice, map `choices` to one column, or map one column per choice to `choice_a`, `choice_b` and so on, in order. |
| `prompt_template` | The user message with `{role}` placeholders. Every mapped prompt column must appear, and the ID, answer and group columns never can. `{choices}` renders lettered lines. Doubled braces are literal braces. |
| `system_prompt` | Optional system message. |
| `grader_options` | Options for the grader (below). Defaults are recorded in the benchmark. |
| `max_output_tokens` | Output limit per answer, 1 to 4096 (default 256). |
| `split` | Optional label for the scored rows (default `test`). |
| `data_license` | A license identifier recorded with results (default `LicenseRef-Workspace-Private`). |
| `rights_attestation` | Must be `true`. See [Rights and terms](#rights-and-terms). |

</details>

## What the model receives

The dry run, the created benchmark (`delivery` in the API response) and every quote state this contract for your benchmark.

- **Delivered per row, and nothing else:**
  1. your `system_prompt`, verbatim, if you set one
  2. your `prompt_template` as the user message, with each placeholder replaced by that row's mapped value and `{choices}` by lettered lines (`A. text`)

  EvalRouter adds no instructions, format hints or examples. A mapped value must be text or a number. EvalRouter refuses a list or an object.
- **Never delivered:** `answer`, `id` and `group`. The grader alone reads the answer, and the template can't contain `{answer}`. Unmapped columns are never read and stay only in the stored copy of your file.
- **Answer guard:** if a row's delivered messages contain its own answer, EvalRouter refuses the benchmark with `byob_answer_in_input` and names the column. The check uses Unicode NFKC, case folding and collapsed whitespace, and covers answers of 20 or more characters. Shorter answers occur by chance and aren't checked, so make sure no mapped column carries them. Multiple-choice answers are letters and are never checked.
- **How each row runs:** 0 shots, one greedy attempt at temperature 0, no automatic retry, no tools, each answer cut at `max_output_tokens`. Validation refuses a rendered user message over 100,000 characters.
- **How each row scores:** one grader decides 1 or 0 from the text output alone. The benchmark records its options, including defaults, and they can't change.

## Graders

Every grader is built in, deterministic and scores each row 1 or 0. Each grader is a versioned rule, such as `multiple_choice@2`. A new benchmark records the current rule, and it keeps that rule for good. A later rule version never re-scores it. They never execute uploaded code or arbitrary lm-eval or Inspect repositories. Results report accuracy with its standard error. Their label names an EvalRouter built-in grader, and you can compare them only with other runs of the same benchmark ID. Maintained catalog benchmarks can use their own specialized graders.

| Grader | Options (defaults) | How it extracts and matches |
| --- | --- | --- |
| `multiple_choice` | `answer_format`: `letter`, `index` or `text` (`letter`) · `index_base`: `0` or `1` (`0`) | In order: the whole output is one letter, optionally in parentheses and followed by `.`, `)` or `:` (any case). Otherwise the last `answer`, optional `is`, optional `:` or `-`, then an **uppercase** letter (`answer: B` or `The answer is (C)`, but not `answer is b`). Otherwise the one choice whose normalized text equals the whole output. Otherwise, from `multiple_choice@2`, a leading choice label: `B. text`, `B) text`, `**B**`, or `B` alone on the first line. A bare `A word` is an article, not a label. Correct when that letter is the reference. A letter outside the row's choices scores 0. |
| `exact_match` | `extract`: `full`, `last_line` or `answer_prefix` (`full`) · `prefix` (`ANSWER:`) · `normalize` | `full` takes the whole output, `last_line` the last non-blank line, `answer_prefix` the text after the last `prefix` (any case) up to the end of that line. Correct when it equals a reference, both normalized. `answer` may be a list of accepted answers (in CSV, a JSON list of strings). A number in a JSON Lines or Parquet file is compared as its text, such as `12`. |
| `regex_match` | `pattern` (required, up to 512 characters) · `anchored` (`true`) · `ignore_case`, `dot_all` (`false`) · `normalize` | The pattern must match the whole stripped output (`anchored`) or anywhere in it. It takes the group named `answer`, else the only group, else the whole match. Correct when it equals a reference, both normalized. |
| `numeric` | `extract`: `last_number`, `first_number`, `full` or `answer_prefix` (`last_number`) · `prefix` (`ANSWER:`, for `answer_prefix`) · `abs_tol`, `rel_tol` (`0`) | `last_number`/`first_number` take the last or first number in the output. `full` takes the whole output and `answer_prefix` the text after the last `prefix`. Both then drop trailing periods and one trailing `%` and must be a single number. Numbers may have a sign, comma thousands separators, decimals and an exponent. Correct when equal as decimals, or within `abs_tol`/`rel_tol` when set. |
| `json_field_match` | `fields` (required dotted paths) · `extract`: `strict` or `first_object` (`strict`) · `normalize_strings` (`false`) | `strict` parses the whole output. `first_object` takes the first fenced JSON block, else the first decodable JSON object or array. Correct when every named field equals the reference's value. Booleans never equal numbers. Strings are compared exactly unless `normalize_strings`. |

A `choices` column may hold a list of texts, a `{"text": [...], "label": [...]}` struct or a `{"A": text, ...}` mapping. With a struct or a mapping, a letter `answer` may also be the dataset's own label, such as `1`. A letter `answer` may be written `B`, `(B)`, `B.` or `B. text`, when the text is that choice's own text.

`normalize` defaults to Unicode NFKC, case-folded and whitespace-collapsed comparison. Normalization keeps punctuation unless you set `strip_punctuation`, which turns punctuation into spaces. An unparseable or unmatched answer scores 0 with a recorded reason. Timeouts and provider errors are errors, never scores.

## Manage private benchmarks

`evalrouter benchmarks private list` lists them. Archiving one with `benchmarks private archive` stops new runs and keeps earlier runs and results. In [the catalog](/catalog), **Source › Created by me** shows the ones you created in this workspace. Imports with the same name have separate IDs, and the creation date tells them apart. Members, admins and owners can choose **Delete**, which asks you to check the ID and date. Delete then does three things:

- It archives that benchmark.
- It removes the benchmark from the active catalog.
- It frees the benchmark slot.

Delete keeps the uploaded source and its source slot.

A benchmark version never changes. To change the data, template or grader, create a new benchmark.

<details>
<summary>Free an uploaded source slot</summary>

Every completed upload is an active source until you archive it, whether or not a benchmark uses it, and a workspace has 1,000 slots. Each create with a local data file uploads a new source, dry runs included. When all are in use, EvalRouter refuses a new upload with `private_benchmark_limit` and the limit kind `upload_sources`. Archiving a private benchmark doesn't free a source slot.

```sh
evalrouter benchmarks uploads list --status completed
evalrouter benchmarks uploads archive <upload_id>
```

The list shows each upload's ID, status, size, rows, date and files, and how many slots are in use. Archive takes the upload ID or its workspace source reference. Archiving is permanent, and a new manifest can't name the source. EvalRouter deletes nothing. Its files stay stored and count toward storage, and benchmarks already created from it, and their runs, keep working. To reuse a source instead of uploading again, name it as `"source": "workspace://<upload_id>/<file>"` in place of the data file.

</details>

## From a GitHub data file

GitHub imports are available only to enabled workspaces. Elsewhere the page hides the GitHub source, and a saved draft or direct link offers file upload or Hugging Face instead.

1. In [Import a benchmark](/catalog/import?source=github), choose **GitHub**. Use **Public** for a repository URL or `owner/repository`, or **Private** to select one connected private repository. Owners and admins connect or disconnect accounts in [Settings → Integrations](/settings/integrations). Set repository permissions on GitHub. Connecting an account makes its granted repositories available in the importer. You don't need a personal access token.
2. Files load automatically from the default branch. Expand folders in the file tree to select one CSV, JSONL or Parquet file, or filter by typing a path. You can also paste an exact file path. Use the arrow keys to move through files and expand or collapse folders, and Enter or Space to select a file. To use another revision, choose a branch, tag or recent commit from the dropdown. You can also enter a reference. Files can be up to **16 MiB** and within your workspace limits. The import keeps the commit it shows, even if the branch later moves.
3. Map the columns, choose one of the [built-in graders](#graders), confirm your data rights, then validate and create the benchmark. The verified file and benchmark stay private to your workspace, including data from a public repository.

This imports data for your chosen prompts and scoring rules. It does not run repository code, install dependencies, or reproduce an author's custom evaluation protocol. JSON arrays, Git LFS pointers, symlinks and submodules are not supported. Upload the underlying supported data file instead.

`evalrouter benchmarks imports capabilities`, or the capability endpoint below, shows whether GitHub imports are available to your workspace. Where they are, the GitHub import commands (refs, preview and import) follow the same steps as the browser.

## Privacy, rights and limits

- A private benchmark and its upload are visible only inside your workspace. Anyone else gets "not found", never "forbidden". It never enters the shared catalog, and you can't share its results through a public link.
- EvalRouter stores your file once per upload in your workspace's private storage. It reads the file back by its SHA-256 before every run. It never deduplicates or compares identical files in two workspaces.
- Each run you start sends the rows to its model provider, like any benchmark prompt.
- EvalRouter treats uploaded content as untrusted data and executes nothing in it. Previews and errors replace strings that look like credentials with `[redacted]` and warn you. The run still sends the file itself to the model, so remove real credentials.

### Rights and terms

In the browser, choose the dataset’s existing license. Use **Private or proprietary** for data governed by your own private terms, a named license when it applies, or **Other license** to enter its identifier. This records the existing terms. It does not change them or make the benchmark public. The CLI records this choice in `data_license`. Hugging Face imports preserve the verified source license.

By setting `rights_attestation: true` you confirm: "This workspace has the right to use this data for evaluation and to send its rows to the model providers selected for each run. EvalRouter keeps it private to this workspace and never publishes, shares or reuses it." The benchmark records the statement and your `data_license`, and every result's notices show them. Don't upload data that you may not process this way. Don't upload personal data that you have no basis to send to model providers.

### Limits

| Limit | Value |
| --- | --- |
| File size for a private benchmark | 64 MiB |
| GitHub source file | 16 MiB |
| Rows | 100,000 |
| Rendered prompt | 100,000 characters per row |
| Private benchmarks created per workspace per day | 10 |
| Active private benchmarks per workspace | 1,000 (archive one to free a slot) |
| Active uploaded sources per workspace | 1,000 (archive one with `evalrouter benchmarks uploads archive`) |
| Uploads per workspace per day | 10, at most 2 open at a time |
| Minimum available balance to upload | $5 |

The 64 MiB limit applies to the file that you build a benchmark from. An upload or a Hugging Face import can be larger, up to your workspace's upload limit. EvalRouter refuses to create a private benchmark from a file over 64 MiB, with `private_benchmark_too_large`.

Uploads and imported GitHub and Hugging Face files share your workspace's storage allowance. A GitHub import reserves its bytes before it writes to storage. Retrying the same import after a storage failure reuses that reservation. Archiving a benchmark or completed upload does not free those stored bytes.

<details>
<summary>HTTP API</summary>

| Operation | Purpose |
| --- | --- |
| `GET /v1/benchmark-imports/capabilities` | Return `{ "workspace_id": "<uuid>", "github": true }` or `github: false` for the authorized workspace, with `Cache-Control: no-store`. Use the same authentication and workspace headers as importing. Clients should hide GitHub actions on false or an unavailable endpoint. |
| `POST /v1/benchmark-uploads` | Create an upload, then send each file to its returned upload URL with `PUT` (what `evalrouter benchmarks uploads create` does). |
| `POST /v1/benchmark-uploads/{id}/complete` | Complete the upload and verify the data file. |
| `GET /v1/benchmark-uploads` | List your uploads (`status`, `cursor`, `limit` up to 100. With `active_sources` and `active_source_limit`). |
| `POST /v1/benchmark-uploads/{id}/archive` | Archive a completed upload to free a source slot. |
| `POST /v1/benchmark-imports/huggingface` | Import a pinned Hugging Face dataset into your workspace (what `evalrouter benchmarks imports huggingface` does). |
| `POST /v1/benchmark-imports/github/refs` | When enabled, list the default branch, up to 100 branches, 100 tags and 20 recent commits for a repository. Branches and tags include `name` and `commit_sha`. Commits include `commit_sha`. `branches_limited` and `tags_limited` indicate lists that reached the limit. Private repositories require both workspace connection/repository identifiers. Responses use `Cache-Control: no-store`. |
| `POST /v1/benchmark-imports/github/preview` | When enabled, resolve a repository and revision and preview a selected data file. |
| `POST /v1/benchmark-imports/github` | Import the selected file using the preview's full `commit_sha` as `revision`. Private repositories also require the workspace's selected `github_connection_id` and `github_repository_id`. |
| `POST /v1/private-benchmarks` | Validate (`"dry_run": true`) or create a private benchmark from `source` (with `source_sha256` and `source_bytes` for imported `hf://` or `github://` references) plus the manifest keys above. GitHub sources require current workspace availability and repository access. |
| `GET /v1/private-benchmarks` | List your private benchmarks. |
| `GET /v1/private-benchmarks/{id}` | Read a private benchmark. |
| `GET /v1/evals?include_private=true&catalog_view=true&private_only=true` | Read the workspace catalog. Uploaded private benchmark records include `created_by` (user ID, or `null` when unavailable) and `created_at`. |
| `POST /v1/private-benchmarks/{id}/archive` | Stop new runs and free the benchmark slot while keeping earlier runs and results. The browser's **Delete** action uses this operation. |
| `POST /v1/runs` | Run with `"selection": {"profile_ids": ["byob-<uuid>@1"]}`. A spending cap is optional. |

</details>

Not supported yet: model-graded (judge) scoring, partial credit, few-shot examples, more than one attempt per row, and custom grader code.
