Availability and limits
Run any maintained benchmark in the catalog. Where your deployment has private imports, you can also make a benchmark from a file or a Hugging Face dataset.
Benchmarks and models#
A ready benchmark may still reject your model. Some tasks need continuation likelihoods that a chat-only model can't provide. Reasoning settings, output format and context limit must also fit. EvalRouter checks compatibility again before a run starts.
Check the catalog, or run evalrouter benchmarks list and evalrouter models list. Maintained benchmarks use their version's recorded grader. Private imports use EvalRouter's built-in graders. Read the formats and limits before you import your data.
Credits and execution#
Runs need workspace credit. Credits and billing explains caps, holds and provider charges. Results separate wrong answers from system errors and unfinished work, and a terminal timeout doesn't stop the run on the server.
Self-service limits#
Private imports accept JSON Lines, CSV and Parquet with a built-in grader. They can't run uploaded code, custom graders or arbitrary evaluation repositories. Public catalog additions need separate review.
Each deployment turns on hosted source connections separately from catalog evaluations. A visible interface or SDK method doesn't mean a deployment has them. The API refuses disabled operations. See workload imports.