Resource: Jevals.com #372

What are you sharing?

Article, tutorial, or other resource

Canonical product or source URL

https://jevals.com/

Jev-specific value and suggested category

Suggested category: Model behavior and evaluation (the section is currently all official TypeSafe documentation; this would be a community entry).

Jevals runs hosted Jev (typesafe-ai/jev through Vercel AI Gateway) and six LLMs over the same typed questions and grades every answer against the dataset's human labels: PubMedQA for Noul, Banking77 for Choice, HelpSteer2 for Score, 300 items and 5 runs each. It reports accuracy, calibration error, the coverage and accuracy you get at a given confidence threshold, cost per 1,000 decisions and p95 latency.

What it adds beyond the existing entries in that section: those document how confidence and self-consistency work; this measures what the confidence is worth on labelled data. For example, on the 77-option Banking77 task a 0.85 threshold keeps 76% of decisions at 88.6% accuracy, and on the 0 to 4 HelpSteer2 rubric no system tested, Jev included, beat predicting the label base rates.

Note on naming: the README already lists jevals, an unrelated local evaluation workbench. This is a different project, the site jevals.com, so the entry should probably read "Jevals.com" to keep them apart.

App platform and availability (for apps)

Not an app.

App source-access tag

Not an app

App pricing tag

Not an app

Product homepage, pricing, and Jev evidence (for apps)

Not applicable.

Commercial and source-review disclosure (for apps)

Not applicable. Free to read, no account, no paid tier.

Project guide details (for an application, starter kit, library, or tool)

Not applicable.

Your review

Model and API: hosted Jev through Vercel AI Gateway, which reports the model as typesafe-ai/jev without a version string, so the runs are pinned by date (release 2026-09-18) rather than by version alias. LLM answers come through an adapter prompt asking for the same probabilities, via OpenRouter.

Limitations: one release, three tasks, English only. The per-decision logs, suite files and release board are published as CC BY 4.0 at https://github.com/Jevals/jevals-data, so every number can be recomputed, but the harness code is private, so the runs cannot be re-executed as-is. Method and formulas: https://jevals.com/methodology/.

Affiliation and AI assistance

I maintain Jevals.

Before submitting

• I checked existing entries and open proposals for duplicates.
• I read CONTRIBUTING.md and verified the resource is accessible.