jev-classifier

Local stdio MCP that answers agents' classification-shaped questions — verify a finding against its cited evidence, label, triage, "is X true of this?" — with TypeSafe Jev typed judgments (probabilities + confidence) instead of subagent prose. Faster and cheaper than a reasoning model for judgments that fit the evidence you already have.

Measured classifiers have fixed questions and thresholds chosen on a dev split; the generic jev_ask escape is uncalibrated.

Install

Requires Python ≥3.10 and TYPESAFE_API_KEY (never written to files).

pipx install 'git+https://github.com/heliowap/jev-classifier'   # or: uvx --from 'git+...' jev-classifier
# dev:  uv venv && uv pip install -e '.[test]' && python3 -m pytest

Register (stdio)

VS Code .vscode/mcp.json — "jev-classifier": {"type": "stdio", "command": "jev-classifier", "env": {"TYPESAFE_API_KEY": "${env:TYPESAFE_API_KEY}"}} Codex .codex/config.toml — [mcp_servers.jev-classifier] command = "jev-classifier" (env inherited) Claude Code — claude mcp add jev-classifier --env TYPESAFE_API_KEY=$TYPESAFE_API_KEY -- jev-classifier Devin — add to the MCP config: command jev-classifier, env TYPESAFE_API_KEY

Start a new session after registering; a running session keeps its original tool inventory.

Tools

finding_verify

Verify review findings against their cited evidence — call BEFORE spawning a subagent or re-reading files to confirm a finding.

Input: findings (1–32) {id, claim, evidence, quote?, source_context?}. Per id: verdict (supported | contradicted | unsupported | fabricated | error), probabilities, confidence, band (stand | review), threshold, policy. A quote absent from evidence is fabricated locally — no API call. band=review means the verdict is a hint: verify by your own means. Batch: one tool call fans out to one Jev request per finding (≤8 parallel).

jev_ask

state (string or JSON) + questions ({id: {type: noul|choice|score, instructions, criteria}}) → raw answers + usage. Marked calibrated: false: unvetted questions carry no measured precision; thresholds are the caller's responsibility. Inputs with obvious credentials are rejected; state+questions capped at 128k chars.

Decision log

~/.cache/jev-classifier/decisions.jsonl (JEV_CLASSIFIER_LOG overrides, off disables). Hashes of inputs, not text. JEV_CLASSIFIER_LOG_RAW=1 stores raw inputs for dataset building — off by default.

Evaluating a classifier

TYPESAFE_API_KEY=... python3 evals/finding_verify_eval.py \
    --data evals/private/finding_verify.jsonl --split dev

Thresholds are chosen on the dev split for precision; the test split is evaluated once. Datasets under evals/ stay private (gitignored); committed results are aggregates only. See evals/finding_verify_results.md.

Public repository

CLAUDE.md has the publishing rules. git config core.hooksPath .githooks enables the local checks (run it yourself); CI runs the regex scan on every push. Set PUBLIC_SAFETY_JEV=0 to skip the optional Jev layer in pre-push.