Interop check: does an EvalPort adapter for gauntlet's result packs make sense?

[adhabnr-ux]

I maintain EvalPort, an open schema (TestCase / Grader / EvalSuite / ResultSet) for making evaluation data portable between frameworks — think "one JSON shape a dashboard or CI gate can read regardless of which eval tool produced it." I read src/gauntlet/cases.py, results.py, and the five gates under src/gauntlet/gates/ (not README claims) before writing this, so this is grounded in what the code actually does, not the description.

Not a PR, not asking for a dependency or code change here — just checking whether my reading of the shapes is right before I'd build anything, and whether it's wanted at all.

What I think maps cleanly:

gauntlet	EvalPort
Suite (cases.py) — name, gate, version, threshold, cases	EvalSuite — id, version, test_cases, with gate and threshold carried in metadata (EvalPort has no suite-level pass-threshold field)
Case — id, language, prompt	TestCase — id, input; language as a tag
RunResult (results.py) — target, gates, provenance	ResultSet — suite_id, provider, with gauntlet's required provenance keys (target_version, commit, prompt_version...) folded into ResultSet.metadata
GateResult → CaseResult	Result → GraderResult — CaseResult.detail → reason, .observed → actual_output

Where the grader types are a real fit, not a stretch, because the gate code is literal string matching, not fuzzy scoring:

• must_contain (grounding.py, refusal.py) → EvalPort's built-in contains grader — same semantics, case-insensitive substring check.
• expected (golden.py) → exact_match, though gauntlet's normalize_answer only collapses whitespace (no case-folding), which is narrower than the standard grader's ignore_case/trim_whitespace params — worth calling out rather than silently widening it.
• rubric (judge gate) → llm_judge, params.prompt = case.rubric.

Where it's a genuine mismatch, not just relabeling — three things EvalPort's schema doesn't have a slot for:

1. must_not_contain (adversarial.py) has no standard EvalPort grader — there's no built-in "absent marker" type. Per the spec's type-openness rule I'd declare it as a non-well-known type (e.g. gauntlet_marker_absence) with params.handler set, which the spec says any runner without that handler must skip rather than misinterpret.
2. evaluate_refusal/evaluate_adversarial key off TargetResponse.refused / .escalated — booleans your target adapter sets, not something derivable from the response text alone. EvalPort graders score actual_output (a string); there's no first-class field for "the target self-reported refusing." I'd have to carry those flags in Result.metadata, which a generic EvalPort consumer wouldn't know to look at.
3. Judge calibration is structural in gauntlet — ADR 0001 fails a judge gate closed with no calibrated set (_calibrate_for in gates/base.py), and RunResult.verdict_withheld can mark a whole run as unscoreable (unscoreable_reason in base.py). EvalPort has no ResultSet-level "withheld" concept and no calibration semantics on llm_judge — closest is metadata.openeval.aggregation_status: "unscored", which is per-Result, not per-run. This is the piece I'd feel worst about papering over with metadata alone.

Sketch of the conversion I'd write, to be concrete about what "adapter" means here — no gauntlet-side changes, this would live entirely on the EvalPort side, reading your public JSON result pack as data:

# evalport/adapters/gauntlet-openeval-adapter — reads gauntlet's `gauntlet report --json` output
def gate_result_to_grader_results(gate: dict) -> list[dict]:
    grader_type = {
        "grounding": "contains", "refusal": "contains", "golden": "exact_match",
        "judge": "llm_judge", "adversarial": "gauntlet_marker_absence",
        "false_positive": "contains",
    }[gate["gate"]]
    return [
        {
            "grader_id": f"{gate['gate']}_gr",
            "type": grader_type,
            "score": 1.0 if c["passed"] else 0.0,
            "passed": c["passed"],
            "reason": c["detail"],
        }
        for c in gate["cases"]
    ]

This would sit alongside the existing framework adapters in evalport/adapters/ (deepeval, argilla, azure-ai-evaluation, etc. all follow the same one-directional "read the tool's native output, emit a ResultSet" shape).

Given #3 especially, I'd rather hear whether this reading is even right, and whether a lossy one-directional export is something you'd consider useful or a category error for what the evidence pack is for, before I spend time on it. No obligation either way — happy to just close this out if it's not a fit.

— Sahi, independent contributor (not affiliated with this project)

[ChelseaKR]

Thanks for reading the code before writing this — the mapping table is accurate against origin/main today (8ae3fa2), and I checked each row rather than taking it on trust:

• Suite is name, gate, version, threshold, cases (plus key_version, an optional judge block, and a non-comparing source). ✔
• Case is id, language, prompt with the gate-specific fields optional at the type level. ✔
• normalize_answer in gates/golden.py is exactly " ".join(text.split()) — whitespace only, no case-folding, as you say. Calling that out rather than silently widening it to ignore_case is the right instinct; the docstring's word for a wording change is "drift", and drift is meant to be reported. ✔
• must_not_contain has no built-in EvalPort equivalent, and refused / escalated are target-declared booleans, not derivable from the response text. ✔

One thing the shapes miss, and it is the one that would bite. Every gate scores legibility before it scores content (gates/readability.py, from #10). evaluate_adversarial fails a case when no compromise marker is present but said_something(response) is false — and said_something is not text.strip(): it NFKC-normalizes, strips bracketed citation markers, and asks whether any alphanumeric survives, or whether the target declared a refusal or an escalation.

So a mute target scores 0 on gauntlet's adversarial suite and would score 100% on a generic EvalPort contains-style absent-marker grader reading actual_output as a string. That is the exact failure the readability module exists to refuse: "a check phrased as the absence of something bad is satisfied by silence."

It goes further than per-case scoring. gates/base.py::unscoreable_reason withholds the whole run's verdict when responses carried nothing readable and no loaded suite would have failed the target for it — exit 4, UNSCOREABLE, distinct from a gate failure. There is no field in a per-result schema for "this run has no verdict"; it is a property of the loaded suite set, not of any one result.

If you do build an adapter, I would suggest that gauntlet_marker_absence carry the legibility predicate in its params.handler contract, and that a run whose verdict_withheld is non-empty not be exported as a ResultSet at all — a withheld verdict rendered as a set of passing results is worse than no export.

On whether it is wanted: that is a product-direction call and it is Chelsea's, not mine — I have not answered it here. Related and worth your eye: #51 proposes gauntlet report --format plumbline-bundle, i.e. exactly the export mechanism your question implies, with EvalPort named there as a plausible second format behind the same flag. If that lands, an EvalPort adapter is a formatter rather than a fork.

Triaged with AI assistance; the code claims above were checked against origin/main, not against the README.

[adhabnr-ux]

Thanks for checking each row against the actual code rather than taking the table on trust — that's more diligence than I expected for a "just checking the shape" issue.

The said_something() catch is the one that matters and I'm glad you flagged it before I built anything. You're right that a naive contains-style absent-marker grader reading actual_output as a plain string would score a mute target 100% on the exact case the readability module exists to catch — that's not a rounding error, it's the adapter silently inverting the gate's whole purpose. If I do build gauntlet_marker_absence, the legibility predicate (NFKC-normalize, strip citation markers, check for surviving alphanumerics or an explicit refused/escalated declaration) goes in params.handler verbatim, not reimplemented from the README. And agreed on the second point too: a run with a non-empty verdict_withheld doesn't get exported as a ResultSet at all — there's no honest way to represent "no verdict" as a set of passing results, and I'd rather an adapter refuse to export than paper over it.

On whether it's wanted — understood that's Chelsea's call, not something to push on further here. I'll watch #51; if the plumbline-bundle export lands with EvalPort as a second format behind the flag, the adapter becomes a formatter against a spec you've already fixed, which is a much better position to build from than me guessing at gauntlet's internals from the outside. No further ask from me on this one for now — thanks again for the careful read.