EvalPort adapter for WeaveBench's score.json rubric results #25

Hi — I maintain EvalPort (https://github.com/adhabnr-ux/evalport), a small open interchange spec + SDK (evalport-sdk on PyPI, openeval.validate.validate_suite() / validate_result_set()) for making per-example eval results portable across tools — one shared JSON shape instead of a bespoke converter per consumer.

I read through weavebench/utils/grading.py and weavebench/eval/judge_helper.py. Each task's grading produces score.json — a dict of rubric-key → 0/1 (from vlm_score_rubric()/llm_score_text(), which also set judge_method and judge_error), and print_summary()/print_global_summary() derive an overall_score per task (either the rubric's own overall_score key or the mean of the numeric rubric entries). Per-model runs roll up into summary_<model>.json (list of {task_id, scores, error, usage}) and summary_all_<model>.json ({global_avg, task_count, scored_task_count, missing_score_task_count, results: [...]}).

That per-task record (rubric scores + overall_score + usage) maps cleanly onto an EvalPort per-example result. Sketch, reading summary_all_<model>.json:

# weavebench-openeval-adapter/src/weavebench_openeval_adapter/__init__.py
import json
from pathlib import Path
from openeval.validate import validate_result_set

def to_openeval(summary_all_path: str, suite_id: str) -> dict:
    data = json.loads(Path(summary_all_path).read_text())
    examples = []
    for r in data["results"]:
        scores = r.get("scores") or {}
        numeric = {k: v for k, v in scores.items() if isinstance(v, (int, float))}
        overall = numeric.get("overall_score", (sum(numeric.values()) / len(numeric)) if numeric else None)
        examples.append({
            "id": r["task_id"],
            "score": overall,
            "metadata": {
                "rubric_scores": numeric,
                "judge_method": scores.get("judge_method"),
                "error": r.get("error"),
                "usage": r.get("usage"),
            },
        })
    result_set = {"suite_id": suite_id, "examples": examples}
    validate_result_set(result_set)
    return result_set

I left passed out rather than guess a threshold — overall_score reads as a continuous rubric-coverage fraction in this codebase, not a documented binary pass/fail cutoff, so I'd rather ask than invent one. Happy to add a passed field with a threshold parameter if you have a convention for it.

I'd build this as adapters/weavebench-openeval-adapter/ (standalone package, pyproject.toml depending on evalport-sdk + the pinned self-referencing extra convention from Discussion #13, tests, README — zero footprint on weavebench itself). Since this is the paper-companion code release I'd keep it entirely outside the areas your CONTRIBUTING.md marks as locked down (desktop_env/providers/*, the *.original.js patches, the pinned VM sha256). Let me know if this is useful.

— Sahi (independent contributor, not affiliated with WeaveBench)