Skip to content

Deepen canonical Benchmark Run publication and reprojection #195

Description

@justsml

Problem

Browser execution currently also owns historical reprojection, cost repair, model-quality eligibility, repeat grouping, release claim guards, CSV/JSON/Markdown output, and publication projections. Understanding whether a Benchmark Run Record can support a comparison claim requires following execution state and several adjacent helpers. This weak locality lets scorer truth, harness health, cost provenance, and publication eligibility drift.

What to build

Create one deep Benchmark Run publication module that accepts canonical immutable records and owns validation, historical reprojection, eligibility, repeat/variance guards, completeness, and all derived publication artifacts. Execution supplies records and cannot recompute publication or claim semantics.

Acceptance criteria

  • Canonical Benchmark Run Record validation and historical reprojection have one owner.
  • Cost provenance, infrastructure/model-quality classification, browser/trace/video evidence gates, repeat grouping, and release eligibility are computed consistently for every output.
  • JSON, CSV, Markdown, dashboard, release-manifest, and Hugging Face projections derive from the same accepted record set and expose totals, omissions, and completeness.
  • Reprojection never overwrites source evidence or silently promotes a previously ineligible row.
  • Execution modules cannot independently recompute model-comparison claim semantics.
  • The module accepts records and returns/writes publication artifacts without relying on browser CLI globals.
  • Integration coverage uses structured records and artifacts rather than locking report prose, HTML, palette, or source text.
  • Existing eval-honesty, hidden-gold, cost, native-trace, cleanup, and MP4 requirements remain fail-closed.

Related work

Architecture evidence

The 2026-08-30 architecture review found that deleting the current projection block would force every consumer to recreate contamination guards, cost accounting, repeat grouping, and eligibility. The complexity earns a module, but it is not yet behind one coherent seam.

Metadata

Metadata

Assignees

No one assigned

    Labels

    difficulty: LLarge cross-module implementation scopeenhancementNew feature or requestpriority: highHigh-value safety or evaluation integrity workready-for-agentReady for an implementation agent

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions