Skip to content

Expand repeated-run benchmark report aggregates - #41

Merged
evandhoffman merged 1 commit into
mainfrom
codex/issue-38-report-aggregates
Apr 19, 2026
Merged

evandhoffman merged 1 commit into
mainfrom
codex/issue-38-report-aggregates

Conversation

@evandhoffman

Copy link
Copy Markdown
Collaborator

Closes #38.

What changed

  • enrich repeated-run model sections with wall-time variance, p90, distinct app-hash counts, distinct solution-shape counts, and median timing breakdowns
  • expand the repeated-run summary table so variance and stability signals stay visible at a glance
  • add an order-effects table that summarizes pass rate and median wall time by run position across rounds
  • keep single-run report output unchanged

Verification

  • uv run python -m unittest tests.test_report
  • uv run python -m unittest discover tests

@evandhoffman
evandhoffman merged commit 6dc8af3 into main Apr 19, 2026
1 check passed
@evandhoffman
evandhoffman deleted the codex/issue-38-report-aggregates branch April 19, 2026 12:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Expand report aggregates for repeated benchmark runs

1 participant