Add BenchPress score-matrix adapter (utils/benchpress)#197
Open
borgr wants to merge 1 commit into
Open
Conversation
Aggregator adapter for microsoft/benchpress-score-matrix, modeled on llm_stats:
source_type=documentation, source_role=aggregator, output logs split by
evaluator_relationship (derived per score from BenchPress source_type). Reads the
public CSV mirror from HuggingFace (--input-json for offline replay).
- Follows BenchPress's documented update manifest (metadata.json): generated_at_utc
-> retrieved_timestamp, and source_git_commit/generated_at_utc recorded on every
record for snapshot/update tracking.
- Metric bounds are the metric's true bounds with +/-inf where unbounded; written
as the JSON Infinity token (EEE's json.loads + pydantic loader reads float('inf');
model_dump_json would null it, so write_log serializes the record itself).
- Per-score citation -> source_data.url; per-score harness/settings -> additional_details.
Adds utils/benchpress/README.md, a utils/README.md row, and
tests/test_benchpress_adapter.py (8 tests). Verified: live fetch -> 366 logs, all
pass 'every_eval_ever validate'; unit tests pass.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds
utils/benchpress/— an aggregator adapter for the BenchPress scorematrix (
microsoft/benchpress-score-matrix), modeled onutils/llm_stats.BenchPress re-reports model scores scraped from provider blogs, tech reports,
model cards, leaderboards and third-party aggregators, each cell carrying its own
citation and provenance — so it's handled as an aggregator:
source_type=documentation,source_role=aggregator, and output logs are split byevaluator_relationship(derived per score from BenchPress's
source_type).Behavior
(
data/{models,benchmarks,scores_all}.csv);--input-jsonreplays a savedpayload offline.
metadata.jsonmanifest —generated_at_utc→retrieved_timestamp, andsource_git_commit/generated_at_utcrecorded on every record so consumers can detect a new snapshot.reference_urlis first insource_data.url(with
reported_by= its host); the dataset URL follows.rangewins, elseper-family bounds with
±infwhere genuinely unbounded (elo/rating/index/raw),written as the JSON
Infinitytoken (EEE'sjson.loads+ pydantic loader readsfloat('inf');write_logserializes the record directly becausemodel_dump_jsonwould null it).model_info.id=<provider-slug>/<benchpress-slug>(raw slug kept inadditional_details);eval_library= the aggregator, per-score harness/settingsin
additional_details.Tests / verification
tests/test_benchpress_adapter.py(8 tests): relationship split, schemavalidity, id/citation, bounded vs unbounded(
inf), version provenance, and theInfinity-token round-trip + output layout.every_eval_ever validate.Also adds
utils/benchpress/README.mdand a row inutils/README.md. No existingfiles are modified beyond the
utils/README.mdtable row.Companion data PR: the records this adapter produces are proposed to the datastore in https://huggingface.co/datasets/evaleval/EEE_datastore/discussions/164 (paired — if one changes, the other will be updated).