Repository navigation
eval: export the LoRA cells as a frozen-retrieval benchmark - #165
Merged
Merged
Conversation
…n-retrieval benchmark
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
`Python lint` fails on this branch. Two things, both mechanical: - `pack_bench.py` carries `# noqa: E402` on the post-`sys.path` import, but E402 is already in the repo-wide ruff ignore list (pyproject.toml), so the suppression suppresses nothing and RUF100 reports it unused. - `ruff format` reflows two of the new files. No behaviour change: both files still compile and the 28 existing eval tests pass. Also merges current main into the branch. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a retrieval-only mode to the eval harness and uses it to export the LoRA cells of Table 1 as a benchmark with frozen retrieval (questions plus their top-5 tiles), so a reader model can be scored without running a search serve.
run_bench.py --dump-retrieval DIRloads examples and builds the retriever exactly as a normal run, then writesrecords.jsonl, the retrieved tiles and any query images instead of calling a reader. It is resumable by example id and logs one start and one end line per example.dump_bench.sh <bench> <out_root>runs it with the example sets, query images, instruction and nprobe of the LoRA cells inreproduce.sh.pack_bench.pypacks the dumps into one Parquet config per bench with the images embedded. Theoriginal_datacolumn is whatlib.graderreads, so a model's answers can be graded after addingfinal_response.missing_tiles.jsonlinstead of failing the example.pack_bench.pyrefuses to pack while any such tile is unfilled, unless it is listed with--allow-missing.Checked on the full export: 4,117 questions across nq, nq_tables, simpleqa, mmsearch and encyclopedic_vqa (landmarks). Grading gold answers as predictions through
lib.grader nqscores 1000/1000.