Skip to content

Say when an agreeing paper leans the other way, and capitalise lowercase names in summaries - #205

Merged
timini merged 1 commit into
mainfrom
claude/contested-summary
Oct 23, 2026
Merged

timini merged 1 commit into
mainfrom
claude/contested-summary

Conversation

@timini

@timini timini commented Oct 23, 2026

Copy link
Copy Markdown
Contributor

Found by rendering the data repository's per-study sample (rewire-bio/rewire-benchmark-data#196) through the disagreement view.

  • The ordering rule pools the agreeing papers' comparisons, so one of them can rank the other model higher on its own. In 11 of the sample's 221 findings that happens, and the summary hid it: "4 papers rank MutPred above PrimateAI in 16 of their 20 comparisons" while one of the four ranks PrimateAI higher in 4 of 6, and the circles placed it beside the outlier. The summary now adds "Of those papers, ranks PrimateAI higher on its own (4 of 6)." It is only added when there are two or more agreeing papers.
  • Summary sentences that start with a lowercase source name ("proteinbench primary benchmark evidence ranks ...") now start with a capital.

Tests in tests/disagreement.test.tsx cover both.

…nd capitalise a summary that starts with a lowercase name
@timini
timini merged commit c53b9c3 into main Oct 23, 2026
11 checks passed
@timini
timini deleted the claude/contested-summary branch October 23, 2026 19:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant