Skip to content

[Remove] Drop the decision-model review pre-screen and diff risk hints - #3173

Merged
mrubens merged 1 commit into
developfrom
chore/remove-review-prescreen
Sep 23, 2026
Merged

mrubens merged 1 commit into
developfrom
chore/remove-review-prescreen

Conversation

@mrubens

@mrubens mrubens commented Sep 23, 2026

Copy link
Copy Markdown
Contributor

Removes the decision-model pre-screen from pull request reviews and the get_diff_risk_hints tool built on it (added in #3119, #3130, and #3141).

Why

A paired eval found that the hints don't help a reviewer find more bugs, and they may make it miss some.

Setup:

  • 40 merged pull requests whose first review predates the hints; that review's 51 inline findings are the answer key.
  • A reviewer model reviewed each diff 3 times with the hint block and 3 times without.
  • A blind judge matched each run's findings against the answer key.
Without hints With hints
Findings caught 47.1% 40.5%
Inside hinted hunks 56.8% 54.3%
Outside hinted hunks 36.1% 25.0%
  • Hints left 38 findings unchanged, lowered the catch rate on 9 and raised it on 4.
  • Overall change: -6.6 points (95% bootstrap interval -16.3 to +2.0).
  • The ranking itself is informative: about half of real findings fall inside the three hinted hunks. But putting it in the reviewer's prompt narrows attention. The reviewer checks the hinted hunks, finds them fine, and reports less elsewhere.
  • On the author side, a look at tasks that called get_diff_risk_hints found no case where an agent changed code because of a hint. Later reviews still found bugs inside the hunks it had flagged.

The eval reviewer is one model call over the diff, not the full review agent, so this measures what a reviewer notices, not end-to-end review quality. If the signal comes back, it should be an additive pass on the ranked hunks rather than guidance in the main reviewer's prompt.

Changes

  • The review and review-sync workflows no longer compute or pass review_prescreen, and the review-code skill no longer mentions it.
  • Removes the pre-screen module, its offline eval script, and the review-prescreen:eval script.
  • Removes the get_diff_risk_hints MCP tool, its run-token API route, the shared server helper, and the self-review instruction in the implement-changes workflow.
  • Removes the corresponding line from the judgment-model docs.

No schema or settings changes.

@roomote-community

roomote-community Bot commented Sep 23, 2026 •

Copy link
Copy Markdown
Contributor

No code issues found. See task

Reviewed 2e1b333

@mrubens
mrubens merged commit 735c56b into develop Sep 23, 2026
19 checks passed
@mrubens
mrubens deleted the chore/remove-review-prescreen branch September 23, 2026 19:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant