[BasRizk]

What this changes

Step 1 (EXTRACT_BOT_SUGGESTIONS in online/etl/llm/prompts.py) gets one new paragraph, after the existing skip rules. The rest of the prompt is unchanged:

Also skip inputs the bot lists as having used for its review, such as linter or static-analysis
output, code guidelines, or configured review instructions (for example a section where the bot
lists the inputs or context it used). Do not extract findings from these inputs, even when a listed
tool result names a file and line of the changed code: the bot did not raise it as its own comment.
Tool output quoted as supporting context inside a comment, such as a linter or static-analysis log
next to the bot's own text, is not its own finding either: only problems the bot states in its own
words are findings. A tool result the bot posts as its own comment at a location in the changed code
is a finding, and so is a linter or static-analysis result the bot writes up as a problem in its own
comment text. An aggregate check result is not a finding either: a check whose result is a count, a
percentage, or a pass/fail measured over many functions or files (for example docstring or test
coverage against a threshold), including a request to add docstrings or tests to "the touched
functions". A check that states a specific problem with the pull request, such as with its title or
description, is not aggregate: it is a finding. So is a request that names a function, or a file and
line, in the changed code. A concrete finding the bot states in its own words in a review-body
section rather than as an inline comment, or inside a collapsed or nested section, is still a
finding: extract it.

It also adds a version stamp, EXTRACTION_PROMPT_VERSION = "extract-v3". Each analysis now stores model_name as "<model>+extract-v3", so analyses made with different prompts can be told apart. tests/test_prompts.py pins the sha256 of every input the version stands for: the step prompts, the code that builds step input, the actor matching, the metric arithmetic, the response schemas and the LLM call. The test fails when one of them changes without a new version. In CI it also compares the table with the one at the merge-base, so a past entry cannot be edited in place. online/etl/README.md describes this.

Why

This is a correctness fix. The rule is the same for every bot.

• Some bots list the inputs they used for a review, such as linter results, guidelines or review instructions. Those inputs are not the bot's findings. The bot did not raise them as its own comments, even when a listed tool result names a file and a line.
• Linter or tool output quoted inside a comment, as context for the bot's own text, is not a separate finding either. A tool result the bot posts as its own comment at a location in the code is still a finding.
• An aggregate check result is not a finding. That is a check whose result is a count, a percentage or a pass/fail over many functions or files, such as docstring coverage below a threshold. A check that states a specific problem with the pull request, such as its title or description, is a finding. So is a request that names a function, or a file and line.
• A concrete finding the bot states in its own words is still a finding when it sits in a review-body section or a collapsed section. This covers only the bot's own words, so it cannot undo the skip of listed inputs.

The rule is the same for every bot. How much it changes a bot's numbers depends on how often that bot posts such content.

Measured effect

We replayed step 1 and step 3 on a sample of public PRs from the benchmark's stored data. The input is the one the default Search API path builds: every review and every review-thread comment of the bot, across all review rounds, and no issue comments. The human actions were held fixed. The judge was gpt-5.2 at temperature 1.0. P and R use the leaderboard arithmetic: the mean over PRs that have both. Intervals are 95% paired bootstrap intervals over PRs. These numbers come from a sample, and the judge may not be the model you run. They are not what the leaderboard would show after this change.

The whole sample

This run used an earlier wording of the paragraph, without the check-result sentence.

bot	PRs	draws per arm	P before	P after	dP [95% CI]	R before	R after	dR [95% CI]
CodeRabbit	313	2	70.0	70.0	-0.0 [-1.1, +1.1]	52.0	52.2	+0.2 [-0.5, +1.1]
Six other bots pooled (Greptile, cubic, Claude, Codex, Copilot, Qodo)	386 bot-PR pairs	2	79.4	79.7	+0.3 [-0.8, +1.3]	42.2	42.6	+0.4 [-0.2, +1.1]

There is no measurable change for CodeRabbit or for the six other bots. Two draws of the unchanged prompt differ by up to 1.2 points. The sample has 395 bot-PR pairs for the six other bots. We stopped spending after 9 of them had only one draw of the new prompt, so the table uses the 386 pairs with two draws each.

The paragraph as shipped

On this input, check results reach step 1 only through review bodies. In the sample, 15 CodeRabbit PRs have a docstring coverage check there. We ran the paragraph as shipped on those 15, on 40 other CodeRabbit PRs picked at random, and on 70 other-bot pairs picked at random. Two draws per arm, against the current prompt:

PRs	scored	dP [95% CI]	dR [95% CI]	coverage-check items, current / this change (matched)
CodeRabbit, with a coverage check	15 (11 with P and R)	+4.3 [-2.6, +13.7]	-0.9 [-6.4, +4.8]	15 (2) / 0
CodeRabbit, others	40 (30)	+0.3 [-1.4, +1.9]	+1.4 [+0.0, +3.4]	0 / 0
Six other bots pooled	70 (57)	+2.1 [-0.7, +5.3]	+1.7 [-0.2, +4.2]	0 / 0

The change removes every coverage-check item. On the 15 PRs with a coverage check, precision moves up by about 4 points. With 11 scored PRs that is within the noise: two draws of the current prompt differ by -4.2 [-9.2, +0.0] there. Elsewhere there is no measurable change.

Three CodeRabbit reviews in the sample state a title, description or linked-issue check in the review body. Neither the current prompt nor this change extracted an item from them, so nothing was lost there. We read the 5 PRs where this change matched fewer human changes than the current prompt. One lost match was an aggregate coverage item, which this change removes on purpose; the human change it had matched added docstrings. In one PR, two items on the same code were extracted as one. In one, a test-gap item that the current prompt extracted in one of its two draws was not extracted. The rest were the same items, paired differently by the matching step.

The version stamp lets the leaderboard tell analyses made with this prompt from older ones.

Tests

cd online/etl
uv sync --dev
uv run ruff check .
uv run pytest tests/

• tests/test_prompts.py pins the wording of each sentence of the new paragraph. It also pins the version table.
• tests/test_extraction_scope_fixtures.py uses 12 trimmed public review fixtures. It checks that the labelled text reaches step 1, and maps each labelled kind to the prompt sentence that decides it, or explicitly to no rule. It checks that the sentence is in the prompt, and that labels of a kind sit on the side its sentence gives: a description-check row is a finding and a coverage row is not. It cannot show how the model applies the sentences.
• An opt-in check runs the prompt on those fixtures with your model. It makes one paid call per fixture:

RUN_EXTRACTION_SCOPE_EVAL=1 MARTIAN_API_KEY=... MARTIAN_MODEL_NAME=... \
  uv run pytest tests/test_extraction_scope_live.py -v