fix(build): gate one-shot patches on evidence, widen defer - #2697
Draft
alexeyv wants to merge 1 commit into
Draft
Conversation
The one-shot Classify contract routed findings by two criteria the model cannot check: patch was gated on ease alone, so it filtered nothing, and defer was scoped to provenance, so a true finding that was not pre-existing had no home but reject. Anything true, cheap, and unwanted therefore landed as shipped code. Gate patch on evidence instead: the finding must show a defect that actually occurs, missing coverage for a specific case, or a broken gate or convention, and the remedy must be the smallest one -- no new public surface, no guards for undemonstrated states. Widen defer to any real finding not being fixed now, improvement ideas included, so worth judgments reach the human as backlog lines instead of diff. Reject keeps its wording and now works by elimination: non-substance only. No criterion in the contract asks the model whether a change is worth making.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The one-shot route's
Classifyblock asked the parent for a judgment it cannot make reliably, and left no lawful home for the answer.patchwas gated on ease alone ("trivially fixable"), so it filtered nothing. Every cheap fix qualified.deferwas scoped to provenance ("pre-existing issue not caused by this change"), so a true finding caused by this change but not worth fixing could not be deferred.rejectsays "noise", which an evidence-deferential parent will not apply to a finding it just agreed is real.Net effect: anything true, cheap, and unwanted has exactly one exit — shipped code. Observed in the benchmark runs behind this change as speculative defensive guards, a
static_asserton adjacent constants, a guard for a state the constructors make unreachable, and public constants exported from a header mainly so a test could reference them.The variance is the other symptom: the same wording, the same reviewer output shape, and the same model produced 5 patches in one run and 12 in another. A criterion the model cannot check does not produce a decision; it produces a temperament draw.
Change
Two bullets. No new categories, no new mechanism.
rejectis untouched and now works by elimination: substance goes to patch or defer, so "noise" is left with only non-substance. The ten-finding reviewer quota, the reviewer prompt, the HALT clause, the deferred-work entry format, and the synchronous subagent architecture are all unchanged.The design rule underneath: the model checks substance, the human judges worth, defer is the channel between them. Every predicate the model is asked to evaluate is mechanically checkable against the code — did anything demonstrate this state, does the remedy add public symbols, does a named case lack coverage, is a gate red. No criterion contains the word "worth". Worth judgments leave as backlog lines the human can strike in seconds, instead of arriving as diff the human has to review and revert.
Widening defer to improvement ideas is deliberate, and is the opposite of churn reduction. A rejected improvement idea dies in terminal scrollback; a deferred one is durable text a human can triage, and frequently seeds a better idea than the one filed.
Expected effect
Replayed by hand against the recorded findings of a benchmark run (18 findings, one blind-hunter layer, Qt dialog auto-sizing):
static_assert, the zero-width guard, a self-added unreachable null guard, and the header export of two constants (the test pins the values literally instead — the export never fixed the reviewer's stated complaint anyway).The change is also intended to be neutral across both intent regimes the one-shot route sees, since the route cannot verify its own "simple change" premise at intake. On an understated-complex intent, remediation still scales with the complexity the reviewer discovers. On an honestly trivial intent, quota filler cannot produce demonstrated defects, so it drains to defer and reject rather than inflating the diff.
Recommended A/B before this is trusted
Hand replay is not evidence. The validation is a frozen-findings A/B, which isolates the Classify wording from reviewer variance:
Setup. Reconstruct the pre-review file state of a recorded run by replaying its logged Edit/Write calls onto the baseline commit. Freeze that run's reviewer output as a fixed findings list. Both arms receive identical inputs: same pre-review tree, same findings, same model and effort. Only the
Classifyblock differs —mainvs. this branch.Arms. N ≥ 2 per arm. Two runs per arm is the minimum given that the current wording produced 5 vs. 12 patches on comparable inputs; a single pair cannot distinguish a wording effect from that spread.
Second pair. One additional pair on an honestly trivial intent (a genuinely small change, quota intact) to probe the opposite failure mode: does the evidence bar suppress real fixes when the reviewer is filling a quota against a small diff?
Scoring — mechanical, from the diffs only. Never from the parent's self-reported patched/deferred/rejected counts; the recorded run reported findings as rejected whose content had shipped inside consolidated rewrites, and counted a self-added guard as a reviewer patch. Counts are unreliable in both directions.
Per arm, measured against the reconstructed pre-review tree:
deferred-work.mdPass condition. Endorsed behavior at 100% in every run, new public symbols and undemonstrated guards strictly lower than control, and dropped substance no worse than control. Failing the last one means the evidence bar is eating real findings and the wording needs to loosen, not tighten.
Follow-on, not in this PR
bmad-build/step-04-review.mdandbmad-build-auto/step-04-review.mdhave the same gap in different clothes:patchgated on "caused by the change; trivially fixable" with no evidence bar,deferscoped to provenance. Their existing "when unsure between defer and reject, prefer reject" tiebreaker is compatible — it is a substance-confidence rule, not a worth rule — so the same two edits port cleanly. Deliberately held back until the one-shot A/B reports.Draft until the A/B runs.