Skip to content

fix(build): gate one-shot patches on evidence, widen defer - #2697

Draft
alexeyv wants to merge 1 commit into
mainfrom
fix/oneshot-triage-evidence
Draft

fix(build): gate one-shot patches on evidence, widen defer#2697
alexeyv wants to merge 1 commit into
mainfrom
fix/oneshot-triage-evidence

Conversation

@alexeyv

@alexeyv alexeyv commented Aug 8, 2026

Copy link
Copy Markdown
Collaborator

Problem

The one-shot route's Classify block asked the parent for a judgment it cannot make reliably, and left no lawful home for the answer.

  • patch was gated on ease alone ("trivially fixable"), so it filtered nothing. Every cheap fix qualified.
  • defer was scoped to provenance ("pre-existing issue not caused by this change"), so a true finding caused by this change but not worth fixing could not be deferred.
  • reject says "noise", which an evidence-deferential parent will not apply to a finding it just agreed is real.

Net effect: anything true, cheap, and unwanted has exactly one exit — shipped code. Observed in the benchmark runs behind this change as speculative defensive guards, a static_assert on adjacent constants, a guard for a state the constructors make unreachable, and public constants exported from a header mainly so a test could reference them.

The variance is the other symptom: the same wording, the same reviewer output shape, and the same model produced 5 patches in one run and 12 in another. A criterion the model cannot check does not produce a decision; it produces a temperament draw.

Change

Two bullets. No new categories, no new mechanism.

  • patch now carries an evidence bar and a remedy bound: the finding must show a defect that actually occurs, missing coverage for a specific case, or a broken gate or convention — and the fix must be the smallest one, with no new public surface and no guards for undemonstrated states.
  • defer now covers any real finding not being fixed now, improvement ideas explicitly included.

reject is untouched and now works by elimination: substance goes to patch or defer, so "noise" is left with only non-substance. The ten-finding reviewer quota, the reviewer prompt, the HALT clause, the deferred-work entry format, and the synchronous subagent architecture are all unchanged.

The design rule underneath: the model checks substance, the human judges worth, defer is the channel between them. Every predicate the model is asked to evaluate is mechanically checkable against the code — did anything demonstrate this state, does the remedy add public symbols, does a named case lack coverage, is a gate red. No criterion contains the word "worth". Worth judgments leave as backlog lines the human can strike in seconds, instead of arriving as diff the human has to review and revert.

Widening defer to improvement ideas is deliberate, and is the opposite of churn reduction. A rejected improvement idea dies in terminal scrollback; a deferred one is durable text a human can triage, and frequently seeds a better idea than the one filed.

Expected effect

Replayed by hand against the recorded findings of a benchmark run (18 findings, one blind-hunter layer, Qt dialog auto-sizing):

  • Kept: line-height rounding correction, the resize-driven refit mechanism, correct measurement timing, and all four added test paths (short, long, failure, shrink-back). These are demonstrated defects and named uncovered cases; they clear the evidence bar unchanged. "Missing coverage for a specific case" is the clause that protects them.
  • Dropped: the static_assert, the zero-width guard, a self-added unreachable null guard, and the header export of two constants (the test pins the values literally instead — the export never fixed the reviewer's stated complaint anyway).
  • Moved to defer: an improvement idea a prior run rejected, and a UX limitation a prior run mentioned only in its terminal summary.

The change is also intended to be neutral across both intent regimes the one-shot route sees, since the route cannot verify its own "simple change" premise at intake. On an understated-complex intent, remediation still scales with the complexity the reviewer discovers. On an honestly trivial intent, quota filler cannot produce demonstrated defects, so it drains to defer and reject rather than inflating the diff.

Recommended A/B before this is trusted

Hand replay is not evidence. The validation is a frozen-findings A/B, which isolates the Classify wording from reviewer variance:

Setup. Reconstruct the pre-review file state of a recorded run by replaying its logged Edit/Write calls onto the baseline commit. Freeze that run's reviewer output as a fixed findings list. Both arms receive identical inputs: same pre-review tree, same findings, same model and effort. Only the Classify block differs — main vs. this branch.

Arms. N ≥ 2 per arm. Two runs per arm is the minimum given that the current wording produced 5 vs. 12 patches on comparable inputs; a single pair cannot distinguish a wording effect from that spread.

Second pair. One additional pair on an honestly trivial intent (a genuinely small change, quota intact) to probe the opposite failure mode: does the evidence bar suppress real fixes when the reviewer is filling a quota against a small diff?

Scoring — mechanical, from the diffs only. Never from the parent's self-reported patched/deferred/rejected counts; the recorded run reported findings as rejected whose content had shipped inside consolidated rewrites, and counted a self-added guard as a reviewer patch. Counts are unreliable in both directions.

Per arm, measured against the reconstructed pre-review tree:

Metric Direction
New public symbols introduced by remediation lower is better
Guards added for states no finding demonstrated lower is better
Presence of each endorsed behavior change (rounding fix, refit mechanism, four test paths) must stay at 100%
Remediation diff size, additions and deletions context, not a target
Substance dropped: findings that were real and landed in neither the diff nor deferred-work.md lower is better — this is the failure mode the change could introduce
Deferred-work entries reported, explicitly not scored as residue
Non-HITL wall time to the same endpoint context; endpoint completeness reported alongside it

Pass condition. Endorsed behavior at 100% in every run, new public symbols and undemonstrated guards strictly lower than control, and dropped substance no worse than control. Failing the last one means the evidence bar is eating real findings and the wording needs to loosen, not tighten.

Follow-on, not in this PR

bmad-build/step-04-review.md and bmad-build-auto/step-04-review.md have the same gap in different clothes: patch gated on "caused by the change; trivially fixable" with no evidence bar, defer scoped to provenance. Their existing "when unsure between defer and reject, prefer reject" tiebreaker is compatible — it is a substance-confidence rule, not a worth rule — so the same two edits port cleanly. Deliberately held back until the one-shot A/B reports.

Draft until the A/B runs.

The one-shot Classify contract routed findings by two criteria the model
cannot check: patch was gated on ease alone, so it filtered nothing, and
defer was scoped to provenance, so a true finding that was not
pre-existing had no home but reject. Anything true, cheap, and unwanted
therefore landed as shipped code.

Gate patch on evidence instead: the finding must show a defect that
actually occurs, missing coverage for a specific case, or a broken gate
or convention, and the remedy must be the smallest one -- no new public
surface, no guards for undemonstrated states. Widen defer to any real
finding not being fixed now, improvement ideas included, so worth
judgments reach the human as backlog lines instead of diff.

Reject keeps its wording and now works by elimination: non-substance
only. No criterion in the contract asks the model whether a change is
worth making.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant