diagnose: harden Phase 5.5 root-cause gate against mined failure modes - #33
Open
milstan wants to merge 6 commits into
Open
diagnose: harden Phase 5.5 root-cause gate against mined failure modes#33milstan wants to merge 6 commits into
milstan wants to merge 6 commits into
Conversation
The previous /report-issue skill grew to 1423 lines with 11 gates and
verbatim output blocks — itself the slop pattern it was trying to
prevent. Non-technical reporters couldn't use it in 3 minutes, and
its rules contradicted milstan's actual writing style (it flagged
sectioned issues like #3684/#3656/#3657 as slop while accepting them
as reference goods).
Rewrite is 600 lines with 4 gates, 6 sins, 1 Iron Law (quoted from
milstan #3667). Key changes:
- Iron Law (no proposed solutions) quoted verbatim from milstan's
#3667 public reply.
- Scope section + Step 0 carve-out distinguishes reporter-to-team
issues (skill applies) from owner-internal backlog notes (skill
exits, team owns its own work).
- Gate 1 reframed from brittle keyword-grep to "subject is product
vs user" — more robust and doesn't false-positive on milstan's own
TODOs.
- Sin 4 (extrapolation) split into "strategy tail" and "customer-wide
claim from n=1", with explicit carve-out for workflow-gap
observations from a dogfooder (#3676 passes correctly).
- Sin 6 (mega-issue / multi-pain) promoted to its own visible Sin.
- Worked example titles tightened to 49-55 chars (was 84) to match
milstan's median of ~38 chars.
- Step 2 prose-ask replaces AskUserQuestion (the shared A/B/C
multiple-choice format silently broke free-text data collection
— runtime bug caught by fresh-context review).
- Three milstan reference issues at the top (#3684 / #3656 / #3657)
show the shape; explicitly flagged as owner-internal Step 0 mode.
- Six sins (was 10), four gates (was 11), no verbatim output blocks.
- Soft length cap: body 200–600 chars, up to ~1500 OK when complex,
over 1500 is a tell to split.
- Maintainer note at end ("fold first, add second") to prevent
re-bloating.
- Worked examples: register-honest (Gate 3) and sourced-not-invented
(Gate 3) each have one concrete source → slop → signal triplet.
Validated against eval set of 11 real leadbay/product issues:
- 5 reference goods (#3684/#3656/#3657/#3649 + LudoGranger #3676)
all PASS (owner-internal exit Step 0; #3676 reporter-to-team
passes all 4 gates).
- 5 slop seeds (#3667/#3670/#3641/#3669/#3665) all FAIL at the
appropriate gate.
- Fresh-context Agent subagent reviewed and surfaced 5 substantive
findings, all addressed in this commit.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two fresh-context Agent subagent reviews surfaced 10 substantive findings
across iters 5 and 8. This commit lands the structural fixes:
Iter-5 round (5 findings, all addressed):
- Gate 1 reframed from brittle keyword-grep to "subject is product vs
user" rule. Keyword list demoted to hints. Resolves the contradiction
with milstan's own reference issues.
- Example A/B titles tightened from 84c to 49-55c (in milstan's
median band of ~38-73c).
- Reporter hunch escape hatch acknowledged in Gate 1 as a regression.
- AskUserQuestion replaced with prose-ask in Step 2 — the A/B/C
multiple-choice format silently broke free-text data collection
(real runtime bug).
- Sin 4 (extrapolation) restructured into two named shapes plus
explicit carve-out for workflow-gap observations from dogfooders
(#3676 passes correctly).
Iter-8 round (5 findings, all addressed):
- Reporter-hunch voice fully unified: Iron Law and Pushback now mirror
Gate 1's regression frame; no place re-grants the slot unconditionally.
- Step 0 classifier rewritten from "issue topic" to "reporter role
vs target repo" — FE engineer ↔ leadbay/frontend etc. Closes the
FE-engineer-vs-sales-rep classification gap.
- Sin 4 carve-out generalized from "matches #3676 surface" to
"verb shape: descriptive workflow gap vs prescriptive strategy."
- Step 4 "<who> can't <do what>" pattern gained an explicit trap:
if the verb-phrase reads like a Jira card title, it's a Sin 3
launderer.
- Description block reconciled with Step 4's prefix advice
("bracket-tag triggers summon the skill from old habits; Step 4
will help drop them").
Additional iter-9 polish: Sin 6 (mega-issue) gained a worked example
(customer-call notes covering 3 frustrations → 3 separate issues).
Skill is now 626 lines (was 621 before iter-9; was 1423 starting state).
Structure: 4 gates, 6 sins, 1 Iron Law, Scope section + Step 0 protocol
check. Regression-tested against 11 real leadbay/product issues (5 GOOD
+ 5 SLOP + 1 FR) with zero drift across the iteration journey.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…section tightening) 4th fresh-context subagent verdict CLEAN — all 15 prior findings (3 rounds) verified ADDRESSED. Applied 2 of 3 minor polish items it surfaced: - Sin 4: distinguish 'forces [I did X]' (workaround, keep) from 'forces [product should X]' (smuggled prescription, cut). - Length section: lead with milstan's ~430-char median rather than cap-stacking, so reporters don't anchor on the 2000-char hard cap. The 3rd item ([For Sider] stale trigger) left in place — removing a routing trigger without verifying partner status is riskier than its zero-signal cost. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Five gates added to the Root-Cause Statement Gate, each traced to failure modes mined from 219 real backend diagnosis sessions (277 verbatim findings): - clause-level trigger proof: an unproven TRIGGER clause must be a hypothesis and cap to PROBABLE_CAUSE, not be welded into the cause sentence - second-source rule: a load-bearing link may stay "unobtainable" only after >=2 source attempts (DB + error-tracker class / logs / reachability) - no-fix-corroboration: confidence can't be raised via a fix/future commit - weigh-in-window-signal: an error-tracker/deploy signature matching the symptom's time window is evidence FOR; dismissing it needs an artifact - resource-exhaustion + trigger-class disambiguation: name which resource from current config, search the exception class, reconcile magnitude, and order competing trigger classes causally (name the upstream, not the effect) The gate already existed; agents routed around it by asserting an unproven trigger narrative. Validated with a relentless A/B on real pre-fix bugs, scoring the reasoning traces with a sealed-ground-truth judge: bug-3659 went 925 -> ~1210 / 1400 (PARTIAL -> HIT, reproduced), no regression on a second cross-repo bug (#3687). Independent fresh-context reviewer: EXTERNAL_ONLY. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Five gates added to the
/diagnosePhase 5.5 Root-Cause Statement Gate (+111 source lines), each traced to a failure mode mined from 219 real backend diagnosis sessions (277 verbatim findings, dataset inleadbay/backend.context/diagnose-improvement/):Why
The gate already existed, yet on a real bug the OLD skill still asserted an unproven "transient fault" trigger narrative — exactly the failure the dataset flags most (D6/D2/D7). These edits make the gate operate at clause granularity and at the cap/confidence decision points, so agents can't route around it.
Proof
A
/relentlessA/B, scoring the/diagnosereasoning traces (not the skill text) produced by subagents on genuinely buggy pre-fix code, judged by a sealed-ground-truth fresh-context judge:+31% on the discriminating bug, verdict flipped, no regression. Three independent fresh-context reviews; the last returned EXTERNAL_ONLY (no fixable improvement left). Full audit trail (every plan, trace, judgement, review) under
~/.leadbay/projects/leadbay-backend/relentless/diagnose-skill-hardening/.Generated for milstan.
🤖 Generated with Claude Code