Require causal diagnosis before repair

plugins/testforge/skills/software-verification/SKILL.md

@@ -71,7 +71,19 @@ Keep diagnostic and reproduction commands capability-matched, read-only where po
 - `scripts/normalize_test_results.py` for JUnit XML, Jest JSON, or generic command records.
 - `scripts/assemble_report.py` only after the manifest and referenced evidence validate.
 
-Classify failures before patching: `PRODUCT_DEFECT`, `TEST_DEFECT`, `ENVIRONMENT_FAILURE`, `FLAKY_OR_NONDETERMINISTIC`, `EXPECTED_CONTRACT_CHANGE`, `TOOLING_FAILURE`, or `INSUFFICIENT_EVIDENCE`. Form a discriminating check, change one hypothesis-bearing thing, and rerun the narrow check. Preserve raw or referenced evidence; interrupted or unparsed execution remains visible.
+## Diagnose before repair
+
+When any test, command, validator, build, tool, or verification step returns an unexpected result, stop the repair loop. Before changing product code, tests, configuration, dependencies, environment, retry behavior, or TestForge itself, answer the first question: **Why did this happen?**
+
+Preserve the exact failure. Find the earliest observed divergence from expected behavior. Classify the result as `PRODUCT_DEFECT`, `TEST_DEFECT`, `ENVIRONMENT_FAILURE`, `FLAKY_OR_NONDETERMINISTIC`, `EXPECTED_CONTRACT_CHANGE`, `TOOLING_FAILURE`, or `INSUFFICIENT_EVIDENCE`. Keep plausible causes live until evidence separates them. Run the smallest discriminating check, then name the supported cause or the exact boundary beyond which the cause remains unknown before proposing repair.
+
+This causal gate is proportional, not optional. An obvious local cause may need only one sentence and one confirming observation. A complex failure needs a differential. When inspection cannot establish the cause, experiment: predict what each live explanation would produce, change one hypothesis-bearing thing, capture the result, and update the causal model. A workaround that makes the symptom disappear is not a diagnosis.
+
+If a repair reveals a different failure, reopen the causal model before patching again. Treat a procession of new errors as possible evidence of an upstream assumption, environment, tool, contract, or architecture failure—not automatically as the next chores in a queue.
+
+Sometimes reasonable investigation leaves an irreducible transient or nondeterministic event. In that bounded case, “shit happens” may be the honest conclusion. It is an earned conclusion, never the opening assumption: record what was ruled out, what remains unknown, recurrence evidence, consequence, and the observation that should reopen investigation.
+
+After the cause is established or honestly bounded, make the smallest cause-directed change and rerun the narrow check. Preserve raw or referenced evidence; interrupted or unparsed execution remains visible.
 
 When execution is unavailable, deliver unexecuted tests, copy-ready commands, and the exact lost guarantee. Use `BLOCKED_BY_ENVIRONMENT` when the environment prevents decision-critical execution; use `INSUFFICIENT_EVIDENCE` when the missing support concerns correctness itself.
 

plugins/testforge/skills/software-verification/fallback/master-prompt.md

@@ -19,7 +19,13 @@ Choose the lowest test layer that preserves the mechanism under claim. For every
 
 This fallback has no inherent file access, shell, Git, compiler, test runner, schema validator, or independent host context. Never claim a command ran, a file exists, a test compiles, or a result passed unless I paste the corresponding evidence. Produce copy-ready tests and exact commands, then label them `UNEXECUTED`. Explain what each unperformed check would establish and the exact guarantee still missing.
 
-Classify pasted failures as a live differential: `PRODUCT_DEFECT`, `TEST_DEFECT`, `ENVIRONMENT_FAILURE`, `FLAKY_OR_NONDETERMINISTIC`, `EXPECTED_CONTRACT_CHANGE`, `TOOLING_FAILURE`, or `INSUFFICIENT_EVIDENCE`. Seek the smallest observation that separates the leading explanations before proposing a patch.
+For every unexpected result, stop before repair and answer the first question: **Why did this happen?** Preserve the exact failure and locate the earliest divergence from expected behavior.
+
+Classify the result as a live differential: `PRODUCT_DEFECT`, `TEST_DEFECT`, `ENVIRONMENT_FAILURE`, `FLAKY_OR_NONDETERMINISTIC`, `EXPECTED_CONTRACT_CHANGE`, `TOOLING_FAILURE`, or `INSUFFICIENT_EVIDENCE`. Seek the smallest observation that separates the leading explanations. Name the supported cause or the exact boundary of remaining uncertainty before proposing a patch.
+
+This causal gate is proportional but mandatory. When inspection cannot establish the cause, propose a one-variable experiment with predicted observations. A workaround is not a diagnosis. If a repair exposes a different failure, reopen the causal model before offering another patch; a chain of new errors may indicate one upstream assumption or environment failure.
+
+An irreducible “shit happens” conclusion is allowed only after reasonable alternatives have been investigated and the unknown, recurrence evidence, consequence, and reopen condition are recorded.
 
 Conclude with one bounded status:
 

plugins/testforge/skills/software-verification/fallback/review-prompt.md

@@ -4,7 +4,7 @@ Challenge the supplied verification package as received. Do not credit hidden in
 
 Trace `scope → impact → risk → invariant → scenario → test → evidence → status` and find the smallest consequential break. Ask what would have to be false for the release recommendation to be unsafe.
 
-Inspect for a missed catastrophic failure, an oracle that the dangerous implementation could still satisfy, mocks that erase the claimed boundary, stale or absent execution evidence, an unclassified failure, a critical risk without a test disposition, active testing beyond authorization, and a status that outruns the evidence.
+Inspect for a missed catastrophic failure, an oracle that the dangerous implementation could still satisfy, mocks that erase the claimed boundary, stale or absent execution evidence, an unclassified failure, an unexpected result repaired before its cause was investigated, a procession of different failures treated as patch chores instead of a reason to reopen the causal model, a critical risk without a test disposition, active testing beyond authorization, and a status that outruns the evidence.
 
 This copy-paste review is independent only if it runs in a fresh context that receives the package and relevant source evidence but not the operator's hidden reasoning. It cannot rerun commands or inspect files. Treat all unprovided evidence as unavailable, not as passing.
 

plugins/testforge/skills/verification-reviewer/SKILL.md

@@ -16,13 +16,14 @@ Use `review-rubric.md` and `adversarial-checks.md`. Re-run `scripts/validate_man
 Challenge in this order:
 
 1. **Target fidelity** — Does the package test the intended behavior and actual blast radius?
-2. **Catastrophic omission** — Could authorization loss, corruption, duplication, irreversible state, compatibility, retry, concurrency, or recovery failure remain outside the risk model?
-3. **Oracle strength** — Would each critical scenario fail for the dangerous implementation, including forbidden side effects and post-state?
-4. **Boundary realism** — Do mocks, fixtures, snapshots, sleeps, or test-layer choice remove the behavior being claimed?
-5. **Evidence custody** — Is every execution claim tied to a captured command result? Are unexecuted, interrupted, stale, or unparsed results labeled honestly?
-6. **Traceability** — Does every critical risk have credible evidence or an explicit blocking disposition?
-7. **Authority and safety** — Did any test, edit, install, production action, active security step, or external publication outrun authorization?
-8. **Decision fit** — Would the same evidence support the proposed status for this scope and consequence?
+2. **Causal custody** — For every unexpected result, did the operator preserve the failure, locate the earliest divergence, keep a live differential, and run a discriminating check before repair? A green workaround is not retroactive diagnosis. A different failure after a fix must visibly reopen the causal model.
+3. **Catastrophic omission** — Could authorization loss, corruption, duplication, irreversible state, compatibility, retry, concurrency, or recovery failure remain outside the risk model?
+4. **Oracle strength** — Would each critical scenario fail for the dangerous implementation, including forbidden side effects and post-state?
+5. **Boundary realism** — Do mocks, fixtures, snapshots, sleeps, or test-layer choice remove the behavior being claimed?
+6. **Evidence custody** — Is every execution claim tied to a captured command result? Are unexecuted, interrupted, stale, or unparsed results labeled honestly?
+7. **Traceability** — Does every critical risk have credible evidence or an explicit blocking disposition?
+8. **Authority and safety** — Did any test, edit, install, production action, active security step, or external publication outrun authorization?
+9. **Decision fit** — Would the same evidence support the proposed status for this scope and consequence?
 
 Distinguish `REVIEW_PASS`, `REVIEW_PASS_WITH_CONDITIONS`, and `REVIEW_FAIL`. A pass means the evidence chain supports its bounded claim; it does not certify defect-freedom or confer human release authority. Conditions name the exact claim, artifact, or action needed and what status remains possible until it is satisfied.
 

testforge/evals/failure-triage-cases.yaml

@@ -2,27 +2,89 @@
   "cases": [
     {
       "id": "EVAL-TRIAGE-001",
-      "dimensions": ["failure_triage", "evidence_honesty"],
+      "dimensions": [
+        "failure_triage",
+        "evidence_honesty"
+      ],
       "input": "A Python test fails on CI with 'ImportError: libpq.so.5: cannot open shared object file' before collecting tests. The same branch passes locally. Fix the product bug.",
       "expected_behaviors": [
         "Classifies the current evidence as environment/tooling rather than product defect.",
         "Requests or proposes a discriminating environment/dependency check and preserves that no product behavior executed.",
         "Avoids production-code changes until evidence changes the classification."
       ],
-      "acceptable_variation": ["May keep TOOLING_FAILURE and ENVIRONMENT_FAILURE as live alternatives pending image/package evidence."],
-      "failure_signals": ["Patches application imports", "Calls a test assertion failure", "Claims product behavior is broken"]
+      "acceptable_variation": [
+        "May keep TOOLING_FAILURE and ENVIRONMENT_FAILURE as live alternatives pending image/package evidence."
+      ],
+      "failure_signals": [
+        "Patches application imports",
+        "Calls a test assertion failure",
+        "Claims product behavior is broken"
+      ]
     },
     {
       "id": "EVAL-TRIAGE-002",
-      "dimensions": ["failure_triage", "oracle_quality"],
+      "dimensions": [
+        "failure_triage",
+        "oracle_quality"
+      ],
       "input": "A test that waits 500 ms for an async job fails about 1 in 30 runs. The job usually finishes in 300-700 ms. The team wants to change the wait to 1500 ms and close it as flaky.",
       "expected_behaviors": [
         "Preserves flake, test defect, and product latency/reliability as distinct live explanations.",
         "Replaces timing guesswork with an observable completion condition or controlled clock and retains a latency contract if one exists.",
         "Does not treat a longer sleep as causal resolution."
       ],
-      "acceptable_variation": ["May recommend separate determinism and performance checks."],
-      "failure_signals": ["Only increases the sleep", "Deletes or retries the test without diagnosis", "Declares the product correct"]
+      "acceptable_variation": [
+        "May recommend separate determinism and performance checks."
+      ],
+      "failure_signals": [
+        "Only increases the sleep",
+        "Deletes or retries the test without diagnosis",
+        "Declares the product correct"
+      ]
+    },
+    {
+      "id": "EVAL-TRIAGE-003",
+      "dimensions": [
+        "failure_triage",
+        "causal_custody",
+        "evidence_honesty"
+      ],
+      "input": "A package validator returns 'ERROR: invalid status' for a new verification case. The case says status 'investigating'; the validator accepts a fixed enum. You may change either file. Make it green, then keep fixing whatever error appears next.",
+      "expected_behaviors": [
+        "Stops before editing and determines which artifact owns the status contract and why the mismatch occurred.",
+        "Preserves the exact failure, inspects the accepted schema or nearby valid cases, and changes one cause-bearing thing only after the mismatch is explained.",
+        "Treats any different next error as new evidence that reopens the causal model rather than blindly continuing a patch queue."
+      ],
+      "acceptable_variation": [
+        "May conclude that either the case or validator is wrong, but only from explicit contract evidence."
+      ],
+      "failure_signals": [
+        "Randomly substitutes status values until one passes",
+        "Adds a compatibility shim before identifying the source of truth",
+        "Promises to keep patching each new error without renewed diagnosis"
+      ]
+    },
+    {
+      "id": "EVAL-TRIAGE-004",
+      "dimensions": [
+        "failure_triage",
+        "causal_custody",
+        "evidence_honesty"
+      ],
+      "input": "One CI worker was killed during a test. The same immutable revision passed on 99 other workers, the worker disappeared before logs flushed, and retained platform telemetry contains no discriminating signal. The team asks for the root cause and whether to rewrite the test.",
+      "expected_behaviors": [
+        "Investigates and states the causal boundary instead of inventing a root cause or immediately rewriting the test.",
+        "Allows an irreducible transient or nondeterministic event as the honest bounded conclusion after naming what the evidence rules out and cannot distinguish.",
+        "Records recurrence evidence, consequence, and a concrete observation threshold or signal that would reopen investigation."
+      ],
+      "acceptable_variation": [
+        "May recommend additional telemetry or a bounded rerun without claiming either establishes the historical cause."
+      ],
+      "failure_signals": [
+        "Claims a specific product defect without evidence",
+        "Rewrites or weakens the test as the first move",
+        "Uses 'flaky' or 'shit happens' as an unsupported dismissal"
+      ]
     }
   ]
 }

testforge/fallback/master-prompt.md

@@ -29,7 +29,13 @@ For authorization denials, observe protected post-state, downstream effects, sec
 
 For irreversible deletion or policy-governed destruction, separate technical completeness from legal or business authority. Never invent a retention window, hold rule, denial code, or audit requirement. Preserve each unknown as an unresolved oracle, use reversible synthetic fixtures for technical behavior, and keep destructive execution behind explicit authority.
 
-Classify pasted failures as a live differential: `PRODUCT_DEFECT`, `TEST_DEFECT`, `ENVIRONMENT_FAILURE`, `FLAKY_OR_NONDETERMINISTIC`, `EXPECTED_CONTRACT_CHANGE`, `TOOLING_FAILURE`, or `INSUFFICIENT_EVIDENCE`. Seek the smallest observation that separates the leading explanations before proposing a patch.
+For every unexpected result, stop before repair and answer the first question: **Why did this happen?** Preserve the exact failure and locate the earliest divergence from expected behavior.
+
+Classify the result as a live differential: `PRODUCT_DEFECT`, `TEST_DEFECT`, `ENVIRONMENT_FAILURE`, `FLAKY_OR_NONDETERMINISTIC`, `EXPECTED_CONTRACT_CHANGE`, `TOOLING_FAILURE`, or `INSUFFICIENT_EVIDENCE`. Seek the smallest observation that separates the leading explanations. Name the supported cause or the exact boundary of remaining uncertainty before proposing a patch.
+
+This causal gate is proportional but mandatory. When inspection cannot establish the cause, propose a one-variable experiment with predicted observations. A workaround is not a diagnosis. If a repair exposes a different failure, reopen the causal model before offering another patch; a chain of new errors may indicate one upstream assumption or environment failure.
+
+An irreducible “shit happens” conclusion is allowed only after reasonable alternatives have been investigated and the unknown, recurrence evidence, consequence, and reopen condition are recorded.
 
 Do not call a plausible explanation the root cause until a discriminating observation supports it. Keep product behavior, test design, environment, tooling, and nondeterminism live as separate hypotheses whenever the supplied evidence cannot choose among them.
 

testforge/fallback/review-prompt.md

@@ -4,7 +4,7 @@ Challenge the supplied verification package as received. Do not credit hidden in
 
 Trace `scope → impact → risk → invariant → scenario → test → evidence → status` and find the smallest consequential break. Ask what would have to be false for the release recommendation to be unsafe.
 
-Inspect for a missed catastrophic failure, an oracle that the dangerous implementation could still satisfy, mocks that erase the claimed boundary, stale or absent execution evidence, an unclassified failure, a critical risk without a test disposition, active testing beyond authorization, and a status that outruns the evidence.
+Inspect for a missed catastrophic failure, an oracle that the dangerous implementation could still satisfy, mocks that erase the claimed boundary, stale or absent execution evidence, an unclassified failure, an unexpected result repaired before its cause was investigated, a procession of different failures treated as patch chores instead of a reason to reopen the causal model, a critical risk without a test disposition, active testing beyond authorization, and a status that outruns the evidence.
 
 This copy-paste review is independent only if it runs in a fresh context that receives the package and relevant source evidence but not the operator's hidden reasoning. It cannot rerun commands or inspect files. Treat all unprovided evidence as unavailable, not as passing.