docs: MAST phase 2 investigated, negative result — no new rule shipped - #14
Merged
Merged
Conversation
Phase 2 of the MAST multi-agent alignment plan (docs/concepts/multi-agent-failures.md "Recommended phasing" item 2) investigated whether MAST 3.3 (verification-claim mismatch) and 2.1 (conversation reset) — both named "structurally-promising candidates worth a real measured prototype" in the phase 1 doc — could become new RulesClassifier rules. Neither can, for two distinct, evidenced reasons, not a generic "too hard" dismissal: - 3.3 splits into two sub-cases. A false verification claim followed by a real later error (MAST's ChatDev textBasedSpaceInvaders example) is already caught by every existing rule, since each one scans the full trajectory, not just the latest step — the "mismatch" detection adds no coverage a fresh scan doesn't already have. A false claim with no error at all (MAST's ChatDev TicTacToe/Sudoku examples: wrong output, clean exit) has no structural signal to match — telling "actually done" from "wrongly claimed done" needs task-specific semantic judgment. - 2.1 has two independent problems: a state_hash-repetition rule would false-positive on triage's own intentional RecoveryAction.ROLLBACK (which legitimately restores an earlier checkpoint's state_hash, and Step has no field distinguishing that from a framework's silent restart), and two real, cited GitHub issues (microsoft/autogen#1942, langchain-ai/langgraph#6064) confirm real-world conversation-reset failures are purely behavioral with zero error string or log line to match on. Both modes move to the semantic-only bucket (twelve of fourteen MAST modes now, up from ten) and wait for phase 3's LLMClassifier prompt extension. This is a complete, valid phase 2 outcome, not a stalled one: shipping a weak or false-positive-prone rule just to ship something would have broken the same 100%-precision-by-construction guarantee every rules.py pattern already holds itself to. No triage/ code changed — this is a documentation-only correction to the previously-stated plan, so it lands under CHANGELOG's ### Changed rather than ### Added. Updated files: - docs/concepts/multi-agent-failures.md — replaces the phase 1 doc's "structurally promising" section with the full investigation and both citations; updates the semantic-only count, the FailureType recommendation, phasing item 2's status, and the top-level Status line - docs/known-limitations.md — updates the "Multi-agent systems" closing pointer to reflect phase 2's negative result - CLAUDE.md — adds a design-decision entry so this isn't re-proposed without reading the investigation first - CHANGELOG.md — records the finding under [Unreleased] ### Changed - mkdocs.yml — drops the now-stale "(proposal)" nav suffix; phases 1-2 are shipped work/findings, only phase 3 remains a proposal Verified: ruff check, ruff format --check, pytest (806 passed), mypy --strict, mkdocs build --strict all clean. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01M4WNEkbnKSx9mTg5Q1jX39
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Ownership. This PR was opened directly by an automated Claude Code session working in
this repo — see the "Exception" paragraph in this template. The
Co-Authored-By:trailer andsession-link footer on the commit and this description are required by the platform this
session runs on and can't be suppressed from inside the PR; nothing else here is exempt, and a
human still decides whether to merge it.
What does this change and why?
Phase 2 of the MAST multi-agent alignment plan (
docs/concepts/multi-agent-failures.md's"Recommended phasing" item 2) investigated whether MAST 3.3 (verification-claim mismatch) and
2.1 (conversation reset) — both named "structurally-promising candidates worth a real measured
prototype" in the phase 1 doc — could become new
RulesClassifierrules. Neither can, fortwo distinct, evidenced reasons, not a generic "too hard" dismissal:
(MAST's own ChatDev
textBasedSpaceInvadersexample) is already caught by every existingrule, since each one scans the entire trajectory, not just the latest step — the "mismatch"
detection would add no coverage a fresh scan doesn't already have. A false claim with no
error at all (MAST's ChatDev TicTacToe/Sudoku examples: wrong output, clean exit) has no
structural signal to match on — telling "actually done" from "wrongly claimed done" needs
task-specific semantic judgment no regex can reach.
state_hash-repetition rule would false-positive ontriage's own intentional
RecoveryAction.ROLLBACK(which legitimately restores an earliercheckpoint's
state_hash, andStephas no field distinguishing that from a framework'ssilent restart), and two real, cited GitHub issues
(
microsoft/autogen#1942,langchain-ai/langgraph#6064)confirm real-world conversation-reset failures are purely behavioral with zero error string
or log line to match on.
Both modes move to the semantic-only bucket (twelve of fourteen MAST modes now, up from ten)
and wait for phase 3's
LLMClassifierprompt extension. This is a complete, valid phase 2outcome, not a stalled one: shipping a weak or false-positive-prone rule just to ship
something would have broken the same 100%-precision-by-construction guarantee every
rules.pypattern already holds itself to.No
triage/code changes — this is a documentation-only correction to the previously-statedplan.
Related issue
None
Type of change
Checklist
description are clean. I am the author and I am responsible for this change.
(N/A — see the "Ownership" exception above.)
pytest tests/ -x --tb=shortpasses locally (806 passed)ruff check .,ruff format --check ., andmypy triage/ --strictare all cleanopenai/anthropic/langchain/langgraph/opentelemetry/etc. insidetriage/core — notriage/files touched at alldocs/concepts/multi-agent-failures.md,docs/known-limitations.md,CLAUDE.md,CHANGELOG.md(under[Unreleased]→### Changed), andmkdocs.yml(droppedthe now-stale "(proposal)" nav suffix — phases 1–2 are shipped work, only phase 3 remains a
proposal)
FailureTypeorRecoveryActionaddedIf this touches
rules.pyor an error corpus — N/A, this PR touches no corpus orrules.pycode.Anything reviewers should look at closely?
This is a negative-result writeup, not a feature — the interesting part is whether the
reasoning in the "Investigated for phase 2" section of
docs/concepts/multi-agent-failures.mdholds up (in particular, the claim that every existing
RulesClassifierrule already scans thefull trajectory, so a later real error is always caught regardless of an earlier false
verification claim).
🤖 Generated with Claude Code
https://claude.ai/code/session_01M4WNEkbnKSx9mTg5Q1jX39
Generated by Claude Code