Skip to content

docs: MAST phase 2 investigated, negative result — no new rule shipped - #14

Merged
mattekudacy merged 1 commit into
mainfrom
claude/serene-ritchie-g11dsf
Sep 12, 2026
Merged

mattekudacy merged 1 commit into
mainfrom
claude/serene-ritchie-g11dsf

Conversation

@mattekudacy

Copy link
Copy Markdown
Owner

Ownership. This PR was opened directly by an automated Claude Code session working in
this repo — see the "Exception" paragraph in this template. The Co-Authored-By: trailer and
session-link footer on the commit and this description are required by the platform this
session runs on and can't be suppressed from inside the PR; nothing else here is exempt, and a
human still decides whether to merge it.

What does this change and why?

Phase 2 of the MAST multi-agent alignment plan (docs/concepts/multi-agent-failures.md's
"Recommended phasing" item 2) investigated whether MAST 3.3 (verification-claim mismatch) and
2.1 (conversation reset) — both named "structurally-promising candidates worth a real measured
prototype" in the phase 1 doc — could become new RulesClassifier rules. Neither can, for
two distinct, evidenced reasons, not a generic "too hard" dismissal:

  • 3.3 splits into two sub-cases. A false verification claim followed by a real later error
    (MAST's own ChatDev textBasedSpaceInvaders example) is already caught by every existing
    rule, since each one scans the entire trajectory, not just the latest step — the "mismatch"
    detection would add no coverage a fresh scan doesn't already have. A false claim with no
    error at all (MAST's ChatDev TicTacToe/Sudoku examples: wrong output, clean exit) has no
    structural signal to match on — telling "actually done" from "wrongly claimed done" needs
    task-specific semantic judgment no regex can reach.
  • 2.1 has two independent problems: a state_hash-repetition rule would false-positive on
    triage's own intentional RecoveryAction.ROLLBACK (which legitimately restores an earlier
    checkpoint's state_hash, and Step has no field distinguishing that from a framework's
    silent restart), and two real, cited GitHub issues
    (microsoft/autogen#1942,
    langchain-ai/langgraph#6064)
    confirm real-world conversation-reset failures are purely behavioral with zero error string
    or log line to match on.

Both modes move to the semantic-only bucket (twelve of fourteen MAST modes now, up from ten)
and wait for phase 3's LLMClassifier prompt extension. This is a complete, valid phase 2
outcome, not a stalled one: shipping a weak or false-positive-prone rule just to ship
something would have broken the same 100%-precision-by-construction guarantee every
rules.py pattern already holds itself to.

No triage/ code changes — this is a documentation-only correction to the previously-stated
plan.

Related issue

None


Type of change

  • Documentation

Checklist

  • No AI-attribution trailers or badges anywhere in this PR — commit messages and this
    description are clean. I am the author and I am responsible for this change.
    (N/A — see the "Ownership" exception above.)
  • pytest tests/ -x --tb=short passes locally (806 passed)
  • ruff check ., ruff format --check ., and mypy triage/ --strict are all clean
  • No new imports of openai/anthropic/langchain/langgraph/opentelemetry/etc. inside
    triage/ core — no triage/ files touched at all
  • Documentation updated: docs/concepts/multi-agent-failures.md, docs/known-limitations.md,
    CLAUDE.md, CHANGELOG.md (under [Unreleased]### Changed), and mkdocs.yml (dropped
    the now-stale "(proposal)" nav suffix — phases 1–2 are shipped work, only phase 3 remains a
    proposal)
  • N/A — no new FailureType or RecoveryAction added

If this touches rules.py or an error corpus — N/A, this PR touches no corpus or
rules.py code.


Anything reviewers should look at closely?

This is a negative-result writeup, not a feature — the interesting part is whether the
reasoning in the "Investigated for phase 2" section of docs/concepts/multi-agent-failures.md
holds up (in particular, the claim that every existing RulesClassifier rule already scans the
full trajectory, so a later real error is always caught regardless of an earlier false
verification claim).

🤖 Generated with Claude Code

https://claude.ai/code/session_01M4WNEkbnKSx9mTg5Q1jX39


Generated by Claude Code

Phase 2 of the MAST multi-agent alignment plan (docs/concepts/multi-agent-failures.md
"Recommended phasing" item 2) investigated whether MAST 3.3 (verification-claim
mismatch) and 2.1 (conversation reset) — both named "structurally-promising
candidates worth a real measured prototype" in the phase 1 doc — could become new
RulesClassifier rules. Neither can, for two distinct, evidenced reasons, not a
generic "too hard" dismissal:

- 3.3 splits into two sub-cases. A false verification claim followed by a real
  later error (MAST's ChatDev textBasedSpaceInvaders example) is already caught by
  every existing rule, since each one scans the full trajectory, not just the
  latest step — the "mismatch" detection adds no coverage a fresh scan doesn't
  already have. A false claim with no error at all (MAST's ChatDev TicTacToe/Sudoku
  examples: wrong output, clean exit) has no structural signal to match — telling
  "actually done" from "wrongly claimed done" needs task-specific semantic
  judgment.
- 2.1 has two independent problems: a state_hash-repetition rule would
  false-positive on triage's own intentional RecoveryAction.ROLLBACK (which
  legitimately restores an earlier checkpoint's state_hash, and Step has no field
  distinguishing that from a framework's silent restart), and two real, cited
  GitHub issues (microsoft/autogen#1942, langchain-ai/langgraph#6064) confirm
  real-world conversation-reset failures are purely behavioral with zero error
  string or log line to match on.

Both modes move to the semantic-only bucket (twelve of fourteen MAST modes now, up
from ten) and wait for phase 3's LLMClassifier prompt extension. This is a
complete, valid phase 2 outcome, not a stalled one: shipping a weak or
false-positive-prone rule just to ship something would have broken the same
100%-precision-by-construction guarantee every rules.py pattern already holds
itself to.

No triage/ code changed — this is a documentation-only correction to the
previously-stated plan, so it lands under CHANGELOG's ### Changed rather than
### Added.

Updated files:
- docs/concepts/multi-agent-failures.md — replaces the phase 1 doc's "structurally
  promising" section with the full investigation and both citations; updates the
  semantic-only count, the FailureType recommendation, phasing item 2's status, and
  the top-level Status line
- docs/known-limitations.md — updates the "Multi-agent systems" closing pointer to
  reflect phase 2's negative result
- CLAUDE.md — adds a design-decision entry so this isn't re-proposed without
  reading the investigation first
- CHANGELOG.md — records the finding under [Unreleased] ### Changed
- mkdocs.yml — drops the now-stale "(proposal)" nav suffix; phases 1-2 are shipped
  work/findings, only phase 3 remains a proposal

Verified: ruff check, ruff format --check, pytest (806 passed), mypy --strict,
mkdocs build --strict all clean.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01M4WNEkbnKSx9mTg5Q1jX39
@mattekudacy
mattekudacy merged commit 0183354 into main Sep 12, 2026
7 checks passed
@mattekudacy
mattekudacy deleted the claude/serene-ritchie-g11dsf branch September 12, 2026 06:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants