Skip to content

docs: scope MAST multi-agent failure alignment; mark 3 low-adoption features experimental - #12

Merged
mattekudacy merged 1 commit into
mainfrom
claude/serene-ritchie-g11dsf
Sep 11, 2026
Merged

mattekudacy merged 1 commit into
mainfrom
claude/serene-ritchie-g11dsf

Conversation

@mattekudacy

Copy link
Copy Markdown
Owner

What does this change and why?

Two independent items from the same roadmap-reprioritization pass, kept in one PR since both are docs/labeling-only and small.

1. docs/concepts/multi-agent-failures.md (design proposal, not implemented): scopes what it would take to detect multi-agent failures against the published MAST taxonomy (Cemri et al. 2025, arXiv:2503.13657) — 14 failure modes, 3 categories, definitions quoted verbatim from the taxonomy's own repository, not paraphrased from secondary summaries.

Key finding: Step/Trajectory has no concept of agent identity today — no MAST inter-agent failure mode is representable regardless of classifier sophistication, until a Step.agent_id field exists. The OTel GenAI semantic conventions already define gen_ai.agent.id/gen_ai.agent.name on invoke_agent spans (verified directly against the live spec, same attribute family otel_ingest.py already reads from — a real, already-connected path to populate it, not a speculative one).

Maps all 14 modes into three honest groups instead of treating the taxonomy as a checklist to satisfy:

  • 2 already covered by existing FailureTypes, no new code needed: Step Repetition → LOOP_DETECTED (triage already has working, tested code for the single-agent case — extending it across agent_id boundaries is a small change, not a new concept), Premature Termination → PLAN_INCOMPLETE (CLAUDE.md's own taxonomy table already defines PLAN_INCOMPLETE this way).
  • 2 structurally-promising candidates worth a real measured RulesClassifier prototype, not yet built: a verification-claim-vs-actual-outcome mismatch (No/Incorrect Verification), and reusing triage's existing checkpoint/state machinery to detect an unexpected state reset (Conversation Reset).
  • 10 semantic-only modes that would need an LLMClassifier prompt extension, evaluated before any of them become a stable FailureType member.

Explicitly recommends no new FailureType members in phase 1, naming the v0.7 HALLUCINATED_STATE/GOAL_DRIFT removal as the mistake not to repeat — those were added without real classifier disambiguation logic behind them, exactly the trap 10 of these 14 modes would walk into if added as enum members now. Registered in mkdocs.yml's nav; docs/known-limitations.md gets a new "Multi-agent systems" section pointing to it.

2. Marked RedisSuspensionStore, RedisBreakerStore, and compensating_rollback() (saga compensators) experimental in their module docstrings, plus a README.md pointer on the one that's README-documented (RedisSuspensionStore). Each is tested but has no known production users as of this writing; the docstrings now say so and ask adopters to open an issue with their use case. No functional change — nothing removed, deprecated, or had its behavior altered.

Related issue

None


Type of change

  • Bug fix
  • New feature (failure type, recovery strategy, classifier, adapter, checkpoint store, ...)
  • Breaking change
  • Documentation
  • Classifier / corpus change (triage/classifier/rules.py, triage/scorer/, tests/data/error_corpus_*.json)

Checklist

  • No AI-attribution trailers or badges anywhere in this PR — commit messages and this
    description are clean. I am the author and I am responsible for this change.
    (N/A — see the "Ownership" exception in this template for PRs an automated agent
    session opens directly. Every other item below still applies in full.)
  • pytest tests/ -x --tb=short passes locally (801 passed, unchanged — docs/docstrings only)
  • ruff check ., ruff format --check ., and mypy triage/ --strict are all clean
  • No new imports of openai/anthropic/langchain/langgraph/opentelemetry/etc. inside
    triage/ core — no code changes at all, only docstrings and docs
  • If this adds or changes public API — N/A, no API surface changed; docs/concepts/ multi-agent-failures.md documents a proposal, not a shipped API, and CHANGELOG.md
    (under [Unreleased]) covers both items
  • If this adds a FailureType or RecoveryAction — N/A, and deliberately not yet (see above)

If this touches rules.py or an error corpus — N/A, no changes to rules.py or any corpus file.


Anything reviewers should look at closely?

Whether the "already covered, no new code" mapping (Step Repetition → LOOP_DETECTED, Premature Termination → PLAN_INCOMPLETE) actually holds up once someone tries to extend LOOP_DETECTED across agent_id boundaries in phase 1 — I'm confident in the conceptual mapping from the taxonomy definitions, less confident that the code will turn out as small a change as the doc estimates until someone actually writes it. Also worth a second look: whether mkdocs.yml's "(proposal)" suffix on the nav entry is clear enough that this isn't documenting a shipped feature.

ruff check . && mypy triage/ --strict && mkdocs build --strict were also run clean locally (mkdocs build produces a site/ directory that isn't committed).

🤖 Generated with Claude Code

https://claude.ai/code/session_01M4WNEkbnKSx9mTg5Q1jX39


Generated by Claude Code

…eatures experimental

Two independent items from the same roadmap-reprioritization pass.

1. docs/concepts/multi-agent-failures.md (design proposal, not
   implemented): scopes what it would take to detect multi-agent
   failures against the published MAST taxonomy (Cemri et al. 2025,
   arXiv:2503.13657) — 14 failure modes, 3 categories, definitions
   quoted verbatim from the taxonomy's own repository, not paraphrased
   from secondary summaries.

   Key finding: Step/Trajectory has no concept of agent identity today
   — no MAST inter-agent failure mode is representable regardless of
   classifier sophistication, until a Step.agent_id field exists. The
   OTel GenAI semantic conventions already define gen_ai.agent.id/
   gen_ai.agent.name on invoke_agent spans (verified against the live
   spec, same family otel_ingest.py already reads from) — a real,
   already-connected path to populate it, not a speculative one.

   Maps all 14 modes into three honest groups instead of treating the
   taxonomy as a checklist to satisfy:
   - 2 already covered by existing FailureTypes, no new code needed:
     Step Repetition -> LOOP_DETECTED (triage already has working,
     tested code for the single-agent case; extending it across
     agent_id boundaries is a small change, not a new concept),
     Premature Termination -> PLAN_INCOMPLETE (CLAUDE.md's own
     taxonomy table already defines PLAN_INCOMPLETE this way).
   - 2 structurally-promising candidates worth a real measured
     RulesClassifier prototype, not yet built: a verification-claim
     vs. actual-outcome mismatch (No/Incorrect Verification), and
     reusing triage's existing checkpoint/state machinery to detect an
     unexpected state reset (Conversation Reset).
   - 10 semantic-only modes that would need an LLMClassifier prompt
     extension, evaluated before any of them become a stable
     FailureType member.

   Explicitly recommends no new FailureType members in phase 1, naming
   the v0.7 HALLUCINATED_STATE/GOAL_DRIFT removal as the mistake not to
   repeat — those were added without real classifier disambiguation
   logic behind them, the exact trap 10 of these 14 modes would walk
   into if added as enum members now. Registered in mkdocs.yml nav;
   docs/known-limitations.md gets a new "Multi-agent systems" section
   pointing to it.

2. Marked RedisSuspensionStore, RedisBreakerStore, and
   compensating_rollback() (saga compensators) experimental in their
   module docstrings, plus a README pointer on the one that's
   README-documented (RedisSuspensionStore). Each is tested but has no
   known production users as of this writing; the docstrings now say
   so and ask adopters to open an issue with their use case. No
   functional change — nothing removed, deprecated, or altered.

Verification: ruff check/format clean, pytest 801 passed (unchanged —
docs/docstrings only), mypy --strict clean, mkdocs build --strict
clean.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01M4WNEkbnKSx9mTg5Q1jX39
@mattekudacy
mattekudacy merged commit 751e735 into main Sep 11, 2026
7 checks passed
@mattekudacy
mattekudacy deleted the claude/serene-ritchie-g11dsf branch September 11, 2026 17:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants