docs: scope MAST multi-agent failure alignment; mark 3 low-adoption features experimental - #12
Merged
Merged
Conversation
…eatures experimental
Two independent items from the same roadmap-reprioritization pass.
1. docs/concepts/multi-agent-failures.md (design proposal, not
implemented): scopes what it would take to detect multi-agent
failures against the published MAST taxonomy (Cemri et al. 2025,
arXiv:2503.13657) — 14 failure modes, 3 categories, definitions
quoted verbatim from the taxonomy's own repository, not paraphrased
from secondary summaries.
Key finding: Step/Trajectory has no concept of agent identity today
— no MAST inter-agent failure mode is representable regardless of
classifier sophistication, until a Step.agent_id field exists. The
OTel GenAI semantic conventions already define gen_ai.agent.id/
gen_ai.agent.name on invoke_agent spans (verified against the live
spec, same family otel_ingest.py already reads from) — a real,
already-connected path to populate it, not a speculative one.
Maps all 14 modes into three honest groups instead of treating the
taxonomy as a checklist to satisfy:
- 2 already covered by existing FailureTypes, no new code needed:
Step Repetition -> LOOP_DETECTED (triage already has working,
tested code for the single-agent case; extending it across
agent_id boundaries is a small change, not a new concept),
Premature Termination -> PLAN_INCOMPLETE (CLAUDE.md's own
taxonomy table already defines PLAN_INCOMPLETE this way).
- 2 structurally-promising candidates worth a real measured
RulesClassifier prototype, not yet built: a verification-claim
vs. actual-outcome mismatch (No/Incorrect Verification), and
reusing triage's existing checkpoint/state machinery to detect an
unexpected state reset (Conversation Reset).
- 10 semantic-only modes that would need an LLMClassifier prompt
extension, evaluated before any of them become a stable
FailureType member.
Explicitly recommends no new FailureType members in phase 1, naming
the v0.7 HALLUCINATED_STATE/GOAL_DRIFT removal as the mistake not to
repeat — those were added without real classifier disambiguation
logic behind them, the exact trap 10 of these 14 modes would walk
into if added as enum members now. Registered in mkdocs.yml nav;
docs/known-limitations.md gets a new "Multi-agent systems" section
pointing to it.
2. Marked RedisSuspensionStore, RedisBreakerStore, and
compensating_rollback() (saga compensators) experimental in their
module docstrings, plus a README pointer on the one that's
README-documented (RedisSuspensionStore). Each is tested but has no
known production users as of this writing; the docstrings now say
so and ask adopters to open an issue with their use case. No
functional change — nothing removed, deprecated, or altered.
Verification: ruff check/format clean, pytest 801 passed (unchanged —
docs/docstrings only), mypy --strict clean, mkdocs build --strict
clean.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01M4WNEkbnKSx9mTg5Q1jX39
13 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this change and why?
Two independent items from the same roadmap-reprioritization pass, kept in one PR since both are docs/labeling-only and small.
1.
docs/concepts/multi-agent-failures.md(design proposal, not implemented): scopes what it would take to detect multi-agent failures against the published MAST taxonomy (Cemri et al. 2025, arXiv:2503.13657) — 14 failure modes, 3 categories, definitions quoted verbatim from the taxonomy's own repository, not paraphrased from secondary summaries.Key finding:
Step/Trajectoryhas no concept of agent identity today — no MAST inter-agent failure mode is representable regardless of classifier sophistication, until aStep.agent_idfield exists. The OTel GenAI semantic conventions already definegen_ai.agent.id/gen_ai.agent.nameoninvoke_agentspans (verified directly against the live spec, same attribute familyotel_ingest.pyalready reads from — a real, already-connected path to populate it, not a speculative one).Maps all 14 modes into three honest groups instead of treating the taxonomy as a checklist to satisfy:
FailureTypes, no new code needed: Step Repetition →LOOP_DETECTED(triage already has working, tested code for the single-agent case — extending it acrossagent_idboundaries is a small change, not a new concept), Premature Termination →PLAN_INCOMPLETE(CLAUDE.md's own taxonomy table already definesPLAN_INCOMPLETEthis way).RulesClassifierprototype, not yet built: a verification-claim-vs-actual-outcome mismatch (No/Incorrect Verification), and reusing triage's existing checkpoint/state machinery to detect an unexpected state reset (Conversation Reset).LLMClassifierprompt extension, evaluated before any of them become a stableFailureTypemember.Explicitly recommends no new
FailureTypemembers in phase 1, naming the v0.7HALLUCINATED_STATE/GOAL_DRIFTremoval as the mistake not to repeat — those were added without real classifier disambiguation logic behind them, exactly the trap 10 of these 14 modes would walk into if added as enum members now. Registered inmkdocs.yml's nav;docs/known-limitations.mdgets a new "Multi-agent systems" section pointing to it.2. Marked
RedisSuspensionStore,RedisBreakerStore, andcompensating_rollback()(saga compensators) experimental in their module docstrings, plus aREADME.mdpointer on the one that's README-documented (RedisSuspensionStore). Each is tested but has no known production users as of this writing; the docstrings now say so and ask adopters to open an issue with their use case. No functional change — nothing removed, deprecated, or had its behavior altered.Related issue
None
Type of change
triage/classifier/rules.py,triage/scorer/,tests/data/error_corpus_*.json)Checklist
description are clean. I am the author and I am responsible for this change.
(N/A — see the "Ownership" exception in this template for PRs an automated agent
session opens directly. Every other item below still applies in full.)
pytest tests/ -x --tb=shortpasses locally (801 passed, unchanged — docs/docstrings only)ruff check .,ruff format --check ., andmypy triage/ --strictare all cleanopenai/anthropic/langchain/langgraph/opentelemetry/etc. insidetriage/core — no code changes at all, only docstrings and docsdocs/concepts/ multi-agent-failures.mddocuments a proposal, not a shipped API, andCHANGELOG.md(under
[Unreleased]) covers both itemsFailureTypeorRecoveryAction— N/A, and deliberately not yet (see above)If this touches
rules.pyor an error corpus — N/A, no changes torules.pyor any corpus file.Anything reviewers should look at closely?
Whether the "already covered, no new code" mapping (Step Repetition →
LOOP_DETECTED, Premature Termination →PLAN_INCOMPLETE) actually holds up once someone tries to extendLOOP_DETECTEDacrossagent_idboundaries in phase 1 — I'm confident in the conceptual mapping from the taxonomy definitions, less confident that the code will turn out as small a change as the doc estimates until someone actually writes it. Also worth a second look: whethermkdocs.yml's "(proposal)" suffix on the nav entry is clear enough that this isn't documenting a shipped feature.ruff check .&&mypy triage/ --strict&&mkdocs build --strictwere also run clean locally (mkdocs build produces asite/directory that isn't committed).🤖 Generated with Claude Code
https://claude.ai/code/session_01M4WNEkbnKSx9mTg5Q1jX39
Generated by Claude Code