SOTA: attribute failure before mutating planning memory, execution memory, or skills

[ruvnet]

Finding

Two 2026-09-02 results converge on a useful self-improvement invariant.

CHIME, arXiv:2609.02074, separates planning and execution memory, attributes outcome to planning, execution, both, or neither, then mutates only the implicated bank. The originating team reports average gains of 2.96 and 3.68 absolute percentage points over the strongest baseline on two backbones while retaining 129 memories versus 3,585 for a strong baseline.

MASkills, arXiv:2609.02094, EMNLP 2026 Findings, decomposes multi-agent behavior into individually addressable skills and applies typed Induct, Refine, Consolidate, and Prune operations with held-out validation. The public implementation is MIT licensed.

Claim class: originating-team measured. MASkills has open implementation; CHIME code availability is not treated as independently verified here.

RuV hypothesis

Dream Machine should not mutate a global memory or skill library after every failure. It should first localize the failure to a stage and artifact, then mutate only that bounded surface.

Proposed hierarchy:

trajectory -> stage credit -> artifact credit -> one typed mutation -> held-out validation -> keep or rollback

Stages: planning, execution, retrieval/memory, tool interface, evaluator/infrastructure.

Artifacts: skill, planning memory, execution memory, context policy, router rule, harness code.

Experiment

Compare four fixed-budget conditions:

1. current Dream Machine selection and mutation
2. deterministic stage taxonomy before mutation
3. model-based stage attribution plus skill-level credit
4. hierarchical stage plus artifact attribution with typed operators

Hold model, tasks, seeds, tool versions, evaluator, total model tokens, wall budget, and promotion criteria fixed.

Metrics

Held-out task success, attribution agreement, false attribution, number of mutated artifacts, rollback rate, memory/skill count, token cost, wall time, model cost, protected slice regressions, evaluator mutations, and authority violations.

Falsification

If the deterministic stage taxonomy performs within variance of model-based attribution, prefer the deterministic mechanism. If attribution reduces mutation volume without improving held-out outcomes or cost, record the negative result and stop.

Promotion

Require at least 5 absolute percentage points higher held-out success at matched total cost, or equal success within 2 points with at least 30% fewer mutated artifacts and 20% lower evaluation cost. No evaluator mutation, no authority expansion, and no protected slice regression above 2 points.

Independent reproduction is tracked in MetaHarness #278.

[ruvnet]

2026-09-06 convergence update: DRACO (arXiv:2609.04094) strengthens this experiment rather than justifying a parallel reward subsystem. It generates dynamic rubrics during training, scores each completed trajectory once, then redistributes trajectory judgment over responsible steps with a closed form credit rule. The originating team reports +15.9 points over base on AppWorld, +5.3 over sparse ground truth GRPO, and out of domain Tau Bench improvement. Its public IBM repository is Apache 2.0. Add a DRACO style closed form credit arm to the frozen comparison after instrument qualification in MetaHarness #289 and #290. Keep the deterministic attribution arm as the simpler falsification baseline. Do not allow dynamic rubrics to mutate after outcomes are observed without starting a new experiment family, and do not let rubric score become authority.

[ruvnet]

2026-09-07 SOTA delta: add dual-view causal-inspired attribution as an experiment arm

DCFA, Dual-view Causal-inspired Attribution for Failure Reasoning in LLM-based Multi-agent Systems (arXiv:2609.04749, submitted 2026-09-04), is directly relevant to this existing experiment and does not justify a parallel subsystem.

The method is training-free. It combines a global structured dependency graph over the full trace with a local counterfactual-inspired refinement around the candidate decisive error. On Who&When across six LLMs, the originating team reports up to 8.27% higher step-level attribution accuracy than the compared state-of-the-art baselines.

Evidence class: originating-team measured, not independently reproduced by RuV.

Add condition 5

dual-view trace attribution: global dependency structure plus local counterfactual refinement, implemented using existing Dream Machine / MetaHarness trace primitives rather than copied code.

Additional falsification

The reported gain may come from spending more inference on a smaller candidate region rather than from causal structure itself. Include matched-token controls:

A. deterministic taxonomy

B. single-pass global model attribution

C. local-only counterfactual refinement

D. global plus local dual-view attribution

E. random candidate localization with the same local refinement budget

Require the existing promotion gate. Also report attribution latency and tokens per localized failure. If D does not beat B and C outside variance at matched model cost, reject the added graph machinery.

No evaluator mutation or authority expansion.