feat(activation): drop the Claude-only early-stop restriction, pin coder-eval 0.9.1

tests/.coder-eval-version

@@ -1 +1 @@
-0.9.0
+0.9.1

tests/README.md

@@ -178,7 +178,7 @@ Run-time caps live under `defaults.run_limits` (see coder_eval `RunLimits`).
 | `smoke-windows.yaml` | tempdir | PR-gate smoke (Windows RPA only) | 40 | 900s | 900s |
 | `activation.yaml` | tempdir | Skill activation classifier (benchmark) | 3 + early-stop | 360s | 120s |
 
-`activation.yaml` is a different shape from the tiered configs above — it runs the agent against single-prompt rows to measure whether the right skill fires (precision/recall/F1 per skill). Rows get a small turn budget (`max_turns: 3`) with `stop_early: true`: the armed `skill_triggered` criteria (`stop_when: auto`) end a row as soon as its outcome is live-decided. A positive row pass-stops the moment the expected skill engages; a negative row fail-stops on its first engagement. A wrong-skill engagement alone does NOT end a positive row — fail-stop is deferred while the row's positive criterion is still undecided, so a positive row that only misfires runs to the cap, as do rows with no engagement. Decided rows cost ~1 turn and a late-but-correct invocation is no longer truncated. Requires coder_eval >= 0.8.10 and the Claude Code agent (`--type codex`/`antigravity` must add `-D run_limits.stop_early=false`). It's an opt-in benchmark, not a smoke gate. See [`tasks/activation/README.md`](tasks/activation/README.md).
+`activation.yaml` is a different shape from the tiered configs above — it runs the agent against single-prompt rows to measure whether the right skill fires (precision/recall/F1 per skill). Rows get a small turn budget (`max_turns: 3`) with `stop_early: true`: the armed `skill_triggered` criteria (`stop_when: auto`) end a row as soon as its outcome is live-decided. A positive row pass-stops the moment the expected skill engages; a negative row fail-stops on its first engagement. A wrong-skill engagement alone does NOT end a positive row — fail-stop is deferred while the row's positive criterion is still undecided, so a positive row that only misfires runs to the cap, as do rows with no engagement. Decided rows cost ~1 turn and a late-but-correct invocation is no longer truncated. Requires coder_eval >= 0.9.1. It's an opt-in benchmark, not a smoke gate. See [`tasks/activation/README.md`](tasks/activation/README.md).
 
 For **A/B comparisons between two skill variants** (e.g. `main` vs a feature branch, or two historical commits), see [`experiments/skill-comparison-playbook.md`](experiments/skill-comparison-playbook.md) and the [`experiments/skill-comparison-template.yaml`](experiments/skill-comparison-template.yaml). The playbook covers worktree setup, SHA pinning for reproducibility, getting N>1, and interpreting divergent tasks. To automate the whole flow, use the `/skill-compare <ref_a> <ref_b> [task_selector] [n_reps]` slash command — each ref can be a branch name or a commit SHA, and `task_selector` accepts a skill name (`uipath-maestro-flow`), tag list (`tags:smoke,init`), or path globs (`paths:tasks/uipath-maestro-flow/*.yaml`).
 

tests/experiments/activation.yaml

@@ -11,7 +11,7 @@ defaults:
     # Activation is a recognition decision, not a work loop. We give the agent a
     # small turn budget (3) rather than a hard 1-turn cap, then arm early-stop on
     # the stacked skill_triggered criteria (stop_when: auto, see the task file).
-    # Under coder_eval >= 0.8.10 `auto` arms each instance's own decidable
+    # Under coder_eval >= 0.9.1 `auto` arms each instance's own decidable
     # polarity: the row's positive criterion (skill_name == expected_skill) arms
     # pass, the distractors arm fail. A positive row pass-stops the moment the
     # expected skill is engaged — turn 1, with the exact verdict a max_turns:1
@@ -21,23 +21,19 @@ defaults:
     # wrong-skill engagement alone does NOT stop it (fail-stop is deferred
     # while the row's positive criterion is undecided), so misfire-only and
     # no-engagement rows run to the cap — keep it small. (`decided` is invalid
-    # here: >= 0.8.10's any-engagement latch narrows each instance to ONE
-    # decidable polarity.) See coder_eval "early stop on criterion".
+    # here: the any-engagement latch narrows each instance to ONE decidable
+    # polarity.) See coder_eval "early stop on criterion".
     #
-    # HARNESS CONSTRAINT: cooperative early-stop is Claude Code-only. codex and
-    # antigravity agents do not support it, and an armed run with such an agent
-    # is a hard EarlyStopConfigError at resolution — runs using `--type codex`
-    # or `--type antigravity` MUST add `-D run_limits.stop_early=false` (rows
-    # may then use up to max_turns: 3 — a cap, not a floor). Note: unarmed runs
-    # are the authoritative ones for precision — a pass-stop can hide a later
+    # Note: unarmed runs (`-D run_limits.stop_early=false`) are the
+    # authoritative ones for precision — a pass-stop can hide a later
     # distractor misfire, so armed-run precision is optimistic by comparison
     # (recall.yes is unaffected; see tasks/activation/README.md).
     max_turns: 3
     stop_early: true
     # Scaled with max_turns: a row that legitimately uses all 3 turns (a true
-    # negative, or an unarmed codex/antigravity run) must have wall-clock room
-    # for them — a task_timeout below max_turns x turn_timeout silently drops
-    # the row from every criterion denominator instead of grading it.
+    # negative, or any unarmed run) must have wall-clock room for them — a
+    # task_timeout below max_turns x turn_timeout silently drops the row from
+    # every criterion denominator instead of grading it.
     task_timeout: 360
     turn_timeout: 120
 

tests/scripts/activation_gate.py

@@ -62,7 +62,7 @@ def _build_task_yaml(skill: str, dataset: Path) -> str:
     # stop_when: auto is REQUIRED, not an optimization: the experiment's
     # defaults arm run_limits.stop_early, and an armed run with no stop
     # criterion is a hard EarlyStopConfigError at resolution (coder_eval >=
-    # 0.8.10). Gate rows are all positives, so auto arms pass-stop: the run
+    # 0.9.1). Gate rows are all positives, so auto arms pass-stop: the run
     # ends the moment {skill} engages, with the verdict a full run would
     # have produced (any-engagement latch is monotonic), and a recall miss
     # never fires a live event so it still runs to the cap. With a single

tests/tasks/activation/README.md

@@ -36,7 +36,7 @@ uv run --project /path/to/coder_eval coder-eval run \
 
 Reports land in `tmp/<run-id>/`. The suite gate fails per criterion on `recall.yes < 0.70`. Class imbalance (~50 yes / ~1260 no per skill) makes accuracy and recall.no trivially high; recall.yes is the only meaningful gate.
 
-Early stop: the experiment arms `run_limits.stop_early` and every criterion sets `stop_when: auto` (coder_eval >= 0.8.10), so a row ends as soon as its outcome is live-decided. A positive row pass-stops the moment its expected skill engages; a negative row fail-stops on its first engagement. A wrong-skill engagement alone does not end a positive row — fail-stop is deferred while the row's positive criterion is undecided, so a positive that only misfires runs to `max_turns: 3`, as do no-engagement rows. Claude Code agent only: for `--type codex` or `--type antigravity` add `-D run_limits.stop_early=false`. For authoritative precision numbers (a pass-stop can hide a distractor misfire on a later tool call), run with `-D run_limits.stop_early=false`.
+Early stop: the experiment arms `run_limits.stop_early` and every criterion sets `stop_when: auto` (coder_eval >= 0.9.1; below that, an armed `--type codex`/`antigravity` run fails at resolution with `EarlyStopConfigError`), so a row ends as soon as its outcome is live-decided. A positive row pass-stops the moment its expected skill engages; a negative row fail-stops on its first engagement. A wrong-skill engagement alone does not end a positive row — fail-stop is deferred while the row's positive criterion is undecided, so a positive that only misfires runs to `max_turns: 3`, as do no-engagement rows. For authoritative precision numbers (a pass-stop can hide a distractor misfire on a later tool call), run with `-D run_limits.stop_early=false`.
 
 ## Adding a new skill
 
@@ -46,7 +46,7 @@ Early stop: the experiment arms `run_limits.stop_early` and every criterion sets
 
 ## Cost
 
-On Sonnet 4.6 via Bedrock, ~$0.05–$0.10 per row. The dataset is ~1309 rows total (1252 positives + 57 negatives). The agent runs ONCE per row regardless of criteria count, and the 23 stacked criteria are pure-Python evaluation against the same trace, so the full benchmark costs **~$65–130**. Those numbers assume the default armed run, where decided rows end at ~1 turn; an unarmed run (`-D run_limits.stop_early=false` — required for codex/antigravity and for authoritative precision) lets every row use up to `max_turns: 3`, so budget roughly 2–3× for it. Use `--sample N` for cheaper iteration (note: `--sample` slices first-N, which biases toward the first-listed paths; useful for smoke runs, not for metrics).
+On Sonnet 4.6 via Bedrock, ~$0.05–$0.10 per row. The dataset is ~1309 rows total (1252 positives + 57 negatives). The agent runs ONCE per row regardless of criteria count, and the 23 stacked criteria are pure-Python evaluation against the same trace, so the full benchmark costs **~$65–130**. Those numbers assume the default armed run, where decided rows end at ~1 turn; an unarmed run (`-D run_limits.stop_early=false` — optional for routine checks, required for authoritative precision) lets every row use up to `max_turns: 3`, so budget roughly 2–3× for it. Use `--sample N` for cheaper iteration (note: `--sample` slices first-N, which biases toward the first-listed paths; useful for smoke runs, not for metrics).
 
 ## Provenance