diff --git a/tests/.coder-eval-version b/tests/.coder-eval-version index ac39a106c4..f374f6662e 100644 --- a/tests/.coder-eval-version +++ b/tests/.coder-eval-version @@ -1 +1 @@ -0.9.0 +0.9.1 diff --git a/tests/README.md b/tests/README.md index 1b79348bfe..85c6069925 100644 --- a/tests/README.md +++ b/tests/README.md @@ -178,7 +178,7 @@ Run-time caps live under `defaults.run_limits` (see coder_eval `RunLimits`). | `smoke-windows.yaml` | tempdir | PR-gate smoke (Windows RPA only) | 40 | 900s | 900s | | `activation.yaml` | tempdir | Skill activation classifier (benchmark) | 3 + early-stop | 360s | 120s | -`activation.yaml` is a different shape from the tiered configs above — it runs the agent against single-prompt rows to measure whether the right skill fires (precision/recall/F1 per skill). Rows get a small turn budget (`max_turns: 3`) with `stop_early: true`: the armed `skill_triggered` criteria (`stop_when: auto`) end a row as soon as its outcome is live-decided. A positive row pass-stops the moment the expected skill engages; a negative row fail-stops on its first engagement. A wrong-skill engagement alone does NOT end a positive row — fail-stop is deferred while the row's positive criterion is still undecided, so a positive row that only misfires runs to the cap, as do rows with no engagement. Decided rows cost ~1 turn and a late-but-correct invocation is no longer truncated. Requires coder_eval >= 0.8.10 and the Claude Code agent (`--type codex`/`antigravity` must add `-D run_limits.stop_early=false`). It's an opt-in benchmark, not a smoke gate. See [`tasks/activation/README.md`](tasks/activation/README.md). +`activation.yaml` is a different shape from the tiered configs above — it runs the agent against single-prompt rows to measure whether the right skill fires (precision/recall/F1 per skill). Rows get a small turn budget (`max_turns: 3`) with `stop_early: true`: the armed `skill_triggered` criteria (`stop_when: auto`) end a row as soon as its outcome is live-decided. A positive row pass-stops the moment the expected skill engages; a negative row fail-stops on its first engagement. A wrong-skill engagement alone does NOT end a positive row — fail-stop is deferred while the row's positive criterion is still undecided, so a positive row that only misfires runs to the cap, as do rows with no engagement. Decided rows cost ~1 turn and a late-but-correct invocation is no longer truncated. Requires coder_eval >= 0.9.1. It's an opt-in benchmark, not a smoke gate. See [`tasks/activation/README.md`](tasks/activation/README.md). For **A/B comparisons between two skill variants** (e.g. `main` vs a feature branch, or two historical commits), see [`experiments/skill-comparison-playbook.md`](experiments/skill-comparison-playbook.md) and the [`experiments/skill-comparison-template.yaml`](experiments/skill-comparison-template.yaml). The playbook covers worktree setup, SHA pinning for reproducibility, getting N>1, and interpreting divergent tasks. To automate the whole flow, use the `/skill-compare [task_selector] [n_reps]` slash command — each ref can be a branch name or a commit SHA, and `task_selector` accepts a skill name (`uipath-maestro-flow`), tag list (`tags:smoke,init`), or path globs (`paths:tasks/uipath-maestro-flow/*.yaml`). diff --git a/tests/experiments/activation.yaml b/tests/experiments/activation.yaml index b981eed05a..0121f8442d 100644 --- a/tests/experiments/activation.yaml +++ b/tests/experiments/activation.yaml @@ -11,7 +11,7 @@ defaults: # Activation is a recognition decision, not a work loop. We give the agent a # small turn budget (3) rather than a hard 1-turn cap, then arm early-stop on # the stacked skill_triggered criteria (stop_when: auto, see the task file). - # Under coder_eval >= 0.8.10 `auto` arms each instance's own decidable + # Under coder_eval >= 0.9.1 `auto` arms each instance's own decidable # polarity: the row's positive criterion (skill_name == expected_skill) arms # pass, the distractors arm fail. A positive row pass-stops the moment the # expected skill is engaged — turn 1, with the exact verdict a max_turns:1 @@ -21,23 +21,19 @@ defaults: # wrong-skill engagement alone does NOT stop it (fail-stop is deferred # while the row's positive criterion is undecided), so misfire-only and # no-engagement rows run to the cap — keep it small. (`decided` is invalid - # here: >= 0.8.10's any-engagement latch narrows each instance to ONE - # decidable polarity.) See coder_eval "early stop on criterion". + # here: the any-engagement latch narrows each instance to ONE decidable + # polarity.) See coder_eval "early stop on criterion". # - # HARNESS CONSTRAINT: cooperative early-stop is Claude Code-only. codex and - # antigravity agents do not support it, and an armed run with such an agent - # is a hard EarlyStopConfigError at resolution — runs using `--type codex` - # or `--type antigravity` MUST add `-D run_limits.stop_early=false` (rows - # may then use up to max_turns: 3 — a cap, not a floor). Note: unarmed runs - # are the authoritative ones for precision — a pass-stop can hide a later + # Note: unarmed runs (`-D run_limits.stop_early=false`) are the + # authoritative ones for precision — a pass-stop can hide a later # distractor misfire, so armed-run precision is optimistic by comparison # (recall.yes is unaffected; see tasks/activation/README.md). max_turns: 3 stop_early: true # Scaled with max_turns: a row that legitimately uses all 3 turns (a true - # negative, or an unarmed codex/antigravity run) must have wall-clock room - # for them — a task_timeout below max_turns x turn_timeout silently drops - # the row from every criterion denominator instead of grading it. + # negative, or any unarmed run) must have wall-clock room for them — a + # task_timeout below max_turns x turn_timeout silently drops the row from + # every criterion denominator instead of grading it. task_timeout: 360 turn_timeout: 120 diff --git a/tests/scripts/activation_gate.py b/tests/scripts/activation_gate.py index c027b6121f..5ee2c4a281 100755 --- a/tests/scripts/activation_gate.py +++ b/tests/scripts/activation_gate.py @@ -62,7 +62,7 @@ def _build_task_yaml(skill: str, dataset: Path) -> str: # stop_when: auto is REQUIRED, not an optimization: the experiment's # defaults arm run_limits.stop_early, and an armed run with no stop # criterion is a hard EarlyStopConfigError at resolution (coder_eval >= - # 0.8.10). Gate rows are all positives, so auto arms pass-stop: the run + # 0.9.1). Gate rows are all positives, so auto arms pass-stop: the run # ends the moment {skill} engages, with the verdict a full run would # have produced (any-engagement latch is monotonic), and a recall miss # never fires a live event so it still runs to the cap. With a single diff --git a/tests/tasks/activation/README.md b/tests/tasks/activation/README.md index eeb2ded8e6..c2ca335b7d 100644 --- a/tests/tasks/activation/README.md +++ b/tests/tasks/activation/README.md @@ -36,7 +36,7 @@ uv run --project /path/to/coder_eval coder-eval run \ Reports land in `tmp//`. The suite gate fails per criterion on `recall.yes < 0.70`. Class imbalance (~50 yes / ~1260 no per skill) makes accuracy and recall.no trivially high; recall.yes is the only meaningful gate. -Early stop: the experiment arms `run_limits.stop_early` and every criterion sets `stop_when: auto` (coder_eval >= 0.8.10), so a row ends as soon as its outcome is live-decided. A positive row pass-stops the moment its expected skill engages; a negative row fail-stops on its first engagement. A wrong-skill engagement alone does not end a positive row — fail-stop is deferred while the row's positive criterion is undecided, so a positive that only misfires runs to `max_turns: 3`, as do no-engagement rows. Claude Code agent only: for `--type codex` or `--type antigravity` add `-D run_limits.stop_early=false`. For authoritative precision numbers (a pass-stop can hide a distractor misfire on a later tool call), run with `-D run_limits.stop_early=false`. +Early stop: the experiment arms `run_limits.stop_early` and every criterion sets `stop_when: auto` (coder_eval >= 0.9.1; below that, an armed `--type codex`/`antigravity` run fails at resolution with `EarlyStopConfigError`), so a row ends as soon as its outcome is live-decided. A positive row pass-stops the moment its expected skill engages; a negative row fail-stops on its first engagement. A wrong-skill engagement alone does not end a positive row — fail-stop is deferred while the row's positive criterion is undecided, so a positive that only misfires runs to `max_turns: 3`, as do no-engagement rows. For authoritative precision numbers (a pass-stop can hide a distractor misfire on a later tool call), run with `-D run_limits.stop_early=false`. ## Adding a new skill @@ -46,7 +46,7 @@ Early stop: the experiment arms `run_limits.stop_early` and every criterion sets ## Cost -On Sonnet 4.6 via Bedrock, ~$0.05–$0.10 per row. The dataset is ~1309 rows total (1252 positives + 57 negatives). The agent runs ONCE per row regardless of criteria count, and the 23 stacked criteria are pure-Python evaluation against the same trace, so the full benchmark costs **~$65–130**. Those numbers assume the default armed run, where decided rows end at ~1 turn; an unarmed run (`-D run_limits.stop_early=false` — required for codex/antigravity and for authoritative precision) lets every row use up to `max_turns: 3`, so budget roughly 2–3× for it. Use `--sample N` for cheaper iteration (note: `--sample` slices first-N, which biases toward the first-listed paths; useful for smoke runs, not for metrics). +On Sonnet 4.6 via Bedrock, ~$0.05–$0.10 per row. The dataset is ~1309 rows total (1252 positives + 57 negatives). The agent runs ONCE per row regardless of criteria count, and the 23 stacked criteria are pure-Python evaluation against the same trace, so the full benchmark costs **~$65–130**. Those numbers assume the default armed run, where decided rows end at ~1 turn; an unarmed run (`-D run_limits.stop_early=false` — optional for routine checks, required for authoritative precision) lets every row use up to `max_turns: 3`, so budget roughly 2–3× for it. Use `--sample N` for cheaper iteration (note: `--sample` slices first-N, which biases toward the first-listed paths; useful for smoke runs, not for metrics). ## Provenance