Add cross-skill triggering corpus for evals

CHANGELOG.md

@@ -9,6 +9,8 @@ tag releases both in lockstep, so entries below are keyed by the engine version.
 
 ## [Unreleased]
 
+## [1.12.3] - 2026-09-15
+
 ### Fixed
 
 - **`explore profile` no longer offers composite keys that are artifacts of a
@@ -185,6 +187,33 @@ tag releases both in lockstep, so entries below are keyed by the engine version.
 
   Also available as `DexEngine.test_mutations(model)`.
 
+- **A cross-skill, externally authored triggering corpus for the Tier-2 eval
+  harness** ([#216]). Each skill's own `evals.json` `positive`/`negative` list
+  is written by whoever wrote the description it tests, at the same time,
+  and checked with every other skill disabled; neither weakness is visible
+  from inside that suite. `evals/corpus/ade_bench_triggering.json` sources 30
+  real analytics-engineering requests from
+  [dbt-labs/ade-bench](https://github.com/dbt-labs/ade-bench) (Apache-2.0),
+  hand-labeled with the skill each should fire (or `none`), and run with
+  every skill available at once, one live call per prompt rather than one
+  per prompt-per-skill.
+
+  `python -m evals --corpus evals/corpus/ade_bench_triggering.json` reports
+  per-skill precision and recall plus which cases missed, and always exits
+  0: it is a measurement against externally authored prompts, not a release
+  gate, since the initial pass rate is expected to be low and that is the
+  signal the corpus exists to produce.
+
+  `Classifier.classify` reports every skill marker a call finds, not a single
+  winner: a prompt where two skills both fire is real evidence of cross-skill
+  contamination, and picking one silently would hide exactly the failure
+  mode this corpus exists to catch. A per-call failure is caught by the
+  classifier itself and recorded per case, excluded from precision/recall;
+  a setup failure (the `claude` binary missing) is left to propagate
+  immediately instead, so it aborts the run once with the existing
+  prerequisite message and exit code 2, rather than being recorded as 30
+  separate case errors while the command still reports a clean exit.
+
 - **`maintain verify` reports a built relation whose grain is not unique**
   ([#229]). For every selected model, the intended grain is determined from a
   declared `unique` test, a declared composite `unique_combination_of_columns`
@@ -1812,23 +1841,6 @@ tag releases both in lockstep, so entries below are keyed by the engine version.
   contributions rather than only the final score, which is new design work
   the capping fix does not need.
 
-- **A cross-skill, externally authored triggering corpus for the Tier-2 eval
-  harness** ([#216]). Each skill's own `evals.json` `positive`/`negative` list
-  is written by whoever wrote the description it tests, at the same time,
-  and checked with every other skill disabled; neither weakness is visible
-  from inside that suite. `evals/corpus/ade_bench_triggering.json` sources 30
-  real analytics-engineering requests from
-  [dbt-labs/ade-bench](https://github.com/dbt-labs/ade-bench) (Apache-2.0),
-  hand-labeled with the skill each should fire (or `none`), and run with
-  every skill available at once, one live call per prompt rather than one
-  per prompt-per-skill.
-
-  `python -m evals --corpus evals/corpus/ade_bench_triggering.json` reports
-  per-skill precision and recall plus which cases missed, and always exits
-  0: it is a measurement against externally authored prompts, not a release
-  gate, since the initial pass rate is expected to be low and that is the
-  signal the corpus exists to produce.
-
 - **`maintain schema` detects a model added, removed, or content-changed
   since the baseline** ([#164]). The transform layer's fingerprint (model
   names, per-file content hashes) was captured on every snapshot and

evals/README.md

@@ -46,13 +46,43 @@ a workspace with the dex plugin installed. The command exits non-zero unless the
 suite passes (clean triggering and no regression versus baseline), so the same
 invocation works locally and as a release gate.
 
+## The cross-skill triggering corpus
+
+The per-skill `positive`/`negative` lists above have a structural blind spot: they
+are written by the same person who wrote the description they test, at the same
+time, so a positive is often just a paraphrase of the description, and each skill
+is only ever checked with the *other* skills disabled. Neither failure mode is
+visible from inside one skill's own suite.
+
+`evals/corpus/` holds externally authored fixtures for exactly this: real prompts
+from someone with no knowledge of these descriptions, run with every skill
+available at once, each hand-labeled with the skill (or `none`) it should fire.
+`evals/corpus/ade_bench_triggering.json` is the first one, sourced from
+[dbt-labs/ade-bench](https://github.com/dbt-labs/ade-bench) (Apache-2.0); see the
+file's own `source` field for the exact provenance and what was and was not
+carried over.
+
+```
+python -m evals --corpus evals/corpus/ade_bench_triggering.json
+python -m evals --corpus evals/corpus/ade_bench_triggering.json --json
+```
+
+One live call per prompt (not per prompt-per-skill, since every skill is available
+in the same call), and the report is per-skill precision/recall plus which cases
+missed. This mode always exits 0: it is a measurement, not a release gate. Expect
+the first run's numbers to be low and record them as the baseline (there is no
+committed baseline yet; run it and note what you get in the PR, and treat later
+runs as measuring drift from that number rather than pass/fail).
+
 ## Layout
 
-- `suite.py` loads and validates a skill's `evals.json` (stdlib dataclasses).
-- `runner.py` is the deterministic scoring core: triggering, output quality, and
-  uplift. It takes an agent and a judge by dependency injection.
-- `claude_agent.py` is the live backend: the `AgentRunner` and `Judge` driven by
-  the `claude` CLI. A non-Claude agent is a second backend behind the same two
-  protocols, with no change to the core.
+- `suite.py` loads and validates a skill's `evals.json`, and a cross-skill
+  `Corpus` from `evals/corpus/*.json` (stdlib dataclasses both ways).
+- `runner.py` is the deterministic scoring core: triggering, output quality,
+  uplift, and the corpus's per-skill precision/recall. It takes an agent, a judge,
+  or a classifier by dependency injection.
+- `claude_agent.py` is the live backend: the `AgentRunner`/`Judge`/`Classifier`
+  protocols driven by the `claude` CLI. A non-Claude agent is a second backend
+  behind the same protocols, with no change to the core.
 - `__main__.py` is the CLI.
 - `tests/` covers the scoring core with fake backends (no model, free, in CI).

evals/__main__.py

@@ -1,13 +1,18 @@
-"""CLI: run a skill's Tier-2 eval suite.
+"""CLI: run a skill's Tier-2 eval suite, or the cross-skill triggering corpus.
 
     python -m evals skills/explore                 # full suite (live Claude)
     python -m evals skills/explore --triggering    # triggering only (cheaper)
     python -m evals skills/explore --json          # machine-readable report
+    python -m evals --corpus evals/corpus/ade_bench_triggering.json
+                                                     # cross-skill corpus (#216)
 
 The default backend drives Claude Code headless; it needs the ``claude`` CLI and
 a workspace with the dex plugin installed. Exit code is non-zero if the suite does
 not pass (clean triggering and no regression versus baseline), so the same command
-serves both local runs and the release gate.
+serves both local runs and the release gate. ``--corpus`` always exits 0: it is a
+measurement (per-skill precision/recall against externally authored prompts), not
+a pass/fail gate, per the issue that added it (#216) -- the initial numbers are
+expected to be low, and that is the signal, not a bug to chase away.
 """
 
 from __future__ import annotations
@@ -16,14 +21,21 @@
 import json
 import sys
 
-from .claude_agent import ClaudeCliAgent, ClaudeCliJudge, ClaudeNotAvailableError
-from .runner import run_suite, run_triggering
-from .suite import load_suite
+from .claude_agent import (
+    ClaudeCliAgent,
+    ClaudeCliClassifier,
+    ClaudeCliJudge,
+    ClaudeNotAvailableError,
+)
+from .runner import CorpusReport, run_corpus, run_suite, run_triggering
+from .suite import load_corpus, load_suite
 
 
 def main(argv: list[str] | None = None) -> int:
     parser = argparse.ArgumentParser(prog="evals", description="Tier-2 skill evals")
-    parser.add_argument("skill", help="skill dir or path to evals.json")
+    parser.add_argument(
+        "skill", nargs="?", default=None, help="skill dir or path to evals.json"
+    )
     parser.add_argument("--model", default=None, help="model override for the agent")
     parser.add_argument(
         "--triggering", action="store_true", help="run only the triggering check"
@@ -32,14 +44,35 @@ def main(argv: list[str] | None = None) -> int:
     parser.add_argument(
         "--timeout", type=int, default=180, help="per-agent-call timeout (seconds)"
     )
-    args = parser.parse_args(argv)
-
-    suite = load_suite(args.skill)
-    agent = ClaudeCliAgent(
-        skill_name=suite.skill_name, model=args.model, timeout=args.timeout
+    parser.add_argument(
+        "--corpus",
+        default=None,
+        help=(
+            "path to a cross-skill triggering corpus (see evals/corpus/); runs "
+            "independently of the skill argument, one live call per prompt"
+        ),
     )
+    args = parser.parse_args(argv)
 
     try:
+        if args.corpus:
+            corpus = load_corpus(args.corpus)
+            classifier = ClaudeCliClassifier(model=args.model, timeout=args.timeout)
+            report = run_corpus(corpus, classifier)
+            if args.json:
+                print(json.dumps(report.to_dict(), indent=2))
+            else:
+                _print_corpus(report)
+            return 0  # a measurement, never a gate; see the module docstring
+
+        if args.skill is None:
+            parser.error("skill is required unless --corpus is given")
+
+        suite = load_suite(args.skill)
+        agent = ClaudeCliAgent(
+            skill_name=suite.skill_name, model=args.model, timeout=args.timeout
+        )
+
         if args.triggering:
             trig = run_triggering(suite, agent)
             if args.json:
@@ -60,6 +93,31 @@ def main(argv: list[str] | None = None) -> int:
         return 2
 
 
+def _print_corpus(report: CorpusReport) -> None:
+    print(
+        f"[{report.corpus_name}] {len(report.results)} cases, "
+        f"accuracy {report.accuracy:.0%}"
+    )
+    for skill in sorted(report.per_skill):
+        pr = report.per_skill[skill]
+        print(
+            f"  {skill:10s} precision {pr.precision:.0%}  recall {pr.recall:.0%}  "
+            f"f1 {pr.f1:.0%}  (tp={pr.true_positives} fp={pr.false_positives} "
+            f"fn={pr.false_negatives})"
+        )
+    errors = [r for r in report.results if r.error is not None]
+    misses = [r for r in report.results if not r.correct and r.error is None]
+    if misses:
+        print(f"  {len(misses)} miss(es):")
+        for r in misses:
+            fired = sorted(r.fired_skills) or ["none"]
+            print(f"    {r.task_id}: expected {r.expected_skill!r}, got {fired}")
+    if errors:
+        print(f"  {len(errors)} call(s) failed (excluded from precision/recall):")
+        for r in errors:
+            print(f"    {r.task_id}: {r.error}")
+
+
 def _print_triggering(skill: str, report) -> None:
     fired = f"{report.positives_fired}/{report.positives_total} positives fired"
     print(f"[{skill}] triggering")

evals/claude_agent.py

@@ -20,7 +20,7 @@
 import subprocess
 from dataclasses import dataclass, field
 
-from .runner import AgentResult
+from .runner import AgentResult, ClassifyResult
 from .suite import EvalCase
 
 
@@ -93,6 +93,45 @@ def _plugin(self) -> str:
         return "dex"
 
 
+@dataclass
+class ClaudeCliClassifier:
+    """Runs one prompt with every dex skill available and reports every
+    marker that showed up, for the cross-skill corpus (``run_corpus``).
+
+    Unlike :class:`ClaudeCliAgent`, this never suppresses a skill: ``args``
+    is whatever invokes Claude Code with the plugin installed normally, the
+    same condition a real request arrives in. Reports *every* marker found,
+    not just one: picking a single "winner" when two fire would hide the
+    unwanted one entirely, which is exactly the cross-skill contamination
+    this corpus exists to catch, not a detail to summarize away.
+    """
+
+    skill_names: tuple[str, ...] = ("explore", "transform", "maintain")
+    binary: str = "claude"
+    model: str | None = None
+    timeout: int = 180
+    args: list[str] = field(default_factory=list)
+    plugin: str = "dex"
+
+    def classify(self, prompt: str) -> ClassifyResult:
+        # _require_claude runs outside the try, same as ClaudeCliAgent.run:
+        # a missing binary is a setup problem every subsequent call would hit
+        # identically, so it raises here and propagates out of run_corpus
+        # rather than being caught into one case's error.
+        binary = _require_claude(self.binary)
+        args = list(self.args)
+        if self.model:
+            args += ["--model", self.model]
+        try:
+            output = _invoke(binary, args, prompt, self.timeout)
+        except Exception as exc:
+            return ClassifyResult(error=str(exc))
+        fired = frozenset(
+            name for name in self.skill_names if f"/{self.plugin}:{name}" in output
+        )
+        return ClassifyResult(fired_skills=fired)
+
+
 @dataclass
 class ClaudeCliJudge:
     """Grades one assertion against an agent result with an LLM yes/no judge."""