skills: one command per operation

.claude-plugin/plugin.json

@@ -1,7 +1,7 @@
 {
   "name": "fairtask",
   "version": "0.1.0",
-  "description": "Screen SWE-bench-style coding tasks for fairness before they become benchmark or RL tasks: is the issue specified well enough, and do the graded tests accept every reasonable fix? Two skills: /fairtask (screen a task) and /fairtask-eval (reproduce and inspect the evaluation) that drives an agent pipeline with verified evidence.",
+  "description": "Screen SWE-bench-style coding tasks for fairness before they become benchmark or RL tasks: is the issue specified well enough, and do the graded tests accept every reasonable fix? A command set: /fairtask (screen a task), /fairtask-baseline (one-prompt control), /fairtask-report, /fairtask-score, /fairtask-cases, /fairtask-trajectory (reproduce and inspect the evaluation) and /fairtask-eval (evidence audit, code check, provenance).",
   "author": {
     "name": "Manos Kaparos",
     "url": "https://github.com/mnkprs"

.github/workflows/ci.yml

@@ -41,5 +41,5 @@ jobs:
       - name: plugin manifests are semantically valid and describe one skill
         run: |
           npm run validate:manifests
-          npx --yes @anthropic-ai/claude-code@2.1.250 --plugin-dir . plugin details fairtask | grep -E "Skills \(2\)"
+          npx --yes @anthropic-ai/claude-code@2.1.250 --plugin-dir . plugin details fairtask | grep -E "Skills \(7\)"
           npx --yes skills add . --list 2>&1 | grep -q "fairtask"

README.md

@@ -256,7 +256,19 @@ the model's memory.
 | Claude Code, as a plugin | `/plugin marketplace add mnkprs/fairtask` then `/plugin install fairtask@fairtask` | both skills (plugin manifest in `.claude-plugin/`) |
 | Codex, as a plugin | the repository carries `.codex-plugin/plugin.json` | both skills |
 
-Both skills need Node ≥ 22.18 and git. `/fairtask` also needs a Claude login or `ANTHROPIC_API_KEY`, because it runs
+One install brings the whole command set:
+
+| Command | What it does | Model calls |
+|---|---|---|
+| `/fairtask <id \| task.json \| owner/repo PR#>` | Screen a task with the full pipeline; verdict with verified evidence. | yes (~one dollar) |
+| `/fairtask-baseline <id \| task.json>` | The same task through the one-prompt baseline, for comparison; its evidence is *not* machine-verified. | yes (~fifteen cents) |
+| `/fairtask-report` | The headline and all-systems tables from the committed results. | no |
+| `/fairtask-score [run ids…]` | Metrics of runs against the human labels. | no |
+| `/fairtask-cases [id]` | The 30-case evaluation set, or one instance laid out with its decisive lines quoted. | no |
+| `/fairtask-trajectory <id> [run]` | What the agent did on an instance, tool call by tool call. | no |
+| `/fairtask-eval …` | The supporting checks: evidence audit, zero-LLM code check, data provenance. | no |
+
+All need Node ≥ 22.18 and git. `/fairtask` also needs a Claude login or `ANTHROPIC_API_KEY`, because it runs
 the model; `/fairtask-eval` needs neither.
 
 ### `/fairtask` — screen a task
@@ -303,21 +315,21 @@ release tag** (`--branch v0.1.0`), never a moving branch, runs `npm ci`, and say
 its own read-only tools, following `skills/fairtask/references/method.md`, re-opens every location it cites to
 confirm the quote, and states in the report that the quotes were self-checked rather than machine-verified.
 
-### `/fairtask-eval` — reproduce and inspect the evaluation
+### The evaluation commands — reproduce and inspect every number
 
 Everything a judge or reviewer might want to see, run from inside the session with the output printed verbatim.
-No model calls; offline except the two pinned data downloads; it refuses to start paid reproduction runs and points
+No model calls; offline except the two pinned data downloads; they refuse to start paid reproduction runs and point
 at `REPRODUCE.md` instead.
 
 | You type | What it runs, and shows |
 |---|---|
-| `/fairtask-eval show the evaluation set` | `npm run data:eval-set` — the 30-row table (instance, stratum, human scores, difficulty, sizes) and the per-stratum and per-repository counts. |
-| `/fairtask-eval score baseline v3-verify` | `npm run score -- baseline v3-verify` — decision accuracy, κ, TPR/TNR, missed/false alarms, per-axis agreement, cost and time, side by side. |
-| `/fairtask-eval the headline report, baseline versus final` | `node src/report.ts …` — the two tables in §4, as aligned terminal tables. |
+| `/fairtask-cases` | `npm run data:eval-set` — the 30-row table (instance, stratum, human scores, difficulty, sizes) and the per-stratum and per-repository counts. |
+| `/fairtask-score baseline v3-verify` | `npm run score -- baseline v3-verify` — decision accuracy, κ, TPR/TNR, missed/false alarms, per-axis agreement, cost and time, side by side. |
+| `/fairtask-report` | `node src/report.ts …` — the two tables in §4, as aligned terminal tables. |
 | `/fairtask-eval audit evidence for v3-verify` | `npm run audit -- v3-verify` — share of cited quotes that do not exist where cited (needs `npm run data:workspaces`, which it tells you to run). |
 | `/fairtask-eval check the annotation provenance` | `npm run data:annotations -- --check` — SHA-256 of the committed file against the pinned source. |
-| `/fairtask-eval lay out astropy__astropy-12544` | `npm run show -- astropy__astropy-12544` — writes `examples/<id>/`, then shows the human labels and quotes, with `file:line`, what the issue asks for and what the graded tests require. |
-| `/fairtask-eval what did the agent do on astropy__astropy-12544 in v3-verify` | renders and opens that run's trajectory. |
+| `/fairtask-cases astropy__astropy-12544` | `npm run show -- astropy__astropy-12544` — writes `examples/<id>/`, then shows the human labels and quotes, with `file:line`, what the issue asks for and what the graded tests require. |
+| `/fairtask-trajectory astropy__astropy-12544 v3-verify` | renders and opens that run's trajectory. |
 
 ### What the skills may and may not do
 

scripts/validate-manifests.mjs

@@ -19,7 +19,7 @@ if (codex.version !== claude.version) fail("codex plugin.json: version differs f
 if (typeof codex.skills !== "string" || !existsSync(codex.skills)) fail("codex plugin.json: skills must point at an existing directory");
 const caps = codex.interface?.capabilities ?? [];
 for (const c of ["Read", "Write", "Network", "Execute"]) if (!caps.includes(c)) fail(`codex plugin.json: capabilities must disclose ${c} (the skill clones, installs and runs the engine)`);
-for (const name of ["fairtask", "fairtask-eval"]) {
+for (const name of ["fairtask", "fairtask-baseline", "fairtask-report", "fairtask-score", "fairtask-cases", "fairtask-trajectory", "fairtask-eval"]) {
   const text = readFileSync(`skills/${name}/SKILL.md`, "utf8");
   const front = text.match(/^---\n([\s\S]*?)\n---/)?.[1] ?? "";
   if (!new RegExp(`^name: ${name}$`, "m").test(front)) fail(`skills/${name}/SKILL.md: frontmatter name must be ${name}`);

skills/fairtask-baseline/SKILL.md

@@ -0,0 +1,35 @@
+---
+name: fairtask-baseline
+description: Screen one SWE-bench-style task with fairtask's one-prompt baseline — the same rubric and material as the full pipeline but no repository access and no evidence verification — so its verdict can be compared with /fairtask on the same task. Use when asked to run the baseline on a task, to compare the baseline against the pipeline, or to show what screening looks like without an agent. Costs about fifteen cents and forty seconds per task; needs a Claude login or API key.
+license: MIT
+metadata:
+  origin: fairtask
+  repository: https://github.com/mnkprs/fairtask
+---
+
+# fairtask-baseline
+
+Screen one candidate task with the **one-prompt baseline**: the same rubric, issue, gold patch and test patch the
+full pipeline sees, but a single model call with no tools, no repository access, and no deterministic verification.
+Its purpose is comparison — run it next to `/fairtask` on the same task to show what the agent pipeline adds.
+
+## Steps
+
+1. **Identify the input**: a SWE-bench-style instance id (`--swebench <id> [--dataset …]`) or a task JSON
+   (`--task <file>`), exactly as for `/fairtask`.
+2. **Locate the engine** the same way `/fairtask` does: `$FAIRTASK_HOME` if set, else `~/.fairtask`, else the current
+   directory if its `package.json` is named `fairtask`; if absent, clone
+   `https://github.com/mnkprs/fairtask` at `--branch v0.1.0` into `~/.fairtask` and run `npm ci`.
+3. **Run**: `npm run screen -- <input flags> --variant baseline`. Expect about forty seconds and fifteen cents.
+4. **Report** the verdict in the same order as `/fairtask` (decision, both axes with reasons, every evidence item),
+   and state plainly that the trailer says **evidence NOT verified**: the baseline's quotes are whatever the model
+   wrote — on the evaluation set eight to eleven percent of them do not exist where cited. That is the point of
+   the comparison, not a malfunction.
+5. If the same task was screened with `/fairtask`, put the two verdicts side by side: decision, per-axis scores,
+   whether the evidence verified, cost.
+
+## Gotchas
+
+- Do not use a baseline verdict to accept or reject a task; it exists to be compared against.
+- On the evaluation set the baseline agrees with expert humans 67% of the time — the same headline rate as the
+  pipeline — so differences show up in the evidence and the per-axis scores, not usually in the decision.

skills/fairtask-cases/SKILL.md

@@ -0,0 +1,28 @@
+---
+name: fairtask-cases
+description: Show fairtask's 30-case evaluation set (instances, strata, human scores) or lay out one instance — its issue text, test patch, gold patch and human labels — quoting the decisive lines with file and line numbers. Use when asked which cases the evaluation uses, to show the eval set, or to lay out / inspect a specific instance id. Offline except one pinned public download; no model calls.
+license: MIT
+metadata:
+  origin: fairtask
+  repository: https://github.com/mnkprs/fairtask
+---
+
+# fairtask-cases
+
+The evaluation set, or one case laid out so a reader can see the discrepancy without opening files.
+
+## Steps
+
+1. **Locate the repository**: current directory if its `package.json` is named `fairtask`; else `$FAIRTASK_HOME`;
+   else `~/.fairtask`. If none exists, say so and stop.
+2. **No instance id given** — show the set: `npm run data:eval-set` (needs `data/raw/swebench_test.parquet`; if
+   missing, first run `curl --fail -L -o data/raw/swebench_test.parquet https://huggingface.co/datasets/princeton-nlp/SWE-bench/resolve/e48e2bd1e9fecd5bbd641e9414ac59da9f2e69f6/data/test-00000-of-00001.parquet`
+   and say you did). Show the 30-row table verbatim and name the four strata with their counts. In a clean checkout
+   `git status` stays clean afterwards — if not, say so; that is a reproducibility failure.
+3. **Instance id given** — lay it out: `npm run show -- <instance_id>`, then open
+   `examples/<instance_id>/human-labels.md`, `issue.md` and `test.patch` yourself — do not paste the files. Show the
+   one `wrote …` line, the human labels as one line (`underspecified=N, false_negative=N, difficulty, filter_out`),
+   then the lines that decide the case, quoted exactly with `file:line` — what the issue asks for, and what the
+   graded tests require that the issue does not say (the `+` lines of the FAIL_TO_PASS tests). Two to four quotes,
+   no more; point to `examples/<instance_id>/` for the full text. Done when a reader can see the discrepancy (or its
+   absence) without opening the files.

skills/fairtask-eval/SKILL.md

@@ -1,6 +1,6 @@
 ---
 name: fairtask-eval
-description: Reproduce, inspect or explain the fairtask evaluation from inside an agent session — build the 30-case evaluation set, score runs against the human labels, regenerate the comparison report, audit cited evidence, run the zero-LLM pre-check, verify the data provenance, or lay out one evaluation instance as readable files. Use when asked to show, rerun, check or explain the evaluation numbers, the eval set, a run's results, or a specific evaluation instance of the fairtask project. Read-only and offline except for the data fetch; makes no model calls.
+description: Inspect the fairtask evaluation's supporting checks from inside an agent session — audit cited evidence against the repositories, run the zero-LLM code pre-check, or verify the annotation data's provenance and checksum — and route anything else to the dedicated commands (/fairtask-report, /fairtask-score, /fairtask-cases, /fairtask-trajectory). Use when asked to audit evidence, run the code check, verify data provenance, or when unsure which fairtask command fits. Read-only and offline; makes no model calls.
 license: MIT
 metadata:
   origin: fairtask
@@ -10,33 +10,28 @@ metadata:
 # fairtask-eval
 
 Run the evaluation tooling of the `fairtask` repository and bring its output into the conversation, so the numbers a
-reader sees come from the scripts, never from memory. Every operation is offline (the scorer reads committed results)
-except `data` and `annotations`, which download pinned public files. None calls a model.
+reader sees come from the scripts, never from memory. The dedicated commands cover the common asks; this skill keeps
+the supporting checks (evidence audit, code pre-check, provenance) and routes everything else. All offline; none
+calls a model.
 
 ## Steps
 
 1. **Locate the repository.** The current directory if its `package.json` is named `fairtask`; else `FAIRTASK_HOME`; else
    `~/.fairtask`. If none exists, say so and stop — this skill does not clone.
 2. **Pick the operation** from the request (one per invocation; ask if two are equally plausible):
 
-   | Request mentions | Run from the repository root |
+   | Request mentions | Do |
    |---|---|
-   | the evaluation set, the thirty cases, strata, which tasks | `npm run data:eval-set` (needs `data/raw/swebench_test.parquet`; if missing, run `curl -L -o data/raw/swebench_test.parquet https://huggingface.co/datasets/princeton-nlp/SWE-bench/resolve/e48e2bd1e9fecd5bbd641e9414ac59da9f2e69f6/data/test-00000-of-00001.parquet` first and say you did) |
-   | score, accuracy, kappa, recall, a run id, compare runs | `npm run score -- <run ids…>` (default: `baseline v3-verify`; add `--detail` for per-instance rows) |
-   | the report, the headline table, baseline versus final | `node src/report.ts --baseline baseline --final v3-verify --runs <all run ids>` — the run ids are the directory names under `results/`; the default (`v3-verify`) has no repeat run, so no `--final-repeat` (the script refuses a repeat of a different configuration). Needs cloned workspaces for the bad-evidence rows — run `npm run data:workspaces` (~20 s) first if `workspaces/` is missing. Prints aligned tables; `--markdown` gives the README's pipe tables |
    | evidence audit, bad evidence, fabricated quotes | `npm run audit -- <run ids…>` (needs cloned workspaces: `npm run data:workspaces` first, ~20 s) |
    | code check, novel identifiers, pre-check | `npm run code-check` (needs workspaces) |
    | provenance, annotations, checksum, data source | `npm run data:annotations -- --check` |
-   | show an instance, lay out, issue text, test patch of `<id>` | `npm run show -- <instance_id>`, then open `examples/<instance_id>/human-labels.md`, `issue.md` and `test.patch` yourself and present them as described in step 3 — do not paste the files |
-   | trajectory, what the agent did on `<id>` in run `<run>` | `npm run trajectory -- trajectories/<run>/<instance_id>.jsonl` and open the rendered `.md` |
+   | the report, headline table, baseline versus final | use `/fairtask-report` |
+   | score, accuracy, kappa, a run id, compare runs | use `/fairtask-score` |
+   | the evaluation set, the thirty cases, lay out an instance | use `/fairtask-cases` |
+   | trajectory, what the agent did | use `/fairtask-trajectory` |
+   | screen a task | use `/fairtask` (full pipeline) or `/fairtask-baseline` (one-prompt control) |
 
 3. **Show the output verbatim** in a code block. Do not round, reorder or summarise numbers before the block.
-   Exception — **lay out an instance**: the command only writes files. Show its one `wrote …` line, then the human
-   labels from `human-labels.md` as one line (`underspecified=N, false_negative=N, difficulty, filter_out`), then the
-   lines that decide the case, quoted exactly with `file:line` — what the issue asks for (from `issue.md`), and what
-   the graded tests require that the issue does not say (from `test.patch`; the `+` lines of the FAIL_TO_PASS tests).
-   Two to four quotes, no more; point to `examples/<instance_id>/` for the full text. Done when a reader can see the
-   discrepancy (or its absence) without opening the files.
 4. **Add one or two sentences** on how to read it — which row is the primary metric (decision accuracy), that
    TPR/TNR are over scored cases only, that "bad evidence" is the share of cited quotes not found where cited. For the
    evaluation set, name the four strata and the count per stratum. Stop there; the reader asked for the artifact, not

skills/fairtask-report/SKILL.md

@@ -0,0 +1,31 @@
+---
+name: fairtask-report
+description: Print fairtask's headline comparison (one-prompt baseline vs the final agent pipeline) and the all-systems table over the same 30 human-annotated development-set cases, generated by the repository's own report script from the committed results. Use when asked for the results, the headline numbers, the comparison table, or baseline versus final. Offline, no model calls; clones nothing.
+license: MIT
+metadata:
+  origin: fairtask
+  repository: https://github.com/mnkprs/fairtask
+---
+
+# fairtask-report
+
+The two results tables of the fairtask report — headline comparison and all systems — printed by the repository's
+report script from `results/`, never from memory.
+
+## Steps
+
+1. **Locate the repository**: current directory if its `package.json` is named `fairtask`; else `$FAIRTASK_HOME`;
+   else `~/.fairtask`. If none exists, say so and stop.
+2. **Preflight workspaces**: the bad-evidence rows re-verify every quote against the repositories, so if
+   `workspaces/` is missing run `npm run data:workspaces` first (about twenty seconds, shallow-clones 30 repos)
+   and say you did.
+3. **Run**: `node src/report.ts --baseline baseline --final v3-verify --runs baseline,baseline-rerun,v1-context,v2-specialists,v3-verify,v4-calibrated,v5-cheap-probes,v5-rerun,v6-target-aware,v7-sonnet-nocal`
+4. **Show both tables verbatim** in a code block, then two sentences: decision accuracy is the primary metric and
+   ties at 67% (development set); what changes is flag recall (70% → 75–85%) and cited evidence that fails
+   verification (8–11% → 0%).
+
+## Gotchas
+
+- The default final (`v3-verify`) has no repeat run: no `--final-repeat`. The script refuses a repeat of a
+  different configuration; do not work around it.
+- `--markdown` prints the README's pipe-table form — only useful when regenerating the README.

skills/fairtask-score/SKILL.md

@@ -0,0 +1,23 @@
+---
+name: fairtask-score
+description: Score one or more fairtask runs against the human SWE-bench Verified labels and show the metrics side by side — decision accuracy, kappa, flag precision/recall, TPR/TNR, per-axis agreement, cost and time. Use when asked to score a run, compare runs, or check accuracy, recall or kappa of the fairtask evaluation. Offline, reads the committed results; no model calls.
+license: MIT
+metadata:
+  origin: fairtask
+  repository: https://github.com/mnkprs/fairtask
+---
+
+# fairtask-score
+
+Metrics of fairtask runs against the human labels, straight from the scorer.
+
+## Steps
+
+1. **Locate the repository**: current directory if its `package.json` is named `fairtask`; else `$FAIRTASK_HOME`;
+   else `~/.fairtask`. If none exists, say so and stop.
+2. **Run**: `npm run score -- <run ids…>` — default `baseline v3-verify`; any directory names under `results/`
+   (`baseline`, `baseline-rerun`, `v1-context`, `v2-specialists`, `v3-verify`, `v4-calibrated`, `v5-cheap-probes`,
+   `v5-rerun`, `v6-target-aware`, `v7-sonnet-nocal`). Add `--detail` when asked for per-instance rows.
+3. **Show the output verbatim** in a code block; do not round or reorder numbers.
+4. Add one or two sentences: PRIMARY decision accuracy is the headline metric; TPR/TNR are over scored cases only;
+   these are development-set numbers.