Update pipeline and add diagnostic commands for SCF parser

.claude/skills/scf-regenerate/SKILL.md

@@ -0,0 +1,31 @@
+---
+name: scf-regenerate
+description: Regenerate data/, docs/ and the validation report for a new SCF version once the parser reads it, add tests/ground truth, update the changelog, and run the full verification. Use after scf-release-fix reaches PARSED with 0 fails.
+---
+
+# Regenerate and verify (no commits)
+
+1. **Tests first.** Extend `tests/conftest.py` `build_workbook` with the new shape (a
+   `layout-<version>` flag like `layout-2026-3`; keep older layouts intact) and add tests in
+   `tests/test_parse.py` proving the same synthetic content parses identically across layouts.
+   Add the new `GROUND_TRUTH` row in `tests/test_integration_real.py` from `diagnose` counts
+   (controls, families, frameworks, aos, erls, compensating, privacy, threats, risks).
+2. **Run the suite** (real workbook included):
+   `SCF_WORKBOOK=workbooks/<file>.xlsx nix develop -c python -m pytest -q`
+3. **Regenerate:**
+   `nix develop -c python -m scf_api.cli all --workbook workbooks/<file>.xlsx`
+   This writes `data/`, prunes the old `scf-<old>.json`, rebuilds `docs/`, rewrites
+   `.scf-version` and `validation/validation-report.json`. Expect exit 0 and `fail: 0`.
+4. **Make new files visible to nix:** `git add -N data docs` (flake checks only see tracked files).
+5. **Changelog:** build the old dataset into a scratch dir
+   (`mkdir -p $SCRATCH/old; for f in $(git ls-tree --name-only HEAD data/ | grep json); do git show HEAD:$f > $SCRATCH/old/$(basename $f); done`),
+   run `changelog $SCRATCH/old data`, and put under a new top section of `CHANGELOG.md`,
+   with a short hand-written summary of what upstream changed and how we absorbed it.
+6. **Docs/README:** if output fields were added, mention them in the CHANGELOG; contract
+   additions must be optional in `src/scf_api/schemas/*.schema.json`.
+7. **Verify:** `nix fmt && nix flake check -L` must end with `all checks passed!`.
+8. If an override became unused (upstream fixed a typo), scope it with `applies_to` to the
+   older versions instead of deleting it.
+
+Report to the human: version, counts, any new warnings, anything that needed a judgement
+call (especially mapping semantics). Do not commit.

.claude/skills/scf-release-fix/SKILL.md

@@ -0,0 +1,58 @@
+---
+name: scf-release-fix
+description: Repair the SCF parser after a new upstream workbook release breaks the pipeline (validation gate failed, "no column matching", "no sheet matching", thousands of findings). Use when a new SCF version arrives or an update-scf issue is opened.
+---
+
+# Fix the parser for a new SCF release
+
+Goal: one parser reads every supported release (2026.1.1, 2026.2, 2026.3, + the new one).
+Run everything as `nix develop -c python -m scf_api.cli ...`. Workbook is in `workbooks/`
+(or `fetch` it). Keep output small; do not open `validation/validation-report.json` directly.
+
+## Step 1 — get the workbook and a first reading
+```
+fetch                                   # only if workbooks/ lacks the new file
+diagnose --workbook workbooks/<file>.xlsx
+```
+Output is one of:
+- `STRUCTURE: no tab matches kind 'K'` → go to **Fix A**
+- `STRUCTURE: tab 'T' has no column matching [...]` → go to **Fix B**
+- `PARSED ...` with fails → go to **Fix C**
+- `PARSED ...` with 0 fails → go to **Step 3**
+
+## Fix A — a tab was renamed or removed
+`src/scf_api/workbook.py` `SHEET_PATTERNS[kind] = (accepted patterns, excludes)`.
+- Renamed: put the new name FIRST in the tuple, keep the old names. Match the stable part
+  (patterns are substrings after `norm_key`: lowercase, dashes folded, spaces collapsed).
+- Removed because its data moved elsewhere (2026.3 did this for compensating controls):
+  keep the old pattern, use `wb.has_sheet(kind)` and read the data from the new tab. See
+  `_parse_compensating` in `parse.py` for the exact pattern.
+- Check the new tab doesn't also match another kind's pattern (`excludes`).
+
+## Fix B — a header was renamed or removed
+Find the failing `sheet.col("<old>")` in `src/scf_api/parse.py` (grep the pattern text).
+- Renamed: `sheet.col("<new>", alts=("<old>",))`. New first, old kept.
+- Use `headers --workbook W --sheet <tab> --grep <word>` to see real headers and a sample
+  value; confirm the sample looks like what the field means before mapping to it.
+- Removed (field no longer exists): ask whether another column now carries it
+  (`--grep`). If not, stop and hand back — dropping an output field is a contract change.
+
+## Fix C — parsed, but the gate has fails
+Run `triage`. Typical shapes:
+- Thousands of `control-risk`/`control-threat` fails with values like `Unlikely`: a matrix
+  column is being read as an id list. See `scf-workbook-recon`.
+- A handful of `control-erl` etc. with real-looking ids: probably a genuine workbook typo.
+  Confirm in the sheet cell (`sheet!cell` is in the sample), then add an entry to
+  `overrides/overrides.json` with `applies_to: ["<version>"]` and a `reason`.
+- `xwalk-source` warnings for 3 MCR/DSR columns are normal. Many more = the crosswalk
+  landmark slice moved (`parse.py` `_parse_controls`, `sources.py`).
+
+## Step 3 — prove it is right, not just green
+1. Compare counts printed by `diagnose` with the previous `GROUND_TRUTH` row in
+   `tests/test_integration_real.py`. Big drops mean a misread column.
+2. Spot-check 2 records: `jq` one control from a scratch `--out` dir (see
+   `scf-regenerate`), and compare fields to the sheet row via `headers --sample`.
+3. Then follow `scf-regenerate`.
+
+If stuck after 3 different fixes for the same miss, stop and write the handoff note
+described in `AGENTS.md`.

.claude/skills/scf-workbook-recon/SKILL.md

@@ -0,0 +1,47 @@
+---
+name: scf-workbook-recon
+description: Understand a changed SCF workbook layout cheaply (sheet/column shapes, how upstream has restructured it before) without loading the whole file into context. Use when diagnose/triage show misread columns or a layout change beyond a rename.
+---
+
+# Reading an SCF workbook on a small context budget
+
+Never print a sheet. The control tab has 400+ columns × 1500 rows.
+
+```
+headers --workbook W --sheet "evidence"                # all columns of one tab
+headers --workbook W --sheet "SCF 20" --grep risk --sample
+headers --workbook W --sheet "SCF 20" --grep "errata"
+```
+`--sheet` is a tab-name substring (normalized). `--grep` filters header text.
+`--sample` adds the first data row's cell beside each header: this is how you learn that a
+column holds ratings (`Unlikely`), ids (`R-AC-1`), or free text.
+
+## Tab kinds the parser needs (`SHEET_PATTERNS`)
+domains · sources (focal documents) · controls · compensating (legacy tab) · evidence ·
+aos · privacy · risks · threats. Others (READ THIS, Change Overview, STRM Errata, Lists)
+are ignored.
+
+## Column landmarks on the controls tab
+`AICPA …` is the first crosswalk column. The crosswalk slice ends at the first risk
+column. Between the SCR-CMM columns and AICPA sit profile columns whose header starts
+with a line `SCF` (profiles: marked if non-empty). Order is: identity/description cols →
+ERL → possible solutions → question/weight/PPTDF/NIST/SCRM → (2026.3: compensating cols) →
+maturity 0-5 → profiles → crosswalks → risks/threats → errata.
+
+## How upstream has changed things (so you recognize the pattern)
+| release | change | how we absorbed it |
+|---|---|---|
+| 2026.2 | tab `Authoritative Sources` → `Focal Documents`; compensating `SCF Control #` → `SCF #` | ordered pattern tuples, `alts=` |
+| 2026.3 | compensating controls moved onto the control tab (tab deleted) | `has_sheet` + same reader on controls tab |
+| 2026.3 | risk/threat summary text columns → likelihood matrix (`Risk R-AC-1`, `Threat NT-1`, cells Unlikely/Possible/Likely) | `sources.matrix_columns`; `risks`/`threats` = Possible\|Likely ids, plus `*_likelihood` maps |
+| 2026.3 | `Documentation Artifact` → `ERL Artifact`; ODP column → `Notes`; catalog gained `Old Risk #` | `alts=`, name column by header not position |
+| 2026.3 | tab names embed the version (`SCF 2026.3`) | match the stable part only |
+
+## Misread-column checklist
+- Gate fails with "unknown id <word>": the word is a rating/flag, so you read a matrix or
+  flag column as ids. Find the real id source (header names containing the id) .
+- Count dropped to ~0: wrong tab matched, or the id column is now a different header.
+- A field is silently empty everywhere: the pattern matched an earlier look-alike header.
+  `headers --grep <pattern>` shows every column containing it; `sheet.col` takes the FIRST
+  match, so tighten the pattern or use `exclude=(...)`.
+- Positional assumptions are fragile (`id_col + 1`). Prefer finding a column by header.

.pi/skills

@@ -0,0 +1 @@
+../.claude/skills

.scf-version

@@ -1 +1 @@
-2026.2
+2026.3

AGENTS.md

@@ -0,0 +1,58 @@
+# AGENTS.md — fixing this repo when a new SCF release breaks it
+
+Read `CLAUDE.md` for the rules (clean-room, no commits unless asked, determinism). This file
+is the short operating guide for **small-context agents** (24GB local models, 64-128k
+context). Keep your context lean: never `cat` big files, never open the full validation
+report, never print a whole sheet. Use the bounded tools below.
+
+## What this repo does
+
+SCF Excel workbook → `src/scf_api/parse.py` → validation gate (`validate.py`) → `data/*.json`
+→ `build.py` → `docs/api/**`. Upstream renames tabs/headers and restructures columns almost
+every release. A release breaks us when the parser cannot find a tab/column, or reads the
+wrong column and the gate fails with thousands of nonsense findings.
+
+## Run commands like this
+
+```bash
+nix develop -c python -m scf_api.cli <command>     # dev shell has openpyxl/rapidfuzz/pytest
+```
+
+| command | use |
+|---|---|
+| `diagnose --workbook W` | parse+validate; on a miss prints the closest real tabs/headers and the fix location |
+| `triage` | validation report grouped by check (counts, 3 samples, hint). Use this, not the JSON |
+| `headers --workbook W --sheet SUBSTR [--grep TEXT] [--sample]` | one line per column; `--sample` shows first data row |
+| `all --workbook W` | parse → validate → write `data/` → build `docs/` → bump `.scf-version` |
+| `fetch` | download the latest upstream workbook into `workbooks/` (needs `gh`) |
+
+## The loop (details in `.claude/skills/scf-release-fix/SKILL.md`)
+
+1. `diagnose` → fix the ONE thing it names → `diagnose` again. Repeat until it says PARSED.
+2. If the gate passed but counts look wrong, or the gate fails on thousands of rows, a column
+   was misread: use `headers --sample` and read `.claude/skills/scf-workbook-recon/SKILL.md`.
+3. Add the new shape to the synthetic fixture (`tests/conftest.py`) + a test, and a
+   `GROUND_TRUTH` row in `tests/test_integration_real.py`.
+4. Regenerate and verify (`.claude/skills/scf-regenerate/SKILL.md`): `nix fmt && nix flake check -L`.
+
+## Rules that save you from yourself
+
+- **Add, never replace.** New header/tab name goes FIRST, the old name stays (`alts=(...)`,
+  ordered tuples in `SHEET_PATTERNS`). One parser must read every supported release.
+- **Never hard-code a version** (`"2026.3"`, `"scf 2026.3"`) in a pattern. Tab names embed
+  the version; match the stable part.
+- **Do not weaken the gate to go green**: no deleting tests, no loosening validators, no
+  overrides for anything except a genuine upstream workbook typo (with a written reason).
+- **Do not hand-edit `data/` or `docs/`**; regenerate them.
+- Output contract change (new/removed/renamed JSON fields) = stop and ask the human.
+- No commits, no `git push`. `git add -N data docs` is fine (nix only sees tracked files).
+
+## When to stop and hand back to the human
+
+- The gate reports a real cross-sheet inconsistency (ids that exist in no catalog, after you
+  have confirmed the right column is being read).
+- A fix needs a contract change, or you have tried 3 different fixes for the same miss.
+- Counts changed by >20% and `headers --sample` does not explain why.
+
+Write the handoff as: what broke (tab/column), what you tried, `triage` output, the next
+question for the human. Keep it under 30 lines.