From fcbabf4c16a53c8cd3af1c0451720b04f8c19d35 Mon Sep 17 00:00:00 2001 From: wshallwshall Date: Thu, 30 Jul 2026 14:18:41 -0500 Subject: [PATCH 1/4] quality: re-runnable Steps-view coverage scan (BACKLOG #239) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ADR 0089 §5 promised the coverage scan was repeatable, and then nobody re-ran it after Phase A, the ADR 0106 palette, ADR 0108 fan-out and ADR 0104 picker shipped. This makes re-running it a one-liner. It drives the SHIPPED `lens parse --json` rather than re-implementing the grammar with a second ast walk, so the number describes what the Steps view actually renders and cannot drift from it. PHI-safe by construction: `lens parse` never imports or executes a config module and no message enters the path, so it runs against a production estate unmodified. --hash-names for when results leave the building. First result, samples/config (14 files, 12 handlers, 28 rows, 0 refusals): code 42.9% OPAQUE 57.1% (46.4% --strict-control) send 42.9% EDITABLE 42.9% control 10.7% control (unrecognized) 3.6% handlers with zero editable rows : 0/12 handlers 100% typed : 5/12 (41.7%) median code rows per handler : 1 (max 4) This does NOT answer #239, and the reason is worth recording: samples/config contains ZERO action rows and ZERO lookup rows. Every editable row in it is a `send`. ADR 0089 §4 puts the pre-Phase-A baseline at "~13% (sends only)" -- so this corpus sits at exactly that baseline shape and cannot demonstrate Phase A's lift at all. The demo handlers delegate their transform work to helper modules (_demo_oru_transforms.py, _pdf_mdm_transforms.py) which the lens does not descend into, so it renders as `code`. That is ADR 0089's Phase D helper-descent gap, reproduced in miniature. 28 rows also cannot speak for an estate ADR 0089 measured at 3,852 statements. The decision-grade number needs the production estate; its path is owner-confirmed per #239 and deliberately not recorded here. Co-Authored-By: Claude Opus 5 --- scripts/quality/lens_coverage.py | 187 +++++++++++++++++++++++++++++++ 1 file changed, 187 insertions(+) create mode 100644 scripts/quality/lens_coverage.py diff --git a/scripts/quality/lens_coverage.py b/scripts/quality/lens_coverage.py new file mode 100644 index 00000000..c71032e2 --- /dev/null +++ b/scripts/quality/lens_coverage.py @@ -0,0 +1,187 @@ +"""Measure what the Steps view actually projects over a config estate (BACKLOG #239). + +WHY THIS EXISTS. [ADR 0089](../../docs/adr/0089-recognition-first-lens-native-idioms.md) §1 +measured a real estate -- 87 files, 486 `msg`-manipulating functions, 3,852 statements -- and found +**~66% of projected rows opaque** with **100% of handlers rendering zero editable action rows**; +Phase A moved that to ~42% editable. §5 states the scan is "a **repeatable** coverage check -- +re-running it after each phase measures the coverage lift and surfaces the shrinking residual". +Phase A, the ADR 0106 palette, ADR 0108's fan-out and ADR 0104's picker have all shipped since, and +nobody has re-run it. Whether to build the next Steps-view increment, or to stop, turns on that +number. + +WHAT IT MEASURES, AND WHY NOT WITH ITS OWN PARSER. ADR 0089's original scan classified statements +with its own `ast` walk. This one drives the SHIPPED `messagefoundry lens parse --json` instead, so +the answer describes what the Steps view REALLY renders today rather than a second implementation +of the grammar that could drift from it. The lens is the product surface under evaluation; asking +it directly is the only way the number stays honest as the grammar moves. + + opaque = `code` rows + `control` rows (ADR 0089 §1's "opaque code/UNRECOGNIZED control"). + A recognized `control` row is still counted opaque: `stepsView.ts` renders control + rows read-only ("`code`/`control` rows stay read-only ... visibly disabled"), so it is + opaque to EDITING even where the grammar recognized it. `--strict-control` narrows + this to unrecognized control rows only, which is the looser reading. + editable = action / lookup / send -- the kinds that expose enabled param inputs. + +PHI. `lens parse` is a static `ast` parse that never imports or executes a config module, and no +message ever enters this path. The scan reads only code and emits only counts and file names, so it +is safe to run against a production estate. Pass `--hash-names` when sending results anywhere. +""" + +from __future__ import annotations + +import argparse +import hashlib +import json +import statistics +import subprocess +import sys +from collections import Counter +from pathlib import Path + +EDITABLE = frozenset({"action", "lookup", "send"}) + + +def parse_module(py: Path, python: str, cwd: Path) -> tuple[dict | None, str]: + """Return (parsed JSON, error). `encoding=` is REQUIRED, not cosmetic. + + `text=True` alone decodes with the LOCALE default, which is cp1252 on a stock Windows box; + config modules are UTF-8, so the decode raises inside subprocess's reader thread and stdout + comes back None -- a refusal that reads as "this estate has no handlers". + """ + proc = subprocess.run( # nosec B603 - fixed argv, no shell, interpreter is operator-supplied + [python, "-m", "messagefoundry", "lens", "parse", str(py), "--json"], + capture_output=True, + text=True, + encoding="utf-8", + errors="replace", + cwd=cwd, + ) + if proc.returncode != 0: + tail = (proc.stderr or "").strip().splitlines() + return None, (tail[-1] if tail else f"exit {proc.returncode}") + try: + return json.loads(proc.stdout), "" + except json.JSONDecodeError as exc: + return None, f"non-JSON output ({exc})" + + +def _label(py: Path, root: Path, hash_names: bool) -> str: + rel = py.relative_to(root).as_posix() + if not hash_names: + return rel + return f"<{hashlib.sha256(rel.encode()).hexdigest()[:12]}>.py" + + +def main(argv: list[str] | None = None) -> int: + ap = argparse.ArgumentParser(description=__doc__.splitlines()[0]) + ap.add_argument("config_dir", type=Path, help="directory of config modules to scan") + ap.add_argument("--python", default=sys.executable, help="interpreter with messagefoundry") + ap.add_argument("--cwd", type=Path, default=Path.cwd(), help="working dir for the CLI call") + ap.add_argument( + "--strict-control", + action="store_true", + help="count only UNRECOGNIZED control rows as opaque (recognized control is not opaque)", + ) + ap.add_argument("--hash-names", action="store_true", help="hash file names in the output") + ap.add_argument("--json", dest="as_json", action="store_true", help="emit JSON") + args = ap.parse_args(argv) + + if not args.config_dir.is_dir(): + print(f"not a directory: {args.config_dir}", file=sys.stderr) + return 2 + + kinds: Counter[str] = Counter() + files = handlers = zero_editable = fully_typed = 0 + code_per_handler: list[int] = [] + refusals: list[dict[str, str]] = [] + + for py in sorted(args.config_dir.rglob("*.py")): + files += 1 + out, err = parse_module(py, args.python, args.cwd) + if out is None: + refusals.append({"file": _label(py, args.config_dir, args.hash_names), "reason": err}) + continue + for handler in out.get("handlers", []): + handlers += 1 + rows = handler.get("rows", []) + n_edit = n_opaque = n_code = 0 + for row in rows: + kind = row.get("kind", "?") + if kind == "control" and not row.get("recognized", True): + kind = "control (unrecognized)" + kinds[kind] += 1 + if kind in EDITABLE: + n_edit += 1 + continue + if kind == "code": + n_code += 1 + # A recognized control row counts as opaque unless --strict-control. + if kind != "control" or not args.strict_control: + n_opaque += 1 + if n_edit == 0: + zero_editable += 1 + if rows and n_opaque == 0: + fully_typed += 1 + code_per_handler.append(n_code) + + total = sum(kinds.values()) + opaque = kinds["code"] + kinds["control (unrecognized)"] + if not args.strict_control: + opaque += kinds["control"] + editable = sum(count for kind, count in kinds.items() if kind in EDITABLE) + + result = { + "corpus": str(args.config_dir), + "files_scanned": files, + "parse_refused": len(refusals), + "handlers": handlers, + "rows": total, + "row_kinds": dict(kinds.most_common()), + "opaque_rows": opaque, + "editable_rows": editable, + "opaque_pct": round(100.0 * opaque / total, 1) if total else None, + "editable_pct": round(100.0 * editable / total, 1) if total else None, + "handlers_zero_editable": zero_editable, + "handlers_fully_typed": fully_typed, + "median_code_rows_per_handler": statistics.median(code_per_handler) + if code_per_handler + else None, + "max_code_rows_in_one_handler": max(code_per_handler, default=None), + "strict_control": args.strict_control, + "refusals": refusals, + } + + if args.as_json: + print(json.dumps(result, indent=2)) + return 0 + + def pct(n: int, d: int) -> str: + return f"{(100.0 * n / d):.1f}%" if d else "n/a" + + print(f"corpus : {args.config_dir}") + print(f"files scanned : {files} (parse-refused: {len(refusals)})") + print(f"handlers projected : {handlers}") + print(f"rows projected : {total}") + print("\nrow kinds:") + for kind, count in kinds.most_common(): + print(f" {kind:<24} {count:>6} {pct(count, total)}") + print(f"\nOPAQUE rows : {opaque:>6} {pct(opaque, total)}") + print(f"EDITABLE rows : {editable:>6} {pct(editable, total)}") + print( + f"\nhandlers with ZERO editable rows : {zero_editable}/{handlers} {pct(zero_editable, handlers)}" + ) + print( + f"handlers 100% typed (no opaque) : {fully_typed}/{handlers} {pct(fully_typed, handlers)}" + ) + if code_per_handler: + print(f"median `code` rows per handler : {statistics.median(code_per_handler)}") + print(f"max `code` rows in one handler : {max(code_per_handler)}") + if refusals: + print("\nparse refusals (whole-file: the lens steps aside to the text editor):") + for refusal in refusals: + print(f" {refusal['file']}: {refusal['reason']}") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) From b6548c11dd413a3ac02d2e3a6cd7e202c23d9916 Mon Sep 17 00:00:00 2001 From: wshallwshall Date: Thu, 30 Jul 2026 14:30:50 -0500 Subject: [PATCH 2/4] quality: skip vendored trees, and measure WHY rows are opaque Two gaps found by running the scan against a real estate rather than samples/. 1. A real estate keeps its own .venv beside the config modules. A naive rglob walked ~1,900 site-packages files, spawned a CLI call per file, and buried the estate's own numbers. Dot-dirs, __pycache__, site-packages, node_modules, build and dist are now skipped. 2. The pre-registered decision rule (PR comment) referenced "median opaque rows per handler" and a cause mix, and the tool reported neither -- so it could not actually adjudicate its own rule. Both now reported. The cause classifier is a deliberate HEURISTIC on first-statement shape, not a parse. It exists to answer the one question the row contract cannot: how much of the opaque mass is a delegating call into a helper module, i.e. how much ADR 0089 Phase D would convert WITHOUT widening the grammar. Shapes only -- no source is ever emitted, only counts, so it stays estate-safe. Co-Authored-By: Claude Opus 5 --- scripts/quality/lens_coverage.py | 75 +++++++++++++++++++++++++++++++- 1 file changed, 74 insertions(+), 1 deletion(-) diff --git a/scripts/quality/lens_coverage.py b/scripts/quality/lens_coverage.py index c71032e2..063cd534 100644 --- a/scripts/quality/lens_coverage.py +++ b/scripts/quality/lens_coverage.py @@ -40,6 +40,23 @@ EDITABLE = frozenset({"action", "lookup", "send"}) +# A real estate keeps its own .venv beside the config modules, so a naive rglob walks thousands of +# site-packages files, spawns a CLI call for each, and buries the estate's own numbers. Any path +# component matching these is skipped, as is any dot-directory. +SKIP_DIRS = frozenset({"__pycache__", "site-packages", "node_modules", "build", "dist"}) + + +def config_modules(root: Path) -> list[Path]: + """Every `.py` under ``root`` that is plausibly a config module.""" + return sorted( + py + for py in root.rglob("*.py") + if not any( + part in SKIP_DIRS or (part.startswith(".") and part not in {".", ".."}) + for part in py.relative_to(root).parts + ) + ) + def parse_module(py: Path, python: str, cwd: Path) -> tuple[dict | None, str]: """Return (parsed JSON, error). `encoding=` is REQUIRED, not cosmetic. @@ -65,6 +82,33 @@ def parse_module(py: Path, python: str, cwd: Path) -> tuple[dict | None, str]: return None, f"non-JSON output ({exc})" +def classify_code_row(first_stmt: str) -> str: + """Bucket an opaque `code` row by the shape of its first statement. + + A HEURISTIC on source text, not a parse -- it exists to answer one question the row contract + cannot: how much of the opaque mass is a delegating call into a helper module, i.e. how much + ADR 0089's Phase D (helper descent) would convert without widening the grammar at all. Buckets + are shapes; no source is ever emitted, only counts. + """ + line = first_stmt.strip() + if not line or line.startswith("#"): + return "comment/blank" + if line.startswith(("import ", "from ")): + return "import" + if line.startswith("return"): + return "return" + if line.startswith(("try", "except", "finally", "with", "raise", "while", "assert", "yield")): + return "exception/other construct" + if line.startswith("msg."): + return "unrecognized msg API" + head, sep, rest = line.partition("=") + if sep and not head.rstrip().endswith(("=", "!", "<", ">")) and head.strip().isidentifier(): + return "assignment from call" if "(" in rest else "assignment (literal/expr)" + if "(" in line and line.split("(", 1)[0].replace(".", "_").isidentifier(): + return "bare helper call" + return "other" + + def _label(py: Path, root: Path, hash_names: bool) -> str: rel = py.relative_to(root).as_posix() if not hash_names: @@ -91,21 +135,37 @@ def main(argv: list[str] | None = None) -> int: return 2 kinds: Counter[str] = Counter() + causes: Counter[str] = Counter() files = handlers = zero_editable = fully_typed = 0 code_per_handler: list[int] = [] + opaque_per_handler: list[int] = [] refusals: list[dict[str, str]] = [] - for py in sorted(args.config_dir.rglob("*.py")): + for py in config_modules(args.config_dir): files += 1 out, err = parse_module(py, args.python, args.cwd) if out is None: refusals.append({"file": _label(py, args.config_dir, args.hash_names), "reason": err}) continue + try: + src = py.read_text(encoding="utf-8", errors="replace").splitlines() + except OSError: + src = [] for handler in out.get("handlers", []): handlers += 1 rows = handler.get("rows", []) n_edit = n_opaque = n_code = 0 for row in rows: + if row.get("kind") == "code": + start = row.get("line_start") + end = row.get("line_end", start) + stmt = "" + if isinstance(start, int) and src: + for raw in src[start - 1 : (end if isinstance(end, int) else start)]: + if raw.strip() and not raw.strip().startswith("#"): + stmt = raw + break + causes[classify_code_row(stmt)] += 1 kind = row.get("kind", "?") if kind == "control" and not row.get("recognized", True): kind = "control (unrecognized)" @@ -123,6 +183,7 @@ def main(argv: list[str] | None = None) -> int: if rows and n_opaque == 0: fully_typed += 1 code_per_handler.append(n_code) + opaque_per_handler.append(n_opaque) total = sum(kinds.values()) opaque = kinds["code"] + kinds["control (unrecognized)"] @@ -176,6 +237,18 @@ def pct(n: int, d: int) -> str: if code_per_handler: print(f"median `code` rows per handler : {statistics.median(code_per_handler)}") print(f"max `code` rows in one handler : {max(code_per_handler)}") + if opaque_per_handler: + print(f"median OPAQUE rows per handler : {statistics.median(opaque_per_handler)}") + print(f"max OPAQUE rows in one handler : {max(opaque_per_handler)}") + if causes: + total_code = sum(causes.values()) + print("\nwhy `code` rows are opaque (first-statement shape, heuristic):") + for cause, count in causes.most_common(): + print(f" {cause:<30} {count:>5} {pct(count, total_code)}") + descent = causes["bare helper call"] + causes["assignment from call"] + print( + f" -> helper-descent candidates : {descent:>5} {pct(descent, total_code)} of code rows" + ) if refusals: print("\nparse refusals (whole-file: the lens steps aside to the text editor):") for refusal in refusals: From 3eadbb85e57bab7a939d22d998884b19a1787136 Mon Sep 17 00:00:00 2001 From: wshallwshall Date: Thu, 30 Jul 2026 14:35:57 -0500 Subject: [PATCH 3/4] quality: drop hashlib from the scan -- the crypto gate was right The ASVS 11.1.3 crypto-inventory gate failed this branch: --hash-names used hashlib.sha256 to redact file names, and hashlib is a tracked crypto module with no inventory entry for a dev script. Adding an inventory entry would have been the wrong fix, because the hash was also the wrong PRIMITIVE. Estate modules follow a rigid naming convention (IB____handler.py), so a candidate list is cheap to generate and digest -- a truncated SHA-256 of a file name is reversible by dictionary attack. The flag is now --anonymize and assigns stable per-run opaque indices (.py), which carry no preimage at all. More private, and no crypto import in a quality script. Gate now reports "OK - 57 documented crypto call site(s), no drift." Co-Authored-By: Claude Opus 5 --- scripts/quality/lens_coverage.py | 28 +++++++++++++++++++++------- 1 file changed, 21 insertions(+), 7 deletions(-) diff --git a/scripts/quality/lens_coverage.py b/scripts/quality/lens_coverage.py index 063cd534..3cd7639d 100644 --- a/scripts/quality/lens_coverage.py +++ b/scripts/quality/lens_coverage.py @@ -24,13 +24,12 @@ PHI. `lens parse` is a static `ast` parse that never imports or executes a config module, and no message ever enters this path. The scan reads only code and emits only counts and file names, so it -is safe to run against a production estate. Pass `--hash-names` when sending results anywhere. +is safe to run against a production estate. Pass `--anonymize` when sending results anywhere. """ from __future__ import annotations import argparse -import hashlib import json import statistics import subprocess @@ -109,11 +108,21 @@ def classify_code_row(first_stmt: str) -> str: return "other" -def _label(py: Path, root: Path, hash_names: bool) -> str: +def _label(py: Path, root: Path, anonymize: bool, seen: dict[str, str]) -> str: + """Estate-relative path, or a stable opaque index when anonymizing. + + Deliberately NOT a hash. A hash of a file name is reversible by dictionary attack -- estate + modules follow a rigid naming convention (`IB____handler.py`), so a candidate + list is cheap to generate and digest. A per-run counter carries no preimage at all. It also + keeps `hashlib` out of a dev script, which the ASVS 11.1.3 crypto-inventory gate tracks as a + deployed-system crypto import. + """ rel = py.relative_to(root).as_posix() - if not hash_names: + if not anonymize: return rel - return f"<{hashlib.sha256(rel.encode()).hexdigest()[:12]}>.py" + if rel not in seen: + seen[rel] = f".py" + return seen[rel] def main(argv: list[str] | None = None) -> int: @@ -126,7 +135,9 @@ def main(argv: list[str] | None = None) -> int: action="store_true", help="count only UNRECOGNIZED control rows as opaque (recognized control is not opaque)", ) - ap.add_argument("--hash-names", action="store_true", help="hash file names in the output") + ap.add_argument( + "--anonymize", action="store_true", help="replace file names with stable opaque indices" + ) ap.add_argument("--json", dest="as_json", action="store_true", help="emit JSON") args = ap.parse_args(argv) @@ -140,12 +151,15 @@ def main(argv: list[str] | None = None) -> int: code_per_handler: list[int] = [] opaque_per_handler: list[int] = [] refusals: list[dict[str, str]] = [] + labels: dict[str, str] = {} for py in config_modules(args.config_dir): files += 1 out, err = parse_module(py, args.python, args.cwd) if out is None: - refusals.append({"file": _label(py, args.config_dir, args.hash_names), "reason": err}) + refusals.append( + {"file": _label(py, args.config_dir, args.anonymize, labels), "reason": err} + ) continue try: src = py.read_text(encoding="utf-8", errors="replace").splitlines() From 151b7e2db4772a8632cfeb28b1ce36aeec4bc1d3 Mon Sep 17 00:00:00 2001 From: wshallwshall Date: Thu, 30 Jul 2026 16:02:02 -0500 Subject: [PATCH 4/4] docs(backlog): mark the estate re-measure shipped, and record the RED override The item's banner said the tooling was unmerged; this PR merges it. Flips it to shipped and carries the numbers the scan produced, so the backlog states the result rather than pointing at a PR comment. Also records, in the item itself, that the pre-registered decision rule fired RED and that the RED prescription was NOT adopted. Both triggers landed exactly on their boundaries and the AMBER prescription was taken instead on a delegated judgment call that the owner never explicitly ratified. That override is load-bearing for the next several items, so it belongs on the record next to the number rather than only in a handoff. Co-Authored-By: Claude Opus 4.8 --- docs/BACKLOG.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/docs/BACKLOG.md b/docs/BACKLOG.md index 5165250e..841c3c61 100644 --- a/docs/BACKLOG.md +++ b/docs/BACKLOG.md @@ -7040,7 +7040,9 @@ The webview cannot import from `src/` (it is loaded as a plain script into a `de ## 239. Re-measure Steps view estate coverage (opaque vs editable rows) after the palette -> 🚧 **PARTIAL — measured 2026-07-30, tooling not yet merged.** The scan ran against the de-identified estate (388 files · 145 handlers · 1,423 rows · 0 parse refusals; editable share **42.0%**, fully-typed handlers **14.5%**) and the full result is recorded on PR #81. The re-runnable `scripts/quality/lens_coverage.py` is still unmerged, so the number is **not yet reproducible from `main`**. +> ✅ **SHIPPED (2026-07-30, PR #81).** `scripts/quality/lens_coverage.py` drives the shipped `lens parse --json` — not a second `ast` walk — so the number cannot drift from what the Steps view actually renders. Measured against the de-identified estate: 388 files · 145 handlers · 1,423 rows · **0 parse refusals**; editable share **42.0%**, fully-typed handlers **14.5%** (21/145), median opaque rows/handler **3**. Full result and the pre-registered decision rule are recorded on PR #81. +> +> ⚠️ **The pre-registered rule fired 🔴 RED, and the RED prescription was *not* adopted** — both triggers landed exactly on their boundaries (B = 14.5% missed the 15% floor by 0.5pp; median opaque = 3 hit `≥ 3` exactly), while A = 42.0% sat mid-AMBER. The AMBER prescription (breadth before depth) was taken instead, on the argument that the opacity is *mechanical* — comment-only rows (28%) plus helper delegation (41.8%) are ~70% of the opaque mass and both are addressable within the projection model. **This override was a delegated judgment call, never explicitly ratified by the owner**; treat it as open if the next measurement does not move. See #240 (comment-only rows) and ADR 0089 Phase D (helper descent). **Cluster:** IDE & Authoring. **Priority:** P1 — this number decides how much further Steps-view investment is justified. **Verdict:** build (cheap, reproducible). **Severity:** none (measurement).