fix(msdmd): never read own outputs; close consumer-review findings

CHANGELOG.md

@@ -0,0 +1,52 @@
+# Changelog
+
+Notable changes to shipped skill behaviour that consumers must act on. Newest
+first. Propagation PRs cite the skill-lib commit; this file explains why.
+
+## Unreleased: msdmd collector consumer review (skill-lib #118)
+
+Consumers re-syncing past `9867ab3` must:
+
+- **Install the native reader runtimes** where the collector runs:
+  `python -m pip install -r .agents/skills/msdmd/requirements.txt` and
+  `npm ci --ignore-scripts --prefix .agents/skills/msdmd` (Node required).
+  A missing runtime now exits **3** without writing. Opt-out:
+  `--allow-missing-reader-runtimes` writes output marked `runtime-unavailable`
+  with error diagnostics and still warns.
+- **Propagate the current `msdmd/collection.ts` helper.** Schema-2 output now
+  requires `MSDMD_COLLECTION_HELPER_VERSION` >= 1 in the helper named by
+  `--import-path`; an older or schema-1 helper exits **4** without writing.
+  Opt-out: `--legacy-blocks-only` for explicit schema-1 block-only output.
+  When `--out` is outside the repository (for example a CI file under /tmp),
+  the helper is located from the repository root.
+- **Run the collector inside a usable git checkout.** If git cannot list the
+  visible files (git missing from PATH, a corrupt index, a broken `.git`
+  marker) or the root is ignored by an enclosing repository, the collector
+  prints an ERROR and exits **5** without writing. There is no opt-out; collect
+  from a checkout git can read. `--check` reports 5, not drift (1).
+  Exit 3, 4 and 5 problems are reported together; precedence is 5, then 3,
+  then 4.
+- **Use `--print-generator-identity`** (add `--json` for components) as the
+  freshness fingerprint. It covers all collector sources, the TypeScript worker
+  and lock file, the Python minor, reader package, Node and TypeScript
+  versions, and digests of the reader modules that actually resolve (a
+  shadowing module on PYTHONPATH changes it).
+
+Behaviour changes:
+
+- The collector never reads its own output, sibling temp/candidate outputs, or
+  a redirected stdout file; differently named runs render identical bytes.
+- In git checkouts only tracked and untracked-not-ignored files are read. A git
+  listing failure or a root ignored by an enclosing repository exits 5. Submodules are excluded ledger entries that
+  record their pinned commit.
+- Retained source bytes have an aggregate budget (`--max-total-bytes`, default
+  256 MiB, must be positive); overflow is an error diagnostic and a warning.
+- Redaction covers camel/PascalCase secret names whose final word is sensitive
+  (`authorization` added), systemd URL credentials and credential directives,
+  SVG metadata and raw DSSE payloads. A DSSE payload that is not an in-toto
+  Statement is diagnosed and kept as a decoded `signed-payload` fact.
+- Requirements files: unnamed URL/VCS/path/Windows-path entries stay `hmmm`;
+  continuations are joined as pip does (comment lines never continue) and a
+  trailing continuation at end of file is diagnosed.
+- ratios semantic graphs resolve schema-2 addresses and never count an
+  ambiguous or unresolved target as resolved.

ORG_DISTRIBUTION.md

@@ -123,6 +123,21 @@ python -m msdmd.collect --root . --repo The-Interdependency/skill-lib --snapshot
 python -m unittest tests.test_msdmd_native_contract_docs
 ```
 
+Native reader runtimes must be installed where the collector runs, from the
+skill's own directory (`msdmd/` here, `.agents/skills/msdmd/` in consumers):
+`python -m pip install -r <skill>/requirements.txt` and
+`npm ci --ignore-scripts --prefix <skill>` (Node required). The collector exits
+**3** without writing when a runtime is missing (opt-out:
+`--allow-missing-reader-runtimes`, which writes output marked
+`runtime-unavailable` and still warns) and **4** without writing when the
+`--import-path` helper is older than the schema-2 output
+(`MSDMD_COLLECTION_HELPER_VERSION`; fix by propagating the current skill, or
+opt out with `--legacy-blocks-only`). It exits **5** without writing when git
+cannot list the visible files (for example git missing from PATH) or the root
+is ignored by an enclosing repository; there is no opt-out, so run it in a
+readable git checkout. `--check` reports 5, not drift. All three are reported in
+one run, with precedence 5, then 3, then 4.
+
 The shipped collector defaults to schema 2. It integrates supported native facts
 through `msdmd/readers.py` and supplemental MSDMD blocks through the universal
 parser path. Its legacy/block-only output remains available only as an explicit

docs/propagation-checklist.md

@@ -55,9 +55,22 @@ Minimum target checks:
 
 ```bash
 python .agents/skills/manifest/generate.py --check
+python -m pip install -r .agents/skills/msdmd/requirements.txt
+npm ci --ignore-scripts --prefix .agents/skills/msdmd
 python -m msdmd.collect --root . --repo <repo> --out <repo>_msdmd.ts
 ```
 
+Exit 3 means a native reader runtime is missing (install the two runtimes
+above; `--allow-missing-reader-runtimes` writes incomplete, flagged output).
+Exit 4 means the target's `.agents/skills/msdmd/collection.ts` helper is older
+than the schema-2 output (`MSDMD_COLLECTION_HELPER_VERSION`); propagate the
+current msdmd skill, or use `--legacy-blocks-only` for explicit schema-1 output.
+Exit 5 means git could not list the visible files (git missing from PATH, a
+corrupt index or broken `.git`) or the root is git-ignored by an enclosing
+repository; run in a readable checkout (no opt-out). `--check` reports 5, not
+drift. Nothing is written on exit 3, 4 or 5. Runners that fingerprint freshness should use
+`python -m msdmd.collect --print-generator-identity`.
+
 Run any target-local checks named in `CLAUDE.md`, `AGENTS.md`, or
 `.agents/skills/README.md`.
 

docs/runner-config-guidance.md

@@ -29,10 +29,23 @@ remain visible findings rather than being converted into missing-block gaps.
 Usage for a schema-2 collection:
 
 ```bash
+python -m pip install -r .agents/skills/msdmd/requirements.txt
+npm ci --ignore-scripts --prefix .agents/skills/msdmd
 python -m msdmd.collect --root . --repo example --out example_msdmd.ts
 python -m msdmd.visualize example_msdmd.ts --out example_msdmd.mmd
 ```
 
+Runners must install both native runtimes first. Exit codes: 1 drift under
+`--check`; 2 error diagnostics under `--strict`; 3 missing native reader runtime
+(opt-out `--allow-missing-reader-runtimes`, which writes `runtime-unavailable`
+output and warns); 4 schema helper older than the output
+(`MSDMD_COLLECTION_HELPER_VERSION`; propagate the skill or opt out with
+`--legacy-blocks-only`); 5 git cannot list visible files (git missing from
+PATH, corrupt index, broken `.git`) or the root is git-ignored by an enclosing
+repository (no opt-out; `--check` reports 5, not drift). Exits 3, 4 and 5 write
+nothing and are reported together, with precedence 5, then 3, then 4.
+Use `--print-generator-identity` as the generator fingerprint.
+
 These commands implement the shipped reader matrix, not every possible metadata
 standard. A complete compliance verdict still depends on the declared repository
 scope, required facts, supported readers, unresolved inputs, exclusions, and

msdmd/SKILL.md

@@ -36,7 +36,10 @@ python -m pip install -r msdmd/requirements.txt
 npm ci --ignore-scripts --prefix msdmd
 ```
 
-Missing dependencies become unsupported-reader diagnostics, never empty success.
+A missing runtime is an error diagnostic and marks the reader run
+`runtime-unavailable`; it never yields empty success. The CLI exits 3 without
+writing unless `--allow-missing-reader-runtimes` is given, which still warns.
+Other TypeScript worker failures are `typescript_reader_failed` errors.
 The universal MSDMD block parsers themselves remain dependency-free.
 
 ## Doctrine
@@ -344,6 +347,25 @@ python -m msdmd.collect --root . --repo The-Interdependency/skill-lib \
 Source revision, worktree state and byte digests remain separate. A snapshot is
 not producer authentication. Excluded output paths are configured even before
 their first write, preventing generation from changing its own source identity.
+The collector never reads its own output: the `--out` path, its hidden
+`.<name>.*` siblings, any `.*_msdmd.ts.*` candidate and a shell-redirected
+stdout file are skipped and unrecorded, so differently named temporaries render
+identical bytes. Only the stable `<repo>_msdmd.ts` name is recorded. Git-ignored
+files are never read in a Git checkout; if git cannot list files there (for
+example, git is not on PATH), or the root is ignored by an enclosing repository,
+the CLI prints an ERROR and exits 5 without writing (`--check` reports 5, not
+drift) instead of reading ignored files. A git submodule is never read: it
+is an `excluded` ledger entry (`entry_kind: submodule`, `reason: git-submodule`)
+carrying the pinned `commit`, which snapshot identities include.
+`--print-generator-identity` prints a digest of every collector file that can
+change output (TypeScript worker and lock file included), the Python minor,
+reader package, Node and TypeScript versions, and digests of the reader modules
+that actually resolve on `sys.path`; add `--json` for the parts. A schema-2
+artifact needs a helper exporting `MSDMD_COLLECTION_HELPER_VERSION` at least the
+collector's; an older helper stops the CLI with exit 4 before writing. When
+`--out` is outside the root, the helper is located from the root. Propagate the
+skill or use `--legacy-blocks-only`. Exit 3, 4 and 5 problems are reported
+together, with precedence 5, then 3, then 4.
 
 ### Required information and disclosure
 
@@ -364,7 +386,10 @@ Parse errors and identity conflicts always invalidate strict collection. Native
 facts and blocks remain separate; no universal precedence overwrites disagreements.
 
 Directory-descriptor discovery does not follow symlinks. It accounts for ignored
-subtrees, unsupported inputs, byte limits and read errors. This safety path
+subtrees, unsupported inputs, byte limits and read errors. Bytes no reader can
+consume are hashed but not retained; retained bytes have an aggregate bound
+(`--max-total-bytes`, default 256 MiB) whose overflow is an error and a CLI
+warning. This safety path
 requires POSIX no-follow directory-descriptor support. Secret-named files are
 excluded; sensitive structured fields and URL credentials are redacted. This is
 not a complete secret detector: public release still requires audience review.