Skip to content

fix(msdmd): never read own outputs; close consumer-review findings - #118

Merged
erinepshovel-code merged 4 commits into
mainfrom
fix/msdmd-consumer-review
Oct 6, 2026
Merged

erinepshovel-code merged 4 commits into
mainfrom
fix/msdmd-consumer-review

Conversation

@erinepshovel-code

@erinepshovel-code erinepshovel-code commented Oct 6, 2026 •

Copy link
Copy Markdown
Contributor

Base: main at 9867ab33877f2b1f50f8a501cf379aa99360bd52. One PR; Erin merges.

This fixes the collector bug that blocks stack #78 and the Codex P1/P2 findings on the 9867ab3 msdmd/ratios code that Erin chose to fix. The findings come from the review comments on the stack #78 and batch-1 consumer sync PRs. Every fix has a regression test in tests/test_msdmd_consumer_review.py (16 tests). All 16 fail on 9867ab3 and pass here.

Stack #78 will be re-synced to the new main with a normal commit after this merges. That re-sync also needs to switch backend/msdmd.py to --print-generator-identity and install the native-reader runtimes (see finding 6 and 8).

Findings, sources, fixes, tests

1. Collector reads its own output or temp files (stack #78 blocker)

  • Sources:
  • Fix:
    • Discovery and dirty-worktree identity skip these paths:
      • the configured output paths
      • .<outname>.* sibling temp and candidate outputs
      • collector .msdmd-* temp files
      • .*_msdmd.ts.* siblings
      • a regular file that stdout is redirected to (matched by inode)
    • Only the canonical <repo>_msdmd.ts is recorded as an output entry.
    • Dirty detection uses git status --porcelain -z -- ., scoped to the collection root.
  • Tests (OwnOutputTests):
    • test_stack_candidate_then_verifier_flow_is_byte_identical reproduces the candidate-then-verifier flow in stack backend/msdmd.py. It writes two differently named outputs into the repo and checks they are byte-identical, then checks a fresh-status verify.
    • test_configured_output_does_not_dirty_the_worktree_or_drift
    • test_stdout_redirect_into_the_scanned_tree_is_not_an_input

2. Gaps in secret redaction

  • Sources:
  • Fix:
    • Sensitive-key matching normalizes camelCase and PascalCase.
    • URL userinfo credentials are removed from systemd Environment values and other directives.
    • SetCredential and SetCredentialEncrypted become ID:<redacted>.
    • SVG metadata goes through _redact_sensitive and emits a diagnostic.
    • The DSSE envelope fact withholds the raw payload, which is projected from the decoded statement (dsse-envelope@1, supersedes the structured-document fact).
  • Tests (RedactionTests):
    • test_camel_and_pascal_case_secret_keys_are_redacted (JSON, YAML, TOML)
    • test_systemd_url_credentials_and_credential_data_are_withheld
    • test_svg_metadata_is_redacted_with_a_diagnostic
    • test_dsse_envelope_does_not_republish_the_raw_payload

3. URL requirements turned into package names

4. Schema-2 addresses in the ratios semantic graph

  • Sources:
  • Fix: semantic_file_graph indexes declarations by schema-2 address as well as id. Owner lookup tries the address first.
  • Test: RatiosSemanticGraphTests.test_schema_two_block_edges_resolve_to_owning_files

5. Memory limit

  • Sources:
  • Fix:
    • Bytes are kept only for files that a reader or comment marker applies to. Other files are hashed and sized only.
    • A total retained-bytes budget applies: --max-total-bytes, default 256 MiB.
    • Files over the budget are excluded with reason aggregate-size-limit, and an aggregate_size_limit error diagnostic is emitted.
  • Test: DiscoveryMemoryTests.test_aggregate_byte_budget_is_enforced_and_diagnosed

6. Generator fingerprint covered only .py files

  • Source: stack#78 typescript-reader.cjs:8 P1
  • Fix: new msdmd.collect.generator_identity() and --print-generator-identity. The fingerprint hashes every .py, .cjs and .json file under msdmd/ (TS reader, package.json, package-lock.json, schema) plus requirements.txt. It excludes node_modules, __pycache__ and references/.
  • Follow-up: stack's backend/msdmd.py has to call this during the stack re-sync.
  • Test: GeneratorIdentityTests.test_identity_covers_typescript_worker_lock_and_assets_only

7. Unusable output for repos that still carry the schema-1 helper

  • Source: stack#78 collect.py:726 P1
  • Fix:
    • Before writing, the collector checks that the relative --import-path helper exports the names the rendered collection imports (e.g. defineMsdmdCollectionV2). Superseded by the follow-up below: the check now compares an exported MSDMD_COLLECTION_HELPER_VERSION.
    • If it does not, the collector exits 4 without writing and tells you to update the helper or pass --legacy-blocks-only.
    • If the helper is not found, it only warns.
  • Test: SchemaHelperTests.test_schema_one_helper_target_is_refused_without_writing (schema-1 helper is refused, legacy flag works, schema-2 helper works)

8. Missing native-reader dependencies gave matching but incomplete output

  • Source: stack#78 requirements.txt:5 P1
  • Fix:
    • missing_docstring_parser, typescript_reader_unavailable and reader_dependency_unavailable are now error severity, and the reader run status is runtime-unavailable (added to collection.ts).
    • The CLI prints an ERROR and exits 3 without writing, unless --allow-missing-reader-runtimes is passed. With that flag it writes and still prints a loud WARNING.
  • Tests (ReaderRuntimeTests):
    • test_missing_runtime_is_an_error_and_marks_the_reader
    • test_cli_fails_closed_unless_explicitly_allowed

9. Git-ignored local files read into collections

  • Source: stack#78 collect.py:272 P1
  • Fix: in git checkouts, discovery is limited to git ls-files --cached --others --exclude-standard. This applies in both commit-bound and snapshot modes.
  • Test: GitIgnoredInputTests.test_ignored_local_files_are_never_read

hmmm (not changed here)

  • a0#111 readers.py:354 (identifier-keyed maps over-redacted, e.g. a dependency named token): fixing it needs schema-aware redaction, which is a design choice. The camelCase widening in finding 2 adds a few false positives (e.g. passwordHash is now redacted).
  • Default total budget of 256 MiB: my choice. It can be changed with --max-total-bytes.
  • Submodule contents are no longer read, because they are not in git ls-files.
  • Codex findings outside this PR's scope are left for a later decision: large TOML/YAML integers, readable output modes, CODEOWNERS precedence and empty rules, shebang scripts, the RATIOS default block set and unsupported-language certification, comment extraction for unsupported languages, expected blocks, comment columns, framework build-tree exclusions, YAML double exponents and JSON out-of-range numbers, uppercase header suffix, parser shadowing and module-scope __all__, llms.txt root restriction, SPDX/JSON Schema/$ref items, TS re-exports and destructuring, visualize indirect helper, ratios named seals, and stack-only vm-mcp and tools/ai.sh.

Gates (local, Node 24.15.0, Python 3.13)

  • unittest: 396 OK
  • msdmd collect --check and tsc --strict: pass. A clean-clone --check also passes.
  • gonol authority, skill-lib drift, skill compliance, Codex adapters: pass
  • ratios --strict: pass
  • llms --check: pass
  • repo LOTO audit and run: pass
  • No tracked bytecode.

skill-lib_msdmd.ts is regenerated. Consumer <repo>_msdmd.ts files stay as they are until each consumer is re-synced.

The second commit (7498f42) renames a fixture dict in the redaction test. This clears a CodeQL py/clear-text-storage-sensitive-data false positive on dummy test values, which I confirmed with a local CodeQL 2.27.1 run. It changes no behaviour, and skill-lib_msdmd.ts is regenerated.

Follow-up commit a97d7fd: Review findings

Tests are in tests/test_msdmd_review_followup.py (18 tests). 16 of them fail on 7498f42; the other 2 are controls. The redaction test in tests/test_msdmd_consumer_review.py now asserts the keep list.

  1. Requirements (readers.py ~847/~823):
    • As in pip, a comment line never continues.
    • A trailing \ at end of file is diagnosed as dangling_requirement_continuation.
    • C:\x and D:/x stay unnamed.
    • Tests: RequirementLineTests.
  2. Git ls-files failure (collect.py ~259): if a .git marker exists but git cannot list files (corrupt index, broken gitdir), nothing is read. The collector emits a git_visibility_unavailable error and marks the snapshot incomplete. Tests: GitVisibilityTests.test_ls_files_failure_* and test_broken_git_marker_*.
  3. Ignored root (collect.py ~247/~399): git check-ignore -q . detects a root that an enclosing repo ignores. It emits a root_git_ignored error and marks the snapshot incomplete. Tracked files are still read. Test: test_root_inside_an_ignored_directory_*.
  4. Helper version (collect.py ~922):
    • collection.ts now exports MSDMD_COLLECTION_HELPER_VERSION = 1, and the collector requires it to be at least REQUIRED_HELPER_VERSION.
    • 9867ab3-era schema-2 helpers have no constant and exit 4. I confirmed that such a helper fails tsc with TS2322 on runtime-unavailable.
    • Schema-1 output still checks only for defineMsdmdCollection.
    • Tests: SchemaHelperVersionTests.
  5. Fingerprint (collect.py ~880):
    • generator_identity() now also covers the Python minor version, the installed version of each pinned reader package, and the Node and TypeScript versions the worker resolves. A missing one is recorded as absent.
    • --print-generator-identity --json shows the components.
    • Test: GeneratorIdentityRuntimeTests.
  6. DSSE (standards.py ~206):
    • A payload that is not an in-toto Statement gets a dsse_payload_not_in_toto_statement diagnostic and is kept as a decoded, redacted signed-payload fact.
    • supersedes is set only after the replacement fact is emitted.
    • Test: DsseSupersessionTests.
  7. Docs:
    • New CHANGELOG.md.
    • ORG_DISTRIBUTION.md, docs/propagation-checklist.md and docs/runner-config-guidance.md now cover runtime installation, exit codes 3 and 4, and their opt-outs.
    • This body now says --legacy-blocks-only.
  8. camelCase redaction:
    • A camelCase name counts as secret-bearing only when its final word is sensitive, or its final pair is api/private/access/secret + key.
    • So tokenUrl, passwordPolicyUrl, passwordHash, apiKeyPrefix, accessKeyId, tokenCount, tokenType, secretName and privateKeyPath are kept.
    • authorization was added.
    • snake_case matching is unchanged.
    • Tests: CamelRedactionFinalTokenTests and the updated RedactionTests.
  9. Ambiguous targets (harmonics.py ~110): edges with target_resolution of ambiguous or external-or-unresolved are listed as unresolved and never added to adjacency. Test: AmbiguousSemanticEdgeTests.
  10. Submodules:
    • A gitlink becomes an excluded ledger entry (entry_kind: submodule, reason: git-submodule, commit). Its contents are never read.
    • The pin is included in the snapshot identity and does not make the snapshot incomplete.
    • Documented in SKILL.md.
    • Test: SubmoduleLedgerTests.
  11. CLI:
    • The exit-3 message names the skill directory actually in use and the opt-out flag.
    • Exit 3 and exit 4 problems are reported in one run (3 takes precedence).
    • Only Cannot find module 'typescript' counts as typescript_reader_unavailable. Other worker exits are typescript_reader_failed errors, and stderr is not published.
    • --out is resolved before the helper check.
    • Tests: test_runtime_and_helper_problems_are_reported_together, TypeScriptWorkerFailureTests, and the relative --out case.
  12. Budget: going over the budget always prints a WARNING. --max-total-bytes and --max-file-bytes must be positive (argparse). Test: BudgetVisibilityTests.
  13. Renames (collect.py ~302): both porcelain columns are read, and the source and destination paths are both checked. Test: RenameDetectionTests.

Follow-up 2 (faf85c7): exit 5, out-of-tree helper, resolved reader modules

  1. Exit 5: git_visibility_unavailable and root_git_ignored now print msdmd: ERROR: <code>: … and exit 5 without writing.
    • This also applies under --legacy-blocks-only.
    • --check returns 5, not drift 1, and --strict cannot mask it as 2.
    • Precedence is 5, then 3, then 4, and all problems are reported in one run.
    • There is no opt-out.
    • Tests: VisibilityExitTests (no git on PATH; ignored root under plain, --strict and --check).
  2. Helper check with an out-of-tree --out (11d): when --out is outside --root (for example /tmp in edcm/ptcna CI), the --import-path helper is located from the root. The MSDMD_COLLECTION_HELPER_VERSION check (exit 4) now runs there. Test: OutOfTreeHelperTests.
  3. Resolved reader modules in the generator identity (B): a new python_modules component holds a sha256 of the files each reader package's import names resolve to through importlib.util.find_spec (nothing is imported). A shadowing docstring_parser on PYTHONPATH now changes the identity even though the metadata versions do not. Test: ShadowedReaderModuleTests.
  4. Docs: exit 5 is documented in SKILL.md, CHANGELOG, ORG_DISTRIBUTION.md, propagation-checklist and runner-config-guidance. skill-lib_msdmd.ts was regenerated.

Known follow-ups (not in this PR)

  • C: --legacy-blocks-only with a custom --out can read its own previous output. This is pre-existing.
  • D: untracked nested repositories are not yet treated like submodules. The redaction split between password_hash (snake_case, redacted) and passwordHash (camelCase, kept) is not yet reconciled.

Gates for faf85c7 (Node 24.15.0, Python 3.13): 418 tests OK; collect --check, tsc, gonol, drift, compliance, Codex adapters, ratios strict, llms, LOTO and bytecode all pass.

Earlier gates (a97d7fd, Node 24.15.0, Python 3.13): 414 tests OK, collect --check, tsc, gonol, drift, compliance, Codex adapters, ratios strict, llms, LOTO and bytecode all pass. A local CodeQL 2.27.1 python-code-scanning suite reports 0 alerts.

On Node 20.20.2, collect --check and tsc pass. The pre-existing test_universal_parser.test_typescript_numeric_field_contract fails because it imports a .ts file directly, which Node 20 cannot do.

Base: main 9867ab3.

- collector excludes the configured output, sibling temp/candidate
  outputs, collector temp files, and a redirected stdout file, so
  differently-named runs produce identical collections (stack #78 blocker)
- git-ignored local files are never read
- aggregate retained-bytes budget (--max-total-bytes, default 256 MiB);
  bytes only retained for files a reader or marker applies to
- missing native reader runtimes are errors (runtime-unavailable, exit 3)
  unless --allow-missing-reader-runtimes
- schema-1 relative helper detected before writing (exit 4)
- generator_identity()/--print-generator-identity covers Python, TS
  worker, package/lock, schema and requirements files
- redaction: camel/PascalCase secret keys, systemd URL credentials and
  credential directives, SVG metadata, DSSE envelope payload
- unnamed URL/VCS/path requirements stay hmmm instead of fabricated
  package names; backslash continuations joined
- ratios semantic graph resolves schema-2 declaration addresses
- regression tests: tests/test_msdmd_consumer_review.py
- regenerated skill-lib_msdmd.ts
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 6, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-06T07:04:44.568035Z faf85c7 New commits
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

Comment thread tests/test_msdmd_consumer_review.py Fixed
py/clear-text-storage-sensitive-data flagged the dummy fixture dict named
'secrets' written into a temp repo. Rename only; behaviour unchanged.
Regenerate skill-lib_msdmd.ts.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 034f0262db

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread msdmd/collect.py
Comment on lines +888 to +892
files = sorted(
path for path in base.rglob("*")
if path.is_file()
and not {"node_modules", "__pycache__", "references"} & set(path.relative_to(base).parts)
and (path.suffix in {".py", ".cjs", ".json"} or path.name == "requirements.txt")

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Include the Python runtime in generator identity

When collection jobs run under different Python minor versions, read_python emits a different Python X.Y convention version and the AST grammar/output can also differ, but generator_identity() returns the same digest because it hashes only files below msdmd. A freshness controller can therefore reuse a receipt and skip regeneration even though the current generator would produce different bytes; include the effective Python/runtime identity in this digest or bind it separately in the freshness key.

AGENTS.md reference: AGENTS.md:L176-L179

Useful? React with 👍 / 👎.

Comment thread msdmd/standards.py Outdated
Comment on lines +204 to +207
envelope = dict(value, payload={'$withheld': True, 'reason': 'decoded-statement-projected', 'pointer': '/payload'})
envelope_fact = emit('dsse.envelope', 'v1', 'signed-envelope', '', envelope, scope='attestation', standing='reported-evidence')
envelope_fact['projection'] = {'mapping_version': 'dsse-envelope@1', 'supersedes': 'structured-document',
'loss': 'raw base64 payload withheld; decoded statement published at /payload'}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Supersede the DSSE root only after projecting its payload

For a DSSE envelope whose payload decodes successfully but is not an in-toto Statement object—for example {"foo":"bar"}—this unconditional supersedes marker removes the generic structured-document fact, while the _type check below emits no decoded-payload fact. The result contains only a $withheld placeholder claiming the payload was projected, with neither the data nor an unsupported/invalid diagnostic; only supersede after a replacement fact is emitted, or preserve the redacted decoded payload as hmmm.

AGENTS.md reference: AGENTS.md:L124-L124

Useful? React with 👍 / 👎.

Comment thread msdmd/readers.py Outdated
Comment on lines +327 to +330
def _is_sensitive_key(name: Any) -> bool:
"""Match secret-bearing names in snake, kebab, camel and Pascal case."""
text = str(name)
return bool(_SENSITIVE_KEY_RE.search(text) or _SENSITIVE_KEY_RE.search(_CAMEL_BOUNDARY_RE.sub("_", text)))

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Preserve descriptive camel-case metadata fields

When a non-secret field contains a sensitive-looking component, such as passwordPolicyUrl, tokenEndpoint, privateKeyAlgorithm, or secretRotationPeriod, splitting camel case and applying the existing unanchored matcher classifies the entire value as secret. This newly erases legitimate schema and configuration metadata—the added test even places passwordPolicyUrl in the keep fixture without asserting it—so camel/Pascal matching should distinguish terminal secret-value names from descriptive fields.

Useful? React with 👍 / 👎.

Comment thread msdmd/readers.py
Comment on lines +822 to +825
# Direct URL, VCS and local-path forms carry no leading distribution name.
_REQUIREMENT_DIRECT_RE = re.compile(
r"^(?:[A-Za-z][A-Za-z0-9+.-]*://|(?:git|hg|svn|bzr)\+|file:|\.{1,2}(?:/|\\|$)|/|~)", re.IGNORECASE)
_REQUIREMENT_ARCHIVE_RE = re.compile(r"\.(?:whl|zip|tar\.gz|tar\.bz2|tar\.xz|tgz|tar)(?:[#?]|$)", re.IGNORECASE)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Recognize Windows drive paths as unnamed requirements

When a requirements file contains an absolute Windows local path such as C:\src\pkg, this matcher does not recognize the drive-letter form, so the fallback name regex records package C and emits a false python-package:C dependency edge. Treat drive-letter paths as unnamed direct requirements, as is already done for ./, /, and ~ paths, so the distribution name remains hmmm rather than guessed.

AGENTS.md reference: AGENTS.md:L118-L124

Useful? React with 👍 / 👎.

Comment thread msdmd/collect.py Outdated
Comment on lines +259 to +260
except (OSError, subprocess.CalledProcessError):
return None

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Fail closed when Git visibility cannot be determined

In a Git checkout where the git executable is unavailable or git ls-files fails, this exception path returns None, which _discover() interprets as permission to scan every filesystem entry. Ignored local files such as private.json are then read and can be emitted even though the new contract says Git-ignored files are never read; distinguish a non-Git directory from a failed Git visibility check and abort or emit an error for the latter.

Useful? React with 👍 / 👎.

Comment thread ratios/harmonics.py
Comment on lines +108 to +110
def owners(key: str) -> set[str]:
# Exact address first; the short id is the legacy/unresolved fallback.
return address_files.get(key) or id_files.get(key, set())

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Keep ambiguous semantic edges unresolved

When a schema-2 edge targets an ID declared by multiple eligible files, the collector leaves the short target and marks it target_resolution: "ambiguous", but this fallback maps that ID to every owning file. Because the source address now resolves, semantic_file_graph() reports concrete edges to all candidates and increments resolved_edges instead of retaining the ambiguity; inspect target_resolution before applying the short-ID fallback.

AGENTS.md reference: AGENTS.md:L118-L124

Useful? React with 👍 / 👎.

Comment thread msdmd/collect.py
Comment on lines +920 to +923
if candidate.is_file():
text = candidate.read_text(encoding="utf-8", errors="replace")
missing = [name for name in names if not re.search(r"\bexport\b[^;]*\b" + re.escape(name) + r"\b", text)]
if missing:

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Verify exports without spanning unrelated declarations

When the helper contains a semicolonless exported declaration followed by a non-exported function named defineMsdmdCollectionV2, this regex starts at the earlier export and scans across lines until a later semicolon, falsely treating the local function as exported. The CLI then writes a collection that imports a value the helper does not provide; use syntax-aware export detection or constrain the match to the actual declaration/export statement.

Useful? React with 👍 / 👎.

Comment thread msdmd/readers.py
Comment on lines +852 to +853
if pending is not None:
records.append((start, line_number, pending))

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Reject unterminated requirement continuations

When a requirements file ends with a continuation backslash, this branch removes the backslash and emits the pending text as a normal dependency—for example, foo==1.0 \ becomes a supported foo==1.0 fact with no diagnostic. This can let malformed or truncated input satisfy required-source coverage; retain it as unresolved/invalid and report the unterminated continuation instead of silently completing it.

AGENTS.md reference: AGENTS.md:L124-L124

Useful? React with 👍 / 👎.

- requirements: comment lines never continue; trailing continuation at EOF
  is diagnosed; Windows paths (C:\x, D:/x) stay unnamed
- git visibility fails closed: ls-files failure or broken .git marker reads
  nothing and errors; a root ignored by an enclosing repo is diagnosed and
  the snapshot marked incomplete
- schema helper check compares exported MSDMD_COLLECTION_HELPER_VERSION
  (new in collection.ts) instead of a regex; 9867ab3-era helpers exit 4
- generator identity adds Python minor, reader package, Node and TypeScript
  versions; --print-generator-identity --json shows the components
- DSSE: non-Statement payload diagnosed and kept as a decoded signed-payload
  fact; supersedes set only after a replacement fact exists
- camelCase redaction only when the final word is sensitive; add
  authorization
- ratios: ambiguous/unresolved short-id targets are not resolved edges
- submodules: excluded ledger entry with pinned commit, in snapshot identity
- CLI: exit-3 message names the real skill dir and the opt-out; exit 3 and 4
  reported together; only a missing typescript package is a missing TS
  runtime (other worker exits are typescript_reader_failed); --out resolved
  for the helper check; budget overflow always warns; --max-*-bytes > 0
- rename detection reads both porcelain columns and both paths
- docs: CHANGELOG, ORG_DISTRIBUTION, propagation checklist, runner guidance
- tests: tests/test_msdmd_review_followup.py; regenerate skill-lib_msdmd.ts

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: a97d7fdbf8

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread msdmd/collect.py
match = re.match(r"\s*([A-Za-z0-9][A-Za-z0-9._-]*)", line.split("#", 1)[0])
if match:
try:
packages[match.group(1)] = metadata.version(match.group(1))

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Bind the actually imported reader code in generator identity

When PYTHONPATH, the working directory, or an editable installation supplies a different docstring_parser implementation under the same distribution version, metadata.version() does not identify the code that read_python() actually imports. Fresh evidence beyond the prior runtime-identity finding: two distinct docstring_parser.py modules selected through PYTHONPATH produced different documentation facts while generator_identity() returned the same digest, allowing a freshness controller to reuse an invalid receipt; isolate dependency resolution or hash the resolved module bytes.

AGENTS.md reference: AGENTS.md:L176-L179

Useful? React with 👍 / 👎.

Comment thread msdmd/collect.py
Comment on lines +1144 to +1145
excluded_inodes = []
if not args.out:

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Exclude outputs in legacy collection mode

When --legacy-blocks-only writes to a noncanonical filename or stdout is redirected into the scanned tree, this output/inode exclusion setup is skipped because it exists only in the non-legacy branch. For example, rerunning legacy collection with --expected-block DOCS --out custom.ts --check reads the first generated custom.ts, adds it as a missing-block gap, and reports drift; pass the actual output and redirected inode exclusions through collect_legacy() as well.

AGENTS.md reference: AGENTS.md:L176-L179

Useful? React with 👍 / 👎.

… resolved reader module identity

- git_visibility_unavailable / root_git_ignored now print an ERROR and exit 5
  without writing (also in --legacy-blocks-only). --check reports 5, not drift
  1; --strict cannot mask it as 2. Precedence 5, then 3, then 4; all reported
  together.
- With --out outside the root (e.g. /tmp in CI), the schema helper is located
  from the root so the MSDMD_COLLECTION_HELPER_VERSION check (exit 4) runs.
- Generator identity adds python_modules: digests of the files each reader
  package's import names resolve to via find_spec, so a shadowing module on
  PYTHONPATH changes the identity.
- Tests: no git on PATH, ignored root (plain/--strict/--check), out-of-tree
  helper, shadowed docstring_parser.
- Docs: exit 5 in SKILL.md, CHANGELOG, ORG_DISTRIBUTION.md,
  propagation-checklist, runner-config-guidance; regenerate skill-lib_msdmd.ts.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: faf85c7584

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread msdmd/collect.py
Comment on lines +1258 to +1259
if args.out and not args.json:
problem, found = helper_incompatibility(args.out, args.import_path, rendered, args.root)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Check the helper for redirected TypeScript output

When schema-2 output is redirected from stdout rather than written with --out, this condition skips helper_incompatibility() even though the collector detects that stdout is a regular file. A consumer can therefore run python -m msdmd.collect ... > repo_msdmd.ts against a schema-1 or unversioned helper, receive exit 0, and persist an artifact whose import cannot type-check; apply the helper gate to redirected TypeScript output as well.

Useful? React with 👍 / 👎.

Comment thread msdmd/collect.py
Comment on lines +407 to +410
for path, commit in sorted(submodules.items()):
# Submodule contents belong to another repository; record the pin, never read it.
ledger.append({"file": path, "entry_kind": "submodule", "status": "excluded", "reason": "git-submodule",
"reader_ids": [], "content_sha256": "hmmm", "commit": commit})

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Bind submodule pins into snapshot_sha256

When a gitlink is updated to a different submodule commit without changing other files, the discovery entry changes only in commit while retaining content_sha256: "hmmm"; the later snapshot_sha256 calculation hashes only file, content digest, and status, so it remains identical across the two source snapshots. Consumers using this advertised snapshot digest as an input identity can consequently reuse stale derived data; include the pinned commit in the snapshot digest, for example by using it as the submodule content identity.

AGENTS.md reference: AGENTS.md:L176-L179

Useful? React with 👍 / 👎.

Comment thread msdmd/collect.py
Comment on lines +1052 to +1055
try:
probe = subprocess.run(["node", "-e", _NODE_PROBE], cwd=base, env=env, capture_output=True, text=True, check=False)
if probe.returncode == 0:
node = {key: str(value) for key, value in json.loads(probe.stdout).items()}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Hash the TypeScript implementation in generator identity

When the installed node_modules/typescript code is modified or replaced by different bytes carrying the same package version, this probe records the same Node and TypeScript version while source_sha256 explicitly excludes node_modules; the worker nevertheless loads those changed files and may emit different facts under the same generator digest. Fresh evidence beyond the earlier Python-module finding is that the effective TypeScript implementation remains unbound, so hash the resolved package bytes or bind a verified package-integrity digest.

AGENTS.md reference: AGENTS.md:L176-L179

Useful? React with 👍 / 👎.

Comment thread msdmd/collect.py
Comment on lines +302 to +303
listed = run("ls-files", "-z", "--cached", "--others", "--exclude-standard")
staged = run("ls-files", "-z", "--stage")

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Account for sparse index entries before declaring completeness

In a sparse checkout, git ls-files --cached returns skip-worktree entries that are absent from the filesystem, but _discover() only visits paths that exist and never reconciles the remaining visible_files; the omitted tracked files therefore receive no ledger entry or diagnostic, dirty_worktree remains false, and snapshot_complete is incorrectly true for an incomplete fact set at the recorded HEAD. The local git ls-files -h describes --cached as “show cached files in the output”; reconcile those index entries against discovery and mark absent sparse paths unresolved rather than silently dropping them.

AGENTS.md reference: AGENTS.md:L176-L179

Useful? React with 👍 / 👎.

@erinepshovel-code
erinepshovel-code merged commit 38c6433 into main Oct 6, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants