Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 20 additions & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,26 @@
Notable changes to shipped skill behaviour that consumers must act on. Newest
first. Propagation PRs cite the skill-lib commit; this file explains why.

## Unreleased: msdmd collector consumer review (skill-lib #118)
## Unreleased: msdmd Node runs jitless; signal deaths exit 3

Consumers re-syncing past `38c6433` get:

- **Node always runs with `--jitless`** in the TypeScript worker and in the
generator-identity probe. V8's JIT needs writable-then-executable memory,
which systemd `MemoryDenyWriteExecute=yes` forbids; Node then died with a V8
fatal error (SIGTRAP, shell exit 133). Output is byte-identical; extraction
is slower. Sandboxed workers can keep `MemoryDenyWriteExecute=yes`.
- **A Node worker killed by a signal is a missing runtime.** It yields the
`node_runtime_unavailable` diagnostic, so the CLI exits **3** and writes
nothing (opt-out unchanged: `--allow-missing-reader-runtimes`). Before, it
was a `typescript_reader_failed` error that exited 0 without `--strict`, so
TypeScript facts silently went missing.
- **`--print-generator-identity` exits 3 with no output** when its Node probe
is killed by a signal, exits nonzero, cannot be spawned, or prints
unexpected output. Before, it reported Node and TypeScript as `absent` and
exited 0. Node missing from PATH is still recorded as `absent`.
Runners must treat a nonzero exit as "no identity", never as an empty one.


Consumers re-syncing past `9867ab3` must:

Expand Down
3 changes: 2 additions & 1 deletion docs/propagation-checklist.md
Original file line number Diff line number Diff line change
Expand Up @@ -69,7 +69,8 @@ Exit 5 means git could not list the visible files (git missing from PATH, a
corrupt index or broken `.git`) or the root is git-ignored by an enclosing
repository; run in a readable checkout (no opt-out). `--check` reports 5, not
drift. Nothing is written on exit 3, 4 or 5. Runners that fingerprint freshness should use
`python -m msdmd.collect --print-generator-identity`.
`python -m msdmd.collect --print-generator-identity` (exit 3 and no output when
its Node probe is killed by a signal or fails).

Run any target-local checks named in `CLAUDE.md`, `AGENTS.md`, or
`.agents/skills/README.md`.
Expand Down
6 changes: 5 additions & 1 deletion docs/runner-config-guidance.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,11 @@ output and warns); 4 schema helper older than the output
PATH, corrupt index, broken `.git`) or the root is git-ignored by an enclosing
repository (no opt-out; `--check` reports 5, not drift). Exits 3, 4 and 5 write
nothing and are reported together, with precedence 5, then 3, then 4.
Use `--print-generator-identity` as the generator fingerprint.
Use `--print-generator-identity` as the generator fingerprint; it exits 3 with
no identity when its Node probe is killed by a signal or fails. Node always runs
with `--jitless`, so sandboxes with systemd `MemoryDenyWriteExecute=yes` work;
a Node worker killed by a signal is a missing runtime (exit 3), never an empty
TypeScript extraction.

These commands implement the shipped reader matrix, not every possible metadata
standard. A complete compliance verdict still depends on the declared repository
Expand Down
7 changes: 6 additions & 1 deletion msdmd/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,9 @@ npm ci --ignore-scripts --prefix msdmd
A missing runtime is an error diagnostic and marks the reader run
`runtime-unavailable`; it never yields empty success. The CLI exits 3 without
writing unless `--allow-missing-reader-runtimes` is given, which still warns.
Node always runs with `--jitless` (byte-identical output, slower), so the
worker starts under systemd `MemoryDenyWriteExecute=yes`. A Node worker killed
by a signal is `node_runtime_unavailable` and also exits 3.
Other TypeScript worker failures are `typescript_reader_failed` errors.
The universal MSDMD block parsers themselves remain dependency-free.

Expand Down Expand Up @@ -360,7 +363,9 @@ carrying the pinned `commit`, which snapshot identities include.
`--print-generator-identity` prints a digest of every collector file that can
change output (TypeScript worker and lock file included), the Python minor,
reader package, Node and TypeScript versions, and digests of the reader modules
that actually resolve on `sys.path`; add `--json` for the parts. A schema-2
that actually resolve on `sys.path`; add `--json` for the parts. Node missing
from PATH is recorded as `absent`; a Node probe that is killed by a signal,
exits nonzero or prints unexpected output exits 3 with no identity. A schema-2
artifact needs a helper exporting `MSDMD_COLLECTION_HELPER_VERSION` at least the
collector's; an older helper stops the CLI with exit 4 before writing. When
`--out` is outside the root, the helper is located from the root. Propagate the
Expand Down
74 changes: 59 additions & 15 deletions msdmd/collect.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# ratios: loc_comments=1064:134 imports_exports=21:9 calls_definitions=374:33
# ratios: loc_comments=1089:146 imports_exports=22:10 calls_definitions=385:36
# === DOCS ===
# id: msdmd_foundational_contract
# source: msdmd/SKILL.md
Expand Down Expand Up @@ -26,7 +26,7 @@
# network_boundary: none
# user_data_boundary: read
# admin_only: false
# tests: tests/test_collect.py, tests/test_native_collection.py, tests/test_msdmd_consumer_review.py, tests/test_msdmd_review_followup.py
# tests: tests/test_collect.py, tests/test_native_collection.py, tests/test_msdmd_consumer_review.py, tests/test_msdmd_review_followup.py, tests/test_msdmd_node_sandbox.py
# rollout: schema 2 is the default CLI output; --legacy-blocks-only keeps explicit schema 1 compatibility
# rollback: use --legacy-blocks-only while preserving schema incompatibility visibility
# === END MODULE_BUILD ===
Expand All @@ -39,7 +39,9 @@
python -m msdmd.collect --print-generator-identity

Exit status: 1 drift under --check, 2 strict diagnostics, 3 missing native
reader runtime (unless --allow-missing-reader-runtimes), 4 schema helper
reader runtime (unless --allow-missing-reader-runtimes; a Node worker killed
by a signal counts, and --print-generator-identity exits 3 with no output when
its Node probe fails), 4 schema helper
older than the rendered output (MSDMD_COLLECTION_HELPER_VERSION), 5 git
could not list visible files or the root is git-ignored by an enclosing
repository. Exit 3, 4 and 5 problems are all reported before exiting, with
Expand Down Expand Up @@ -71,6 +73,7 @@
from typing import Any, Iterable
from urllib.parse import quote

from msdmd.native_code import NODE_ARGV, node_signal
from msdmd.parsers.universal import marker_for, parse_text
from msdmd.readers import READER_MANIFESTS, _redact_sensitive, read_native, readers_for

Expand All @@ -84,6 +87,7 @@
RUNTIME_UNAVAILABLE_CODES = frozenset({
"missing_docstring_parser",
"typescript_reader_unavailable",
"node_runtime_unavailable",
"reader_dependency_unavailable",
})
# Git could not establish which files are visible, or the root is ignored by
Expand Down Expand Up @@ -952,6 +956,10 @@ def runtime_unavailable(collection: dict[str, Any]) -> list[dict[str, Any]]:
return [item for item in collection.get("diagnostics", []) if item.get("code") in RUNTIME_UNAVAILABLE_CODES]


class GeneratorIdentityError(RuntimeError):
"""The Node runtime probe failed, so no trustworthy identity exists."""


_NODE_PROBE = ("let t='absent';try{t=require('typescript/package.json').version}catch(e){}"
"process.stdout.write(JSON.stringify({node:process.version,typescript:t}))")

Expand Down Expand Up @@ -1047,14 +1055,7 @@ def generator_identity_components(base: Path | None = None) -> dict[str, Any]:
except metadata.PackageNotFoundError:
packages[match.group(1)] = "absent"
modules = _resolved_reader_modules(list(packages), metadata)
node = {"node": "absent", "typescript": "absent"}
env = {key: value for key, value in os.environ.items() if key not in {"NODE_OPTIONS", "NODE_PATH"}}
try:
probe = subprocess.run(["node", "-e", _NODE_PROBE], cwd=base, env=env, capture_output=True, text=True, check=False)
if probe.returncode == 0:
node = {key: str(value) for key, value in json.loads(probe.stdout).items()}
except (OSError, ValueError):
pass
node = _probe_node(base)
return {
"source_sha256": digest.hexdigest(),
"python": f"{sys.version_info.major}.{sys.version_info.minor}",
Expand All @@ -1065,9 +1066,42 @@ def generator_identity_components(base: Path | None = None) -> dict[str, Any]:
}


def _probe_node(base: Path) -> dict[str, str]:
"""Return the Node and TypeScript versions the trusted worker would use.

Node missing from PATH is recorded as ``absent``. A node that starts but is
killed by a signal (V8 refusing a sandbox such as MemoryDenyWriteExecute),
exits nonzero, cannot be spawned, or prints something unparseable raises
:class:`GeneratorIdentityError`: a failed probe is never reported as an
absent runtime.
"""
env = {key: value for key, value in os.environ.items() if key not in {"NODE_OPTIONS", "NODE_PATH"}}
try:
probe = subprocess.run([*NODE_ARGV, "-e", _NODE_PROBE], cwd=base, env=env, capture_output=True, text=True, check=False)
except FileNotFoundError:
return {"node": "absent", "typescript": "absent"}
except OSError as exc:
raise GeneratorIdentityError(f"node probe could not start ({type(exc).__name__})") from exc
killed = node_signal(probe.returncode)
if killed:
raise GeneratorIdentityError(f"node probe was killed by {killed}; Node cannot run in this environment")
if probe.returncode:
raise GeneratorIdentityError(f"node probe exited with status {probe.returncode}")
try:
parsed = json.loads(probe.stdout)
except ValueError:
parsed = None
if not isinstance(parsed, dict) or set(parsed) != {"node", "typescript"} or not all(isinstance(v, str) and v for v in parsed.values()):
raise GeneratorIdentityError("node probe printed unexpected output")
return parsed


def generator_identity(base: Path | None = None) -> str:
"""Return ``sha256:<hex>`` over :func:`generator_identity_components`."""
components = generator_identity_components(base)
return _identity_digest(generator_identity_components(base))


def _identity_digest(components: dict[str, Any]) -> str:
return "sha256:" + _sha256(json.dumps(components, sort_keys=True, separators=(",", ":")))


Expand Down Expand Up @@ -1184,10 +1218,17 @@ def main() -> int:
parser.add_argument("--print-generator-identity", action="store_true", help="print the collector identity (sources plus Python, reader package, Node and TypeScript versions) and exit; add --json for components")
args = parser.parse_args()
if args.print_generator_identity:
try:
components = generator_identity_components()
except GeneratorIdentityError as exc:
# An identity computed without a working runtime would key fresh
# output that the same environment cannot produce.
print(f"msdmd: ERROR: native reader runtime unavailable: {exc}; no generator identity printed.", file=sys.stderr)
return 3
if args.json:
print(json.dumps(generator_identity_components(), indent=2, sort_keys=True))
print(json.dumps(components, indent=2, sort_keys=True))
else:
print(generator_identity())
print(_identity_digest(components))
return 0
if not args.repo:
parser.error("--repo is required")
Expand Down Expand Up @@ -1251,6 +1292,9 @@ def main() -> int:
f"the collection is incomplete. Install with: python -m pip install -r {os.path.join(skill, 'requirements.txt')} "
f"&& npm ci --ignore-scripts --prefix {skill} (Node required). To write incomplete output anyway, "
"pass --allow-missing-reader-runtimes.", file=sys.stderr)
killed = sorted({str(item.get("message")) for item in missing_runtimes if item.get("code") == "node_runtime_unavailable"})
for message in killed:
print(f"msdmd: ERROR: {message}; check the sandbox and resource limits of the process running the collector.", file=sys.stderr)
if args.allow_missing_reader_runtimes:
print("msdmd: WARNING: --allow-missing-reader-runtimes set; writing incomplete output.", file=sys.stderr)
else:
Expand Down Expand Up @@ -1292,4 +1336,4 @@ def main() -> int:

if __name__ == "__main__":
raise SystemExit(main())
# ratios: loc_comments=1064:134 imports_exports=21:9 calls_definitions=374:33
# ratios: loc_comments=1089:146 imports_exports=22:10 calls_definitions=385:36
40 changes: 37 additions & 3 deletions msdmd/native_code.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# ratios: loc_comments=200:16 imports_exports=15:3 calls_definitions=84:3
# ratios: loc_comments=219:26 imports_exports=16:4 calls_definitions=88:4
"""Syntax-aware code readers for the unified collection.

Usage: registry calls read_python/read_typescript with bounded bytes and context.
Expand All @@ -13,6 +13,7 @@
import json
import os
import re
import signal
import subprocess
import sys
from pathlib import Path
Expand Down Expand Up @@ -158,14 +159,47 @@ def location(record: dict) -> dict:
return facts, edges, diagnostics


# V8's JIT needs pages that are writable and then executable, which systemd
# MemoryDenyWriteExecute=yes forbids: node then dies with a V8 fatal error
# (SIGTRAP) before running anything. Interpreter-only output is byte-identical,
# so the trusted worker and the identity probe always run without a JIT.
NODE_ARGV = ('node', '--jitless')


def node_signal(returncode: int) -> str | None:
"""Name the signal that ended a node process, or None for an ordinary exit.

Python reports a signal death as ``-N``; a shell or version-manager shim in
front of node reports it as ``128 + N``.
"""
if returncode < 0:
try:
return signal.Signals(-returncode).name
except ValueError:
return f'signal {-returncode}'
if returncode > 128:
try:
return signal.Signals(returncode - 128).name
except ValueError:
return None
return None


def read_typescript(path: Path, data: bytes, context: dict[str, Any]) -> tuple[list, list, list]:
from msdmd.readers import _fact, _diagnostic, _edge, _subject_address
rid = 'typescript-compiler'
helper = Path(__file__).with_name('typescript-reader.cjs')
env = {key: value for key, value in os.environ.items() if key not in {'NODE_OPTIONS', 'NODE_PATH'}}
# Package resolution is rooted at the trusted helper, never the inspected repo.
result = subprocess.run(['node', str(helper)], input=json.dumps({'path': str(path), 'text': data.decode('utf-8-sig')}),
result = subprocess.run([*NODE_ARGV, str(helper)], input=json.dumps({'path': str(path), 'text': data.decode('utf-8-sig')}),
capture_output=True, text=True, cwd=helper.parent, env=env, check=False)
killed = node_signal(result.returncode)
if killed:
# A signal death means node cannot run here (for example a sandbox that
# V8 refuses), not that this input is malformed: a missing runtime.
return [], [], [_diagnostic(context, reader_id=rid, code='node_runtime_unavailable',
message=f'trusted Node worker was killed by {killed}; Node cannot run in this environment; input not extracted',
status='unsupported')]
if result.returncode:
if "Cannot find module 'typescript'" in result.stderr:
return [], [], [_diagnostic(context, reader_id=rid, code='typescript_reader_unavailable',
Expand Down Expand Up @@ -224,4 +258,4 @@ def read_typescript(path: Path, data: bytes, context: dict[str, Any]) -> tuple[l
for target in targets or [fallback]:
edges.append(_edge(fact['subject']['address'], target, 'exports:' + item['exported_name']))
return facts, edges, diagnostics
# ratios: loc_comments=200:16 imports_exports=15:3 calls_definitions=84:3
# ratios: loc_comments=219:26 imports_exports=16:4 calls_definitions=88:4
Loading
Loading