feat(sbom): collect the files each installed component ships

[reyreavman]

Summary

werf can now answer, for an image it has built, which files each installed component ships: shippedscope.CollectFilesets reads the image once and returns one fileset per os-pm package, Python distribution, npm package and Lua rock, each as the set of (path, sha256) the image actually holds, plus whether os-pm packages went without a file list. Nothing calls it yet — it is the component-side input the shipped-scope rule joins against the final image's file index.

What

Boundaries

• A Python distribution owns exactly the paths its RECORD declares, resolved against the directory holding its .dist-info; two distributions writing into one namespace directory (google/auth, google/protobuf) each get only their own files.
• A .py named in RECORD also brings the __pycache__/<stem>.<tag>[.opt-N].pyc compiled from it, whether or not the .py is still in the image; RECORD never lists bytecode an installer did not write. The stem is read from the right, so foo.bar.cpython-312.pyc goes to foo.bar.py, not foo.py.
• Python names are PEP 503-normalised (Typing_Extensions → typing-extensions); the version is the one in the .dist-info directory name.
• An npm package owns the tree under the directory holding its package.json, minus every node_modules subtree inside it (by path segment: lib/my_node_modules/ stays); a second copy nested under a dependent is its own fileset.
• Only a package.json directly under a node_modules directory, under a @scope in one, or under /usr/share/nodejs or /usr/lib/nodejs counts. A fixture or example package.json a package ships inside its own tree yields no fileset and its files stay the package's; an application's own package.json yields nothing; so does one declaring no name or version.
• A Lua rock owns the paths its rock_manifest names: bin, lib, lua sections resolve into <prefix>/bin, <prefix>/lib/lua/<ver>, <prefix>/share/lua/<ver> for the rocks-<ver> tree the rock sits in, every other section resolves inside the rock directory. Names luarocks had to escape (\", \\) are unescaped. A manifest that is not a Lua table yields nothing; the unversioned rocks/ tree of LuaRocks 2 is not read.
• An os-pm package owns the files and symlinks listed under its name in /var/lib/pm/files.json.

Shared rules

• Filesets come out sorted by ecosystem, name, version, the same on every build of the same image.

• A distribution whose RECORD is empty yields an empty fileset, not no fileset.

• Every digest comes from the image, never from the manifest: a file an install hook rewrote after unpacking (update-ca-certificates, compileall) is reported with the content it ships with.

• A path a manifest declares but the image no longer holds is left out; a symlink at a declared path is kept with its target.

• An image with /var/lib/pm/index.json but no files.json yields no os-pm filesets, sets Filesets.OSPMFileListMissing and logs WARNING: /var/lib/pm/index.json is present but /var/lib/pm/files.json is not …; the build goes on. An image with neither file leaves the flag unset. A files.json that does not parse fails the call.

• A manifest the image holds through a hardlink — in the same layer or another — is read like the file itself.

• Components installed some other way — a Python .egg-info, a LuaRocks 2 tree, a Yarn PnP cache — get no fileset; the absence says nothing about whether they shipped.

• UNVERIFIED: the files.json shape ({"<name>": {"version", "files": {path: digest}, "symlinks": {path: target}}}) is proposed by this PR, not yet produced by pm; the matching pm install change settles it.

• IndexImageFiles is unchanged in behavior and cost; the walk gained an internal hook that keeps the bytes of chosen files so manifests cost no second pass over the layers. Only when a hardlinked manifest's target sits in another layer does the hardlink pass read further down than it otherwise would.

Why

The shipped-scope rule decides whether a donor's component survives into the final image by checking whether any of the component's files is still there. That needs the component's file list from the donor, and no existing source gives it: syft's SBOM has no per-file data for these ecosystems, and a directory is the wrong unit for all four — namespace packages share one, npm nests copies inside each other, luarocks spreads one rock over shared trees, and os-pm has no directory at all. Each ecosystem's own install manifest is the only thing that knows, so that is what is read, with the image's content taking precedence over whatever hash the manifest recorded.

A second image pass to read the manifests was rejected: on registry images the pass is dominated by gzip, measured at 51–88% of a full index in #433.

[reyreavman]

Verification

• Mutation, one per spec, all killed (task test:unit paths=./pkg/sbom/shippedscope/... focused per spec):

  • Python: drop the __pycache__ lookup → "credits a compiled file"; resolve against dir instead of its parent → "source it was built from is gone"; ToLower or ReplaceAll("_","-") instead of PEP 503 → "names the distribution the way PEP 503 spells it"; path.Base(record) instead of the declared relative path → "follows a record out of the site directory"; collect taking paths the index lacks → "leaves out records no longer in the image".
  • JavaScript: drop the nested-node_modules exclusion → "leaves a nested package out"; name from the directory instead of package.json → "takes the name the package declares"; drop /usr/share/nodejs/ → "outside node_modules"; accept any package.json → "belongs to no package tree"; drop the name/version guard → "neither name nor version"; drop the symlink sweep → "keeps the symlinks a package tree holds".
  • Lua: pin lib to 5.4 → "lua version of the tree"; put non-root sections straight in the rock dir → "beside itself"; point lua at lib/lua → "install roots its manifest names"; skip the empty-table check → "manifest it cannot read".
  • Review round 1 (f66f377c5d): accept any package.json under node_modules → "leaves a fixture a package ships"; suffix match on node_modules → "merely ends the same way"; drop /usr/lib/nodejs or the @scope step → "outside node_modules"; credit every .pyc to every distribution → "and to no other"; keep the backslash when unescaping → "had to escape"; don't keep a hardlink's bytes (in-layer, cross-layer, or stop reading layers once links settle) → the two hardlinked files.json specs; drop the flag → "and says so"; set it for an image without os-pm → "without os-pm packages".
  • Review round 2 (d32ff126ae): cut the .pyc stem at the first dot → "dotted source name"; ignore .opt-N → same spec; return nil for a kept empty body → "record is empty"; drop the ecosystem or name key of the final sort → "one order". Dropping the version key alone survives: every reader already walks its manifests in sorted order and the sort is stable, so same-name filesets arrive version-ordered by path; dropping both the version key and a reader's sorted walk fails the spec under -until-it-fails within a second. The version key is the explicit statement of what the stable sort was guaranteeing by accident.
  • Review round 3 (b55aeeea34): cut .opt- at its first occurrence, or strip the last dotted component without checking it is opt-N → "dotted source name". Not pinned: Contains("opt-") instead of HasPrefix survives, and is left so — it differs only for an interpreter tag that itself holds opt-, which neither CPython (cpython-312) nor PyPy (pypy310-pp73) writes; a spec for it would assert a name Python never produces.
  • os-pm: collect nothing → "files and symlinks the package manager recorded"; swallow the JSON error, or drop the path from its text → "cannot read"; error on a missing files.json → "older package manager never wrote"; overwrite the index digest with pm's → "digest the image holds".

• Not run: nothing end-to-end — no caller exists yet; the e2e suite joins this with the rule in the next PR.

• Parsed a real rock_manifest (lua-cjson 2.1.0.10-1 installed by luarocks 3 into openresty/openresty:alpine-fat, rocks-5.1): bare and bracketed keys, nested dirs, bin/lib/lua/tests/rockspec sections — 24 paths, all as expected. Confirmed pip's RECORD in python:3.12-slim escapes with ../../../bin/pip3 and lists __pycache__ entries with empty hashes.

Review focus

• file_index.go: the hardlink pass now reads past the point where every link is settled when a captured link's target is still unread (bodyOwed). With capture == nil — every IndexImageFiles caller — the loop stops exactly where it did before.
• lua_fileset.go: the hand-rolled Lua table reader. It reads what luarocks writes (nested tables of string keys and %q string values) and nothing else of the language; a manifest luarocks itself would not write is deliberately out of scope.
• The files.json shape in os_pm_fileset.go is the contract pm install will have to write.

Follow-up

• pm install records /var/lib/pm/files.json in the shape above (pkg/packet/install.go, internal/untar/untar.go in the pm repo).

[reyreavman]

Code Review Report

Base: main (6c93b138; #433 already merged as 42f5d0aa), head f66f377c
Diff: 11 files, +1321/-26 — CollectFilesets plus four ecosystem readers in pkg/sbom/shippedscope; IndexImageFiles gains an internal capture hook. No callers yet.
Context: shipped-scope concept, Kaiten card C (71736129).

Verdict

• Technical: Solid. Eight mutations each fail exactly the spec written for them, including the two subtle ones in the capture hook (bodyOwed, link-name-decides-capture). One real defect found by probe: the .pyc → source mapping cuts the stem at the first dot, so bytecode of a dotted source name is credited to the wrong distribution. Two nits on determinism and a hand-written Lua parser that is fine for what luarocks emits.
• Product: Boundaries match card C and the concept (§3.1, §4). The files.json shape is honestly marked UNVERIFIED in the body — good; the shape has to be pinned with the pm change before F lands, otherwise os-pm stays in fail-open forever.
• Risk: No user impact yet. Deferred: wrong .pyc attribution feeds E a false KEEP for one distribution and a lost proof for another.

DoD Criteria

From card C "Готово, когда" + PR body.

Criteria	Inferred?	Met?	Evidence
os-pm from files.json; missing → fail-open with warning, not build error	no	✅	os_pm_fileset.go:33-42; M8 fails spec :106
Python from RECORD, not directory; namespace split	no	✅	python_fileset.go:35-42; spec :9
PyYAML → yaml (name ≠ directory)	no	✅	spec :49; M5 fails 5 specs
.pyc credited to its .py, with and without the .py present	no	⚠️	spec :49,:93 pass; dotted stem mis-credited — Major #1
JS by last node_modules segment; nested trees split	no	✅	javascript_fileset.go:100-112; M1/M2 each fail 1 spec
/usr/share/nodejs/<name>/ for distribution packages	no	✅	:13, spec :46
Lua from rock_manifest + install roots share/lua, lib/lua, bin	no	✅	lua_fileset.go:36-40; M7 fails spec :68
Digests from the image, never from the manifest	yes	✅	collect() reads index.Files; spec os_pm_fileset_test.go:38
Hashes as sha256:<hex> like FileIndex.Files[].Digest	no	✅	shared File type
Single image pass (no second layer walk for manifests)	yes	✅	indexImageFiles(..., capture), file_index.go:63
IndexImageFiles behaviour unchanged	yes	✅	24 index specs still green; capture=nil path
task format/build/lint/test:unit green	no	✅	build OK, lint 0 issues, 52/52

Issues

Major

1. .pyc stem cut at the first dot credits bytecode to the wrong source — python_fileset.go:99. Inline with reproduction.

Minor

2. Filesets.Filesets order is non-deterministic — fileset.go:65-67, os_pm_fileset.go:50, python_fileset.go:24, javascript_fileset.go:26, lua_fileset.go:17. Every reader ranges over a map. Harmless for the join in E, but the "not shipped …" log lines F emits will shuffle between identical builds and so will any report a consumer writes. slices.Sorted(maps.Keys(...)) as resolveHardlinks already does.
3. Hand-rolled Lua table parser — lua_fileset.go:105-253. 150 lines for a format luarocks writes with one serializer. It is correct for that output (nested tables, %q strings, ["key"] = and bare keys) and the "not a table → skip" guard is right. CODESTYLE says "for complex things use libraries"; I looked and there is no small Lua-table reader in go.mod, so this is acceptable — just noting that a numeric value, a comment, or --[[ ]] would make the whole manifest skip silently. Fine for now; a Debug log on parse failure would make that visible.
4. recordPaths swallows CSV errors — python_fileset.go:69-71. A malformed line is skipped with continue; with LazyQuotes that is rare, but a Debug log would help when a distribution comes up empty.

Mutation log

Mutation	Result
M1: inPackageTree substring match instead of segment	✅ fails …directory whose name merely ends the same way
M2: any package.json under node_modules counts	✅ fails leaves a fixture a package ships…
M3: bodyOwed returns false (early exit ignores owed bodies)	✅ fails …hardlinks to a file another layer wrote
M4: capture decided by target name, not link name	✅ 2 fail — both hardlinked-manifest specs
M5: Python name not normalised	✅ 5 fail
M6: .pyc stem via TrimSuffix(Ext) (last dot)	✅ 2 fail — both compiled-file specs
M7: Lua lib root hardcodes 5.4	✅ fails takes the lua version of the tree…
M8: OSPMFileListMissing never set	✅ fails carries on without the file list…

Tree restored after each; final 52/52.

Risks

№	Risk	Type	Likelihood	Severity	Location	Circumstances	Consequences	Recommendation
1	Bytecode of a dotted source credited to another distribution	Technical	Possible	Medium	python_fileset.go:99	Any site-packages holding name.sub.py beside name.py	In E: one distribution gets a false KEEP(content), the owner loses a proof	Major #1
2	files.json shape diverges from what pm eventually writes	Operational	Possible	Medium	os_pm_fileset.go:19-23	pm install change lands with a different schema	Every os-pm image silently in fail-open (OSPMFileListMissing=true)	Pin the schema with the pm team before F; the body already flags this

Not verified

• Real RECORD/package.json/rock_manifest from live images — fixtures only; no e2e in this PR (card C asks for unit specs only).
• Unscoped task lint / task test:unit — scoped to ./pkg/sbom/shippedscope/...; the package is isolated.
• Caller tracing — not applicable, new exported symbols with no callers.

[reyreavman]

Major. strings.Cut(base, ".") takes everything before the first dot, but a source name may itself hold dots. Probe:

imageOf([]tarEntry{
	regularFile("site/foo.py", "foo"),
	regularFile("site/foo.bar.py", "foobar"),
	regularFile("site/__pycache__/foo.bar.cpython-312.pyc", "compiled foobar"),
	regularFile("site/a-1.0.dist-info/RECORD", "foo.py,,\n"),
	regularFile("site/b-1.0.dist-info/RECORD", "foo.bar.py,,\n"),
})
// a(foo.py) has foo.bar.cpython-312.pyc = true   ← wrong owner
// b(foo.bar.py) has it                  = false  ← real owner loses it

Such names are not importable, but they do exist in site-packages (vendored data, conf.d-style modules, _vendor trees) and compileall compiles them regardless. In E that is a false KEEP(content) for a and a lost proof for b.

The bytecode name is <stem>.<tag>[.opt-N].pyc where <tag> is the interpreter tag (cpython-312, pypy310-pp73). Cut from the right instead: strip .pyc, strip an optional .opt-N, then strip the last dotted component — everything left is the stem. Mutation M6 (TrimSuffix(Ext)) already shows two specs guard the current behaviour; add this probe as the third.

[reyreavman]

Minor. Every reader ranges over a map (manifests, and list in os-pm), so the order of Filesets.Filesets changes run to run. The join in E does not care, but the per-component log lines F will emit (not shipped name@ver: …) and any report a consumer writes will shuffle between two builds of the same image. resolveHardlinks already sorts its keys for exactly this reason — do the same here, or sort once at the end of CollectFilesets by (Ecosystem, Name, Version).

[reyreavman]

Minor. The parser is correct for what luarocks' serializer emits and the fail-closed guard (not a table → no fileset) is the right default. Two notes, neither blocking:

• any token the grammar does not know (a numeric value, a -- comment, a [[long string]]) makes the whole manifest skip silently — a Debug line here would save an hour when a rock comes up empty;
• the .rockspec entry at the top level of the manifest resolves to <rockDir>/<name>.rockspec via the "not an install root" branch, which is right, but the comment at :46-47 reads as if that were the exceptional case; it is the normal one for doc, conf, and the rockspec.

[reyreavman]

Minor. A malformed line is dropped with a bare continue. With LazyQuotes this is rare, but when it happens the distribution silently loses files and E will mark it. A Debug log with the manifest path and line is enough.

[reyreavman]

Thanks — all four addressed in d32ff126ae.

Major #1, .pyc stem. Reproduced with your probe as a spec (credits the bytecode of a dotted source name…, extended with an .opt-2 and a pypy310-pp73 tag). The stem is now taken apart from the right: strip .pyc, strip an optional .opt-N, strip the last dotted component. Mutations "cut at the first dot" and "ignore .opt-" each fail the spec.

Minor #2, order. CollectFilesets sorts by (Ecosystem, Name, Version) at the end, and every reader walks its manifests in sorted order too. New spec lists filesets in one order however the image was read. One honest note in the verification comment: dropping only the version key survives, because the readers' sorted walk plus a stable sort already yields that order by path; the key makes it explicit rather than accidental.

Minor #3, Lua parser. A manifest that is not the table luarocks writes now leaves a Debug line with its path. The comment at the "not an install root" branch now says what it is: the normal case for the rockspec, doc, conf.

Minor #4, RECORD lines. A line the CSV reader rejects leaves a Debug line with the manifest path and the reader's error.

One more thing the order spec surfaced: a kept body was nil for an empty file, so an empty RECORD yielded no fileset at all instead of an empty one. Fixed with its own spec.

On risk #2 (files.json shape) — agreed, it is in the Follow-up as the pm change; nothing in werf can pin it alone.

[reyreavman]

Code Review Report — follow-up on d32ff126

Base: main (6c93b138), head d32ff126
Delta since f66f377c: 8 files, +159/-14.

Verdict

• Technical: All four findings from the previous round are addressed, and the fix for Major #1 is proven by two new specs. One residual in the same function: bytecodeStem cuts .opt- at its first occurrence, so a stem that itself contains .opt- is lost entirely — ok=false, the bytecode is credited to nobody. Same class of defect the commit fixes, one step further in.
• Product: Deterministic order is now a tested contract (fileset_test.go:9); debug lines on unreadable RECORD lines and rock manifests are in. Nothing else outstanding.
• Risk: Residual is narrower than the original (a dotted stem with .opt- in it) and fails closed (file dropped, not mis-credited). Low.

Previous findings

#	Finding	Status
Major 1	.pyc stem cut at first dot	✅ bytecodeStem cuts from the right — but see Minor #1 below
Minor 2	non-deterministic Filesets order	✅ readers iterate sorted keys + SortStableFunc in CollectFilesets; spec fileset_test.go:9
Minor 3	Lua parse failure silent	✅ lua_fileset.go:39; comment at :54-55 reworded
Minor 4	recordPaths swallows CSV errors	✅ python_fileset.go:76

Bonus: readFile now returns a non-nil body for an empty kept file (file_index.go:158), so an empty RECORD yields an empty fileset rather than none — correct and documented.

Issues

Minor

1. strings.Cut(rest, ".opt-") takes the first .opt-, not the last — python_fileset.go:124. Inline with probe.

Verification

task build OK, lint:golangci-lint on the package 0 issues, 55/55 specs. Probe of bytecodeStem on seven names: six correct, one wrong (inline).

Not verified

• Unscoped task lint / task test:unit — scoped to ./pkg/sbom/shippedscope/....

[reyreavman]

Minor. strings.Cut splits at the first .opt-. The optimisation suffix is always the last dotted component before .pyc, so a stem that itself contains .opt- is cut too early:

foo.cpython-312.pyc                  -> "foo"      ok
foo.bar.cpython-312.pyc              -> "foo.bar"  ok
foo.cpython-312.opt-1.pyc            -> "foo"      ok
my.opt-config.cpython-312.pyc        -> "my"       false   ← dropped
my.opt-config.cpython-312.opt-2.pyc  -> "my"       false   ← dropped

In the two bad cases rest becomes "my", cutLast finds no dot, ok=false, and compiledSources skips the file — the bytecode belongs to no distribution. Fails closed, unlike the original defect, but it is the same shape one level down.

Since the structure is <stem>.<tag>[.opt-N], strip from the right in order: .pyc, then the last dotted component if it matches opt-N, then the last dotted component (the tag). cutLast twice with a check on the middle piece does it:

rest := strings.TrimSuffix(name, ".pyc")
if stem, last, ok := cutLast(rest, "."); ok && strings.HasPrefix(last, "opt-") {
	rest = stem
}
stem, _, ok := cutLast(rest, ".")

Please add my.opt-config.cpython-312.pyc to the dotted-stem spec so the two shapes are both pinned.

[reyreavman]

Fixed in b55aeeea34 — you are right, same shape one level in.

.opt-N is the last dotted component before the tag, so it is now taken from the right with a check on the piece, exactly as you wrote it. my.opt-config.cpython-312.pyc and its .opt-2 sibling are in the dotted-stem spec as a third distribution; both mutations — Cut at the first .opt-, and stripping the last component without checking it is opt-N — fail it.

One mutation I left unpinned and want to name rather than quietly skip: Contains("opt-") instead of HasPrefix survives. It differs only for an interpreter tag that itself contains opt-, and neither CPython (cpython-312) nor PyPy (pypy310-pp73) writes one; a spec for it would assert a name Python never produces.

task build OK, unscoped task lint 0 issues, ./pkg/sbom/... green, 55/55 in the package.

[reyreavman]

Code Review — b55aeeea

Verdict: no findings. Closes the last Minor from the previous round.

• bytecodeStem now strips the opt-N component with cutLast + HasPrefix("opt-"), exactly the shape proposed — python_fileset.go:126.
• Both shapes pinned in the dotted-stem spec (my.opt-config.cpython-312.pyc and .opt-2.pyc, HaveLen(3)).
• Mutation (revert to strings.Cut(rest, ".opt-")) fails that spec; 55/55 otherwise.

Nothing outstanding on this PR from my side.