feat(sbom): mark the components whose files an image does not carry

[reyreavman]

Summary

An image that imports part of another inherits the whole SBOM of the image it imports from, so packages whose files never arrived were listed as shipped. Every component an ecosystem can describe is now checked against the files of the built image, and one none of whose files are there carries werf:shipped=false. Nothing is removed from the BOM — this increment records the verdict so it can be read on real builds before anything acts on it.

What

• A component of an ecosystem with a file list (os-pm, pip, npm, luarocks) whose files are in none of the image's layers gets the property werf:shipped=false; one the image ships carries no property at all.
• A component is found by content, not by path: a file imported into another directory, renamed or stripped still holds its component, and only then do the path and file-name tiers run.
• A component nothing is known about is kept: an external image's package, one no ecosystem here describes, one whose only occurrences a scanner wrote.
• Every Go module and crate is kept. They are proven by the binaries the build produced, that set is not established yet, and the rule is given none and fails open on all of them.
• The verdict belongs to the image that made it: a mark inherited from a donor's SBOM is dropped before this image decides, so a package the donor did not carry does not stay marked in an image that does.
• os-pm components are left alone entirely when the image carries a pm index but no file list — a pm that records none makes their files unknowable here, not absent.
• A package the image installed and a later stage deleted, leaving only its manifest, is marked: its own file list names files the image holds none of, and nothing else could show them.
• A file list naming paths the image holds nothing at does not replace occurrences werf recorded before, in a donor's build. Those digests are the only way left to find the bytes where this image put them; a scanner's occurrences are replaced as before.
• The SBOM artifact format version goes from 11 to 12, so every image already built gets its SBOM recomputed with the verdict on the next build. Returning the constant to 11 is the way back.
• A build whose image cannot be read, or whose file lists cannot be parsed, warns and marks nothing; no failure here fails the build.
• The per-component verdict with its reason is logged at debug level; the build log carries only the count.
• VERIFIED: the end-to-end behavior was exercised on a linux host — the e2e suite cannot run on arm64, where the base image of the fixtures is linux/amd64.

Why

The SBOM of a distroless image assembled by import names everything its builder ever installed, because a merged document carries the donor's components whether or not their files came along. On a real project that is the majority of the list: the components are declarations, and nothing tied them to the bytes in the image. The donor already records which files each of its components consists of, and the image already gets indexed by content, so the missing step was the comparison and a place to write its answer.

Marking rather than dropping is deliberate. A false drop in a signed document is expensive and the data to judge the rule does not exist yet; a property is reversible, visible in the same documents, and lets the next increment be decided on what real builds produce. The alternative — gating this behind a flag — was rejected: the flag has to enter the SBOM artifact checksum or it poisons the cache, and once it does, every project builds a second set of SBOM artifacts.

[reyreavman]

Checks run. task format, task build, task lint (0 issues), task test:unit ./pkg/... — green except the two automatic LFS build context specs in pkg/build/image, which fail identically on a clean origin/main worktree here (git-lfs host state, untouched by this diff). e2e test/e2e/sbom labelFilter=pip parallel=1 — 4 Passed | 0 Failed on a linux host, re-run after the review fixes. The suite cannot run on arm64 at all: the fixtures' base is linux/amd64 and the daemon refuses it at the from stage, which the pre-existing pip spec hits the same way.

Mutations. Every new spec was verified by breaking the implementation and watching the right spec fail: the rule never invoked (fails both the build-step spec and the e2e spec), the os-pm fail-open ignored, foreign occurrences treated as "looked and missed", the property appended instead of replaced, nested components not walked, the inherited mark not cleared (3 specs), the donor exception removed (1 spec), the donor exception widened back to every empty fileset (4 specs, the deleted-toolchain one among them).

Review. Two independent reviewers found the same Critical/Major pair before the PR, both fixed: a werf:shipped=false inherited from a donor was never cleared, and an empty occurrence list overwrote the donor's digests. The PR review then showed the second fix was too broad — it also kept a package the image installed and deleted, concept §3.3 — and that the format version had to move here, per card G. Both addressed in follow-up commits; thread above.

Follow-up, already carded. The produced-set — the binaries this image's own build produced, which is what would let Go and Rust be judged instead of failed open — needs layer-to-stage attribution and is its own card. The os-pm file list it all rests on is another; until a pm that writes /var/lib/pm/files.json ships, os-pm components are left unjudged and the mechanism covers pip, npm and luarocks.

Card: 71736203.

[reyreavman]

Code Review Report

Base: main (90a192bf; #453 merged as fcc8e93a), head c929728d
Diff: 10 files, +626/-26 — MarkNotShipped, wired after RecordOccurrences; Filesets.Index exposed; setOccurrences no longer writes an empty list; e2e pip_partial_import.
Context: card F (71736203) — first PR that writes a verdict into users' SBOMs.

Verdict

• Technical: MarkNotShipped itself is right: inherited marks dropped before judging, os-pm gated on OSPMFileListMissing, foreign occurrences ignored, Match invoked over the whole component set, nothing removed. Three mutations fail their specs. But the change to setOccurrences reverses a semantics #453 shipped and documented, and the reversal re-opens the concept §3.3 case the rule exists to catch — probed, reproduced.
• Product: Every SBOM now carries new properties and the occurrence semantics changed, but sbomArtifactFormatVersion stays at "11" — an image whose SBOM was cached by #453 never gets a mark until its stage digest moves. Card G says the bump is what makes F observable; this PR is where it belongs.
• Risk: Medium. A build-only toolchain the image installs and then rm -rfs — the headline scenario of the concept — is now kept unmarked, silently.

DoD Criteria

From card F "Готово, когда" + concept §3.3/§4 + body.

Criteria	Inferred?	Met?	Evidence
component with 0 occurrences matched → werf:shipped=false, stays in BOM	no	✅	mark.go:81-91; specs
rule run after merge and patchers	no	✅	sbom_step.go:210, :272
one debug line per mark with reason	no	✅	mark.go:89-90
no data (old base w/o files.json, non-stapel) → fail-open, build goes on	no	✅	mark.go:63-65; M2 fails spec
fail-open on index Warn and on its error	no	✅	inherited from #453's recordShippedScope
donor's mark dropped before this image decides	yes	✅	mark.go:54; M1 fails 3 specs
own component removed by a later stage → DROP (concept §3.3)	no	❌	occurrences.go:175-177 — Major #1
[] ≠ nil kept apart (concept §4)	no	❌	same change collapses [] into nil for the image's own components
format version bumped (card G: "отдельным коммитом в PR F или следом")	no	❌	sbom_step.go:556 still "11" — Major #2
unit on mark + fail-open; integration; manual build shows property + reasons	no	⚠️	unit ✅; e2e added; body says exercised on linux — not reproduced here
task format/build/lint/test:unit green	no	✅	build OK, lint 0 (incl. e2e compile), 154/154

Issues

Major

1. setOccurrences returning false on an empty fileset erases the "looked for, none found" signal for the image's own components — occurrences.go:175-177. Inline with probe.

2. No format version bump — sbom_step.go:556 (sbomArtifactFormatVersion = "11", unchanged). #453 shipped "11" and attached SBOMs under it. This PR changes two things that make those cached SBOMs stale: every judged component may now carry werf:shipped, and the occurrence semantics changed. With the same version, the cache check at :101 serves the #453 SBOM — no property, old occurrences — for every stage already built, until its digest moves for an unrelated reason. Card G: «Бампнуть на единицу, отдельным коммитом в PR F или следом. Это же путь отката». This is PR F. Bump to "12" here, and the body's "records the verdict so it can be read on real builds" becomes true for existing projects, not only for new stages.

Minor

3. Filesets.Index leaks the whole index to every caller of CollectFilesets — fileset.go:40-45. Justified (one read, two consumers), but the CollectFilesets godoc still describes it as returning filesets; a sentence there, and that Index is never nil from CollectFilesets (the nil-check at mark.go:46 is for a caller-built struct), would stop a future caller from re-reading the image.
4. e2e asserts every donor component unmarked, but checks only requests on app — pip_test.go. app is FROM scratch and inherits python==3.12.12 (os-pm) too; on app, OSPMFileListMissing is unset (no /var/lib/pm), so python is judged by the donor's occurrences and marked. That is correct — python is not in app — and it is the os-pm branch of the same scenario, currently untested end-to-end. Assert it.

Mutation log

Mutation	Result
M1: removeProperty dropped	✅ 3 fail — donor verdict cleared (2), other properties intact
M2: OSPMFileListMissing gate removed	✅ 2 fail
M3: any additionalContext accepted as digest	✅ fails keeps a component whose only occurrences came from elsewhere

Probe — Major #1

Image whose single layer holds tool-1.0.dist-info/RECORD naming tool/main.py, and not the file (installed, then rm -rf):

A  own component, no donor occurrences  → RecordOccurrences writes nothing → Match: no-data → marked=0
B  same, donor recorded /donor/tool/main.py sha256:donorbytes → kept as-is → Match: no-match → marked=1

Case A is concept §3.3 verbatim — «если образ сам ставит пакет и сам же его чистит в более поздней стадии (rm -rf /usr/lib/go после сборки) — это тоже «не доставлено»» — and §4's warning that merging [] into None «превратит §3.3-принятое следствие в ложный fail-open». #453 wrote [] here for exactly this reason (says the files were looked for when a fileset names a component but the image kept none); this PR renames that spec to claims nothing of… and asserts Evidence == nil.

Risks

№	Risk	Type	Likelihood	Severity	Location	Circumstances	Consequences	Recommendation
1	Build toolchain installed and removed in the same image stays unmarked	UX/Product	Likely	Medium	occurrences.go:175	Any multi-stage stapel image that cleans up after itself — the concept's headline case	The mark never appears on the components it was designed for; data for H skews toward "nothing to drop"	Major #1
2	Cached SBOMs never receive the mark	Operational	Likely	Medium	sbom_step.go:556	Every image whose SBOM #453 already attached	F invisible on existing projects until an unrelated rebuild	Major #2

Not verified

• e2e pip_partial_import — compiled only; body says exercised on linux, not reproduced here.
• Real-build log output and property presence — not run.

[reyreavman]

Major. This reverses the semantics #453 shipped. Its body and the previous version of this spec say: present-but-empty = "the component's files were looked for and none are in the image", and that is what Match turns into a drop (Occurrences == [] → no-match, vs nil → no-data).

The motivation in the godoc is a donor-inherited component whose manifest this image also has but whose files moved — keep the donor's digests so content can still find them. That case is real, but the fix is too broad: it also fires when there is no donor record at all, and then "write nothing" means Evidence == nil → no-data → kept. Probe:

layer: tool-1.0.dist-info/RECORD → "tool/main.py"      (file itself deleted by a later stage)
component pkg:pypi/tool@1.0, no prior evidence
→ RecordOccurrences: 0 written   → MarkNotShipped: marked=0

That is concept §3.3 — «образ сам ставит пакет и сам же его чистит … это тоже «не доставлено»» — and the §4 warning about collapsing [] into None. The toolchain-cleanup image is the headline scenario of the whole effort, and after this change it is the one that never gets marked.

Narrow the condition: return false (keep what is there) only when the component already carries werf occurrences (digest or symlink context); when it carries none, write the empty list as before. occurrencesOf in mark.go already distinguishes werf's from foreign ones — lift that check here. Then restore the #453 spec alongside the new one: both shapes are legitimate and both should be pinned.

[reyreavman]

Agreed, and reproduced: with the broad condition the §3.3 image (tool-1.0.dist-info/RECORD naming tool/main.py, file deleted) ended with marked=0.

Narrowed in 64386fc: the empty list is kept back only when the component already carries werf's own occurrences (digest or symlink context); over nothing, or over a scanner's path-only occurrences, it is written as #453 did. occurrencesOf moved into occurrences.go as recordedOccurrences so both sides read the same definition.

Specs: the #453 spec is back (says the files were looked for when a fileset names a component but the image kept none), plus one for the no-donor pypi case, one for the scanner-occurrences case, and a build-step spec for the deleted toolchain itself (marks a component the image installed and then deleted, leaving only its manifest). Mutations: the broad condition back → 4 specs fail (the three RecordOccurrences ones and the build-step one); the exception removed entirely → the donor spec fails.

[reyreavman]

Thanks — both Majors were right, both fixed; the Minors too.

Major 1 — narrowed, see the inline thread: the donor exception now applies only over occurrences werf itself recorded; a package the image installed and deleted gets the empty list and the mark, as #453 and §3.3 intended. The #453 spec is restored alongside the donor one, and a build-step spec pins the deleted-toolchain image itself. (64386fc)

Major 2 — sbomArtifactFormatVersion 11 → 12, own commit as card G asks; the way back is to return it. (28c65e3)

Minor 3 — CollectFilesets godoc now says the index comes back with the filesets and is never nil from there. (2fe429e)

Minor 4 — asserted, but with the opposite expectation, and the run confirms it: on app, python@3.12.12 is inherited and absent, yet not marked. The donor's pm is the current one, which writes no file list, so python never gets occurrences in the donor's BOM; on app it has no fileset of its own, no werf occurrences, and Match sees no-data → kept. That is the os-pm branch of the scenario and the spec says so in its message. It will flip to "marked" once a pm that writes /var/lib/pm/files.json reaches the fixture base — at which point the assertion is what tells us. (2fe429e)

Re-run after the fixes: task lint 0 issues; unit ./pkg/sbom/... ./pkg/build green except the two pre-existing LFS specs; e2e labelFilter=pip parallel=1 on a linux host — 4 Passed | 0 Failed.

[reyreavman]

Code Review — follow-up on 64386fc1…2fe429e4

Delta since c929728d: 3 commits, 7 files, +90/-47.

Verdict

No findings. All four items closed; the Major #1 fix is exactly the narrowing proposed and is proven by the probe that exposed it and by a mutation.

#	Finding	Status
Major 1	own-install-then-delete kept unmarked (§3.3)	✅ occurrences.go:208 — empty list kept only over werf-recorded occurrences; recordedOccurrences moved here and shared with mark.go
Major 2	format version	✅ "12", own commit with the rollback path stated
Minor 3	Index undocumented on CollectFilesets	✅ fileset.go godoc
Minor 4	os-pm branch of the e2e scenario unasserted	✅ pip_test.go — and the assertion corrects my prediction: python is unmarked on app, because the pm that installed it into the donor wrote no files.json, so the donor recorded no occurrences and on app it is no-data. That is the right reading of the "pm too old" fail-open, and pinning it end-to-end is worth more than the marked case would have been.

Verified

• Probe from the previous round, re-run on this head: A (own, deleted) → marked=1 (was 0), B (donor occurrences) → marked=1.
• Mutation — revert :208 to the broad len(contexts) == 0 — fails 3 specs: the restored #453 spec (says the files were looked for when a fileset names a component but the image kept none), the no-donor variant, and the werf-vs-scanner distinction. Both shapes are pinned.
• task build OK, lint 0 (incl. e2e compile), 155/155 shippedscope, recordShippedScope specs pass; pkg/build otherwise as on main.

Not verified

• e2e pip_partial_import — compiled; body says exercised on linux, not reproduced here.

Nothing outstanding on this PR from my side — I would approve. With this, the F increment is in place: #433 → #435 → #442 → #452 → #453 → #459, and the next data point is what real builds mark.