From 381777d0d5555a9ca007ec8b5d46b950c00bab7b Mon Sep 17 00:00:00 2001 From: luoye-cpu <3412829159@qq.com> Date: Wed, 11 Nov 2026 04:43:52 +0800 Subject: [PATCH 1/5] fix(docker): give the production images the Debian zimg record Issue #3242: the production licence manifest named upstream zimg 3.0.6 while every production image ships Debian's libzimg2 3.0.5+ds1-1+b2, so the generated notice named a version and a source the image does not carry, and nothing in those images answered for the library the builder lines copy out of /usr/lib/*-linux-gnu*. The zimg entry of production-cli-image now reads the package exactly as the tester entry has since #3241: a builder that copies libzimg.so* also stores /usr/share/doc/libzimg2/copyright and a dpkg-query SOURCE line under /dist/doc/libzimg2, and the runtime stage installs both at /usr/share/doc/libzimg2; the eight production records that take the component with "from" follow it (ADR-1517). Scope is the maintainer's note of 2026-11-10, which named six builder lines rather than the four the issue text listed: Dockerfile.production, Dockerfile.node, Dockerfile.production-gpu (twice), Dockerfile.go-server and Dockerfile.controller. The go-server and node images had been missed because they inherit the component too, and a seventh copy site is the shared scripts/ci/build-oneapi-libvmaf.sh that stages /dist/lib for the two oneAPI images. test_copied_debian_libraries_carry_their_source becomes one table over every stage that copies a Debian library (the tester image's four cp -a sites and the production cp -d ones) with a second table over the thirteen published runtime stages that must install the record, resolved through each FROM chain: on origin/master six of the ten copy sites and nine of the thirteen runtime stages fail, on this branch all of them pass. The macOS bundle, the Windows zips and release-native link zimg from the 3.0.6 archive themselves and keep their entries, as the issue asks. Signed-off-by: luoye-cpu <3412829159@qq.com> --- Dockerfile.go-server | 7 +- .../fixed/production-zimg-debian-record.md | 12 ++ docker/AGENTS.md | 21 +++ docker/Dockerfile.controller | 4 + docker/Dockerfile.node | 11 +- docker/Dockerfile.production | 5 + docker/Dockerfile.production-gpu | 9 + docs/state.md | 1 + scripts/ci/build-oneapi-libvmaf.sh | 6 + tools/rc1-tester/image/licensing.json | 17 +- .../tests/test_dockerfile_runtime_libs.py | 155 ++++++++++++++++-- .../tests/test_licensing_production.py | 71 ++++++++ 12 files changed, 296 insertions(+), 23 deletions(-) create mode 100644 changelog.d/fixed/production-zimg-debian-record.md diff --git a/Dockerfile.go-server b/Dockerfile.go-server index b3bbc122f2..2a869e39eb 100644 --- a/Dockerfile.go-server +++ b/Dockerfile.go-server @@ -100,6 +100,9 @@ RUN mkdir -p /dist/lib/pkgconfig /dist/bin /dist/model \ && find /build/src -maxdepth 1 \( -name 'libvmaf.so*' -o -name 'libvmafx.so*' \) \ \( -type f -o -type l \) -exec cp -a {} /dist/lib/ \; \ && cp -d /usr/lib/*-linux-gnu*/libzimg.so* /dist/lib/ \ + && mkdir -p /dist/doc/libzimg2 \ + && cp /usr/share/doc/libzimg2/copyright /dist/doc/libzimg2/copyright \ + && dpkg-query -W -f='${source:Package}=${source:Version}\n' libzimg2 > /dist/doc/libzimg2/SOURCE \ && cp /build/meson-private/libvmaf.pc /dist/lib/pkgconfig/libvmaf.pc \ && cp /build/meson-private/libvmafx.pc /dist/lib/pkgconfig/libvmafx.pc \ && cp /build/tools/vmaf /dist/bin/ \ @@ -181,8 +184,10 @@ LABEL org.opencontainers.image.source="https://github.com/VMAFx/vmafx" LABEL org.opencontainers.image.licenses="EUPL-1.2 AND BSD-2-Clause-Patent AND BSD-3-Clause AND BSD-2-Clause AND ISC AND MIT AND Unlicense AND LicenseRef-LIVE-BRISQUE AND WTFPL" LABEL org.opencontainers.image.vendor="Lusoris" -# Copy libvmaf.so* and the vmaf CLI from the vmaf-builder stage. +# Copy libvmaf.so* and the vmaf CLI from the vmaf-builder stage, with the Debian +# record of the libzimg2 the zimg component reads (issue #3242). COPY --from=vmaf-builder /dist/lib/ /usr/local/lib/ +COPY --from=vmaf-builder /dist/doc/libzimg2 /usr/share/doc/libzimg2 COPY --from=vmaf-builder /dist/bin/ /usr/local/bin/ COPY --from=vmaf-builder /dist/model/ /usr/local/share/vmafx/model/ diff --git a/changelog.d/fixed/production-zimg-debian-record.md b/changelog.d/fixed/production-zimg-debian-record.md new file mode 100644 index 0000000000..7a85b01d3a --- /dev/null +++ b/changelog.d/fixed/production-zimg-debian-record.md @@ -0,0 +1,12 @@ +- **The production images credit the libzimg they ship.** The `vmaf` command now links zimg by + default ([ADR-3396](../docs/adr/3396-zimg-on-by-default.md)), so every production image copies + `libzimg.so*` out of Debian's `libzimg2` package — but the `zimg` entry of + `tools/rc1-tester/image/licensing.json` still described upstream zimg 3.0.6 and claimed no file in + the image, so the six builders that copy that library had no Debian record for it (issue #3242). + The `zimg` entry of `production-cli-image` now reads the package's Debian record exactly as the + tester entry already does: each builder stages `usr/share/doc/libzimg2/copyright` plus a + `dpkg-query` `SOURCE` line, the runtime stages install both at `/usr/share/doc/libzimg2`, and the + eight production records that inherit the component follow. The image's source offer names + `libzimg2`'s Debian source package instead of upstream zimg 3.0.6, and + `tools/rc1-tester/tests/test_dockerfile_runtime_libs.py` holds all six builders and the nine + published runtime stages to it. diff --git a/docker/AGENTS.md b/docker/AGENTS.md index a14b9c6403..b59efac7ec 100644 --- a/docker/AGENTS.md +++ b/docker/AGENTS.md @@ -143,6 +143,27 @@ stage Meson's generated `meson-private/libvmaf.pc`; never synthesize it from `scripts/ci/test_e2e_runtime_contract.py` pins these invariants for both Dockerfiles. +## The Debian zimg record every copying builder stages (issue #3242) + +`libzimg.so*` is **Debian's** `libzimg2`, never upstream's 3.0.6 archive, in every +image that copies it out of `/usr/lib/*-linux-gnu*` — which is what makes +`zimg` of `production-cli-image` read +`debian_source_file: usr/share/doc/libzimg2/SOURCE` and take the package's +copyright file as its licence text, as the tester entry already did. Six builder +lines copy it (`docker/Dockerfile.production:86`, `Dockerfile.production-gpu:130` +and `:182`, `docker/Dockerfile.controller:98`, `Dockerfile.go-server:102`, +`docker/Dockerfile.node:184`); a seventh site is the shared +`scripts/ci/build-oneapi-libvmaf.sh` that the two oneAPI images run. **All of them, +and any new copying builder, stage `copyright` + a `dpkg-query` `SOURCE` line +under `/dist/doc/libzimg2`; every runtime stage that publishes `libzimg.so*` +installs that directory at `/usr/share/doc/libzimg2`.** Omitting it fails the +image's `licensing.py sources` (the source offer cannot name a Debian package it +has no record of) and leaves the notice naming a version the image does not +ship. `tools/rc1-tester/tests/test_dockerfile_runtime_libs.py` holds the six +builders, the nine published runtime stages (following each `FROM` chain) and +the shared script to it. The macOS bundle, the Windows zips and `release-native` +build zimg from source themselves and keep the 3.0.6 archive entry. + ## Fedora optional SYCL repository `dev/fedora-40.Dockerfile` writes seven literal oneAPI repository lines with diff --git a/docker/Dockerfile.controller b/docker/Dockerfile.controller index 6d078c15de..57374782c9 100644 --- a/docker/Dockerfile.controller +++ b/docker/Dockerfile.controller @@ -96,6 +96,9 @@ RUN mkdir -p /dist/lib/pkgconfig /dist/bin /dist/model \ && find /build/src -maxdepth 1 \( -name 'libvmaf.so*' -o -name 'libvmafx.so*' \) \ \( -type f -o -type l \) -exec cp -a {} /dist/lib/ \; \ && cp -d /usr/lib/*-linux-gnu*/libzimg.so* /dist/lib/ \ + && mkdir -p /dist/doc/libzimg2 \ + && cp /usr/share/doc/libzimg2/copyright /dist/doc/libzimg2/copyright \ + && dpkg-query -W -f='${source:Package}=${source:Version}\n' libzimg2 > /dist/doc/libzimg2/SOURCE \ && cp /build/meson-private/libvmaf.pc /dist/lib/pkgconfig/libvmaf.pc \ && cp /build/meson-private/libvmafx.pc /dist/lib/pkgconfig/libvmafx.pc \ && cp /build/tools/vmaf /dist/bin/ \ @@ -180,6 +183,7 @@ LABEL org.opencontainers.image.vendor="Lusoris" # Copy libvmaf.so* and the vmaf CLI from the vmaf-builder stage. COPY --from=vmaf-builder /dist/lib/ /usr/local/lib/ +COPY --from=vmaf-builder /dist/doc/libzimg2 /usr/share/doc/libzimg2 COPY --from=vmaf-builder /dist/bin/ /usr/local/bin/ COPY --from=vmaf-builder /dist/model/ /usr/local/share/vmafx/model/ diff --git a/docker/Dockerfile.node b/docker/Dockerfile.node index 122f686415..97e93f11a1 100644 --- a/docker/Dockerfile.node +++ b/docker/Dockerfile.node @@ -182,6 +182,9 @@ RUN mkdir -p /dist/lib/pkgconfig /dist/bin /dist/model \ && find /build/src -maxdepth 1 \( -name 'libvmaf.so*' -o -name 'libvmafx.so*' \) \ \( -type f -o -type l \) -exec cp -a {} /dist/lib/ \; \ && cp -d /usr/lib/*-linux-gnu*/libzimg.so* /dist/lib/ \ + && mkdir -p /dist/doc/libzimg2 \ + && cp /usr/share/doc/libzimg2/copyright /dist/doc/libzimg2/copyright \ + && dpkg-query -W -f='${source:Package}=${source:Version}\n' libzimg2 > /dist/doc/libzimg2/SOURCE \ && cp /build/meson-private/libvmaf.pc /dist/lib/pkgconfig/libvmaf.pc \ && cp /build/meson-private/libvmafx.pc /dist/lib/pkgconfig/libvmafx.pc \ && cp /build/tools/vmaf /dist/bin/ \ @@ -471,8 +474,10 @@ ENV VMAFX_MODEL_DIR="/usr/local/share/vmafx/model" # Canonical ffmpeg path consumed by the node binary and its encoder probe. ENV VMAFX_FFMPEG_BIN="/usr/local/bin/ffmpeg" -# Copy libvmaf.so* and models +# Copy libvmaf.so* and models, with the Debian record of the libzimg2 the zimg +# component reads (issue #3242). COPY --from=vmaf-builder /dist/lib/ /usr/local/lib/ +COPY --from=vmaf-builder /dist/doc/libzimg2 /usr/share/doc/libzimg2 COPY --from=vmaf-builder /dist/model/ /usr/local/share/vmafx/model/ COPY --from=vmaf-builder /dist/bin/vmaf /usr/local/bin/vmaf # pkg/storage resolves s3://, gs://, rclone:// and remote:path inputs by @@ -783,7 +788,9 @@ FROM node-assembled AS node-sycl-assembled # The Intel GPU user-mode stack and its dpkg records (ADR-3371) COPY --from=intel-gpu-runtime-libs /out/ / -# The oneAPI SYCL build of libvmaf, CLI, and Intel SYCL runtime files (ADR-3371) +# The oneAPI SYCL build of libvmaf, CLI, and Intel SYCL runtime files (ADR-3371). +# The libzimg2 Debian record needs no copy here: node-sycl-assembled inherits +# runtime-base, which installs it (issue #3242). COPY --from=builder-oneapi2026 /dist/lib/ /usr/local/lib/ COPY --from=builder-oneapi2026 /dist/bin/vmaf /usr/local/bin/vmaf COPY --from=builder-oneapi2026 /stage/lib/intel/ /usr/local/lib/intel/ diff --git a/docker/Dockerfile.production b/docker/Dockerfile.production index 91cf5babb5..6763553661 100644 --- a/docker/Dockerfile.production +++ b/docker/Dockerfile.production @@ -84,6 +84,9 @@ RUN mkdir -p /dist/lib /dist/bin /dist/model \ && find /build/src -maxdepth 1 \( -name 'libvmaf.so*' -o -name 'libvmafx.so*' \) -not -type d \ -exec cp {} /dist/lib/ \; \ && cp -d /usr/lib/*-linux-gnu*/libzimg.so* /dist/lib/ \ + && mkdir -p /dist/doc/libzimg2 \ + && cp /usr/share/doc/libzimg2/copyright /dist/doc/libzimg2/copyright \ + && dpkg-query -W -f='${source:Package}=${source:Version}\n' libzimg2 > /dist/doc/libzimg2/SOURCE \ && cp /build/tools/vmaf /dist/bin/ \ && cp -r model/. /dist/model/ @@ -153,6 +156,7 @@ LABEL org.opencontainers.image.vendor="Lusoris" # Copy libvmaf.so* into /usr/local/lib and set up symlinks COPY --from=builder /dist/lib/ /usr/local/lib/ +COPY --from=builder /dist/doc/libzimg2 /usr/share/doc/libzimg2 COPY --from=builder /dist/model/ /usr/local/share/vmafx/model/ ENV LD_LIBRARY_PATH="/usr/local/lib" @@ -253,6 +257,7 @@ LABEL org.opencontainers.image.licenses="EUPL-1.2 AND BSD-2-Clause-Patent AND BS LABEL org.opencontainers.image.vendor="Lusoris" COPY --from=builder /dist/lib/ /usr/local/lib/ +COPY --from=builder /dist/doc/libzimg2 /usr/share/doc/libzimg2 COPY --from=builder /dist/model/ /usr/local/share/vmafx/model/ COPY --from=builder /dist/bin/vmaf /usr/local/bin/vmaf diff --git a/docker/Dockerfile.production-gpu b/docker/Dockerfile.production-gpu index 1e0f37d410..fd41c7de22 100644 --- a/docker/Dockerfile.production-gpu +++ b/docker/Dockerfile.production-gpu @@ -128,6 +128,9 @@ RUN meson setup build core --buildtype=release --strip -Db_lto=false \ RUN mkdir -p /dist/lib /dist/bin /dist/model /stage/licenses/nvidia /licence-inputs \ && find build/src -maxdepth 1 \( -name 'libvmaf.so*' -o -name 'libvmafx.so*' \) -not -type d -exec cp -a {} /dist/lib/ \; \ && cp -d /usr/lib/*-linux-gnu*/libzimg.so* /dist/lib/ \ + && mkdir -p /dist/doc/libzimg2 \ + && cp /usr/share/doc/libzimg2/copyright /dist/doc/libzimg2/copyright \ + && dpkg-query -W -f='${source:Package}=${source:Version}\n' libzimg2 > /dist/doc/libzimg2/SOURCE \ && cp build/tools/vmaf /dist/bin/ \ && cp -r model/. /dist/model/ \ && for binary in /dist/bin/vmaf /dist/lib/libvmaf.so.* /dist/lib/libvmafx.so.*; do \ @@ -180,6 +183,9 @@ RUN gfx_targets="$(jq -r '.dist_amdgpu_targets' /opt/rocm/share/therock/dist_inf RUN mkdir -p /dist/lib /dist/bin /dist/model /licence-inputs \ && find build/src -maxdepth 1 \( -name 'libvmaf.so*' -o -name 'libvmafx.so*' \) -not -type d -exec cp -a {} /dist/lib/ \; \ && cp -d /usr/lib/*-linux-gnu*/libzimg.so* /dist/lib/ \ + && mkdir -p /dist/doc/libzimg2 \ + && cp /usr/share/doc/libzimg2/copyright /dist/doc/libzimg2/copyright \ + && dpkg-query -W -f='${source:Package}=${source:Version}\n' libzimg2 > /dist/doc/libzimg2/SOURCE \ && cp build/tools/vmaf /dist/bin/ \ && cp -r model/. /dist/model/ \ && python3 tools/rc1-tester/image/prepare_build.py rocm-runtime \ @@ -277,6 +283,7 @@ ENV LD_LIBRARY_PATH="/usr/local/lib:/usr/lib/wsl/lib" \ NVIDIA_DRIVER_CAPABILITIES=compute,utility \ VMAF_MODEL_PATH="/usr/local/share/vmafx/model" COPY --from=builder-cuda13 /dist/lib/ /usr/local/lib/ +COPY --from=builder-cuda13 /dist/doc/libzimg2 /usr/share/doc/libzimg2 COPY --from=builder-cuda13 /dist/bin/ /usr/local/bin/ COPY --from=builder-cuda13 /dist/model/ /usr/local/share/vmafx/model/ COPY --from=builder-cuda13 /stage/licenses/ /usr/local/share/vmafx/licenses/ @@ -385,6 +392,7 @@ LABEL org.opencontainers.image.vendor="Lusoris" ENV LD_LIBRARY_PATH="/usr/local/lib:/usr/local/lib/rocm" \ VMAF_MODEL_PATH="/usr/local/share/vmafx/model" COPY --from=builder-rocm10 /dist/lib/ /usr/local/lib/ +COPY --from=builder-rocm10 /dist/doc/libzimg2 /usr/share/doc/libzimg2 COPY --from=builder-rocm10 /dist/bin/ /usr/local/bin/ COPY --from=builder-rocm10 /dist/model/ /usr/local/share/vmafx/model/ COPY --from=builder-rocm10 /stage/lib/rocm/ /usr/local/lib/rocm/ @@ -521,6 +529,7 @@ RUN --mount=type=secret,id=github_token,env=GITHUB_TOKEN,required=false \ ENV LD_LIBRARY_PATH="/usr/local/lib:/usr/local/lib/intel:/usr/lib/wsl/lib" \ VMAF_MODEL_PATH="/usr/local/share/vmafx/model" COPY --from=builder-oneapi2026 /dist/lib/ /usr/local/lib/ +COPY --from=builder-oneapi2026 /dist/doc/libzimg2 /usr/share/doc/libzimg2 COPY --from=builder-oneapi2026 /dist/bin/ /usr/local/bin/ COPY --from=builder-oneapi2026 /dist/model/ /usr/local/share/vmafx/model/ COPY --from=builder-oneapi2026 /stage/lib/intel/ /usr/local/lib/intel/ diff --git a/docs/state.md b/docs/state.md index 441a6afeaf..c24e3be70c 100644 --- a/docs/state.md +++ b/docs/state.md @@ -1176,6 +1176,7 @@ landed fix yet._ ## Recently closed +| **T-PRODUCTION-ZIMG-DEBIAN-RECORD-2026-11-10** — the production licence manifest named upstream zimg 3.0.6 while every production image ships Debian's `libzimg2` 3.0.5+ds1-1+b2, so the generated notice named the wrong version and source, and six builders copied `libzimg.so*` out of `/usr/lib/*-linux-gnu*` with nothing in the image the `zimg` component could point at | **FIXED on `fix/issue-3242-zimg-debian-source` (2026-11-11).** Each of the six builder lines the maintainer named (`docker/Dockerfile.production`, `docker/Dockerfile.production-gpu` twice, `docker/Dockerfile.controller`, `Dockerfile.go-server`, `docker/Dockerfile.node`) and the shared `scripts/ci/build-oneapi-libvmaf.sh` behind the two oneAPI images stores `/usr/share/doc/libzimg2/copyright` plus a `dpkg-query` `SOURCE` record under `/dist/doc/libzimg2`; the nine published runtime stages install both at `/usr/share/doc/libzimg2`; the `zimg` entry of `production-cli-image` reads them through `debian_source_file` and an `artifact` text, and the eight production records that inherit it follow (ADR-1517). `tools/rc1-tester/tests/test_dockerfile_runtime_libs.py` holds all six builders, the nine runtime stages and the shared script to it; the macOS, Windows and `release-native` entries stay on the 3.0.6 archive, which each of those links itself. | #3242 | 2026-11-10 | closed | | **T-CI-PRECOMMIT-DIFF-SCOPE-HOOK-INPUT-2026-11-01** — #3237 added the hook `check-fork-python-subprocess` and its script; the Pre-Commit job ran on the pull request's diff only (`--from-ref`), so the new hook never saw the files it flags elsewhere in the tree, the pull request passed, and master `a3ea48309` went red on the whole-tree run of the push | **FIXED on `fix/precommit-scope-config-change` (#3253).** `scripts/ci/precommit_scope.py` runs the whole tree when a pull request changes a hook input: `.pre-commit-config.yaml`, any file under `scripts/ci/` (CI authority) or `scripts/lib/`, any path a hook's command names (local and remote `entry` overrides, `args`, commands inside `bash -c`, `python -m` modules, deleted scripts included) or a module those scripts import; an unreadable file list, a git timeout or an entry it cannot split also select the whole tree. Invariant page: `scripts/ci/AGENTS.d/precommit-scope.md`. | `scripts/ci/tests/test_precommit_scope.py` (21 tests); against the diff-only selector of `634e636a7` 7 of them fail, the #3237 file list among them | `fix/precommit-scope-config-change` | | **T-HOOKS-WINDOWS-POSIX-FIXTURES-2026-09-30** — four local fixture hooks fail on a Windows host | **FIXED in #3282 (part of #2438).** `test_lint_configured.py` normalises the public-model path via `as_posix()` and skips `ConfiguredLintTests` off POSIX with a stated reason; `test_write_compile_commands.py` skips `CompileCommandsExportTest` off POSIX with a stated reason; `test_tidy_ratchet.py` skips off POSIX with a stated reason; `test-sync-pelorus-interop.sh` skips on Windows with a stated reason. `scripts/ci/check-source-adr-citations.py --write` writes the registry with `newline="\n"`, guards against concurrent writes, cleans up temporary files on failure, and `test_check_source_adr_citations.py` verifies canonical byte rewriting. | unit test suite | #3282 | 2026-11-03 | closed | | **T-CI-WARNINGS-NON-MSVC-LEGS-2026-10-07** — remaining: Dev Container Build compiles third-party sources (vpl-gpu-rt -Wstringop-overflow, FFmpeg) and prints gcc's LTO "serial compilation" note | **CLOSED in #3282 (part of #2438).** Recorded the dev container image build as outside the compiler warning gate in `docs/development/ci.md` ("Builds outside the gate") per ADR-2170 and ADR-2828: external container dependencies (`vpl-gpu-rt`, `FFmpeg`) are compiled into temporary container layers during OCI image assembly rather than from repository sources; in-tree VMAFx sources are compiled and gated with `-Dwerror=true` across all CI compiler matrix legs. The Windows `icx-cl` lane was previously resolved and gated with `-Dwerror=true` in PR #2650 (ADR-2828). | `docs/development/ci.md` | #3282 | 2026-11-03 | closed | diff --git a/scripts/ci/build-oneapi-libvmaf.sh b/scripts/ci/build-oneapi-libvmaf.sh index b43a527473..6ee1e83909 100755 --- a/scripts/ci/build-oneapi-libvmaf.sh +++ b/scripts/ci/build-oneapi-libvmaf.sh @@ -40,6 +40,12 @@ nice -n 10 ninja -C "${build_dir}" -j"${jobs}" mkdir -p "${dist_dir}/lib" "${dist_dir}/bin" "${dist_dir}/model" "${licence_inputs_dir}" find "${build_dir}/src" -maxdepth 1 \( -name 'libvmaf.so*' -o -name 'libvmafx.so*' \) -not -type d -exec cp -a {} "${dist_dir}/lib/" \; cp -d /usr/lib/*-linux-gnu*/libzimg.so* "${dist_dir}/lib/" +# The Debian record of the libzimg2 those images ship (issue #3242): the zimg +# component of licensing.json reads usr/share/doc/libzimg2/SOURCE and takes its +# copyright file as the licence text, as the tester images already do. +mkdir -p "${dist_dir}/doc/libzimg2" +cp /usr/share/doc/libzimg2/copyright "${dist_dir}/doc/libzimg2/copyright" +dpkg-query -W -f='${source:Package}=${source:Version}\n' libzimg2 > "${dist_dir}/doc/libzimg2/SOURCE" cp "${build_dir}/tools/vmaf" "${dist_dir}/bin/" cp -r model/. "${dist_dir}/model/" diff --git a/tools/rc1-tester/image/licensing.json b/tools/rc1-tester/image/licensing.json index 59d98f6408..234b0bc7bf 100644 --- a/tools/rc1-tester/image/licensing.json +++ b/tools/rc1-tester/image/licensing.json @@ -2748,22 +2748,23 @@ { "id": "zimg", "kind": "fixed", - "name": "zimg image scaling and colours-pace conversion library (libzimg.so.2)", + "name": "zimg image scaling and colours-pace conversion library (Debian libzimg2)", "paths": [ - "usr/local/lib/libzimg.so*" + "usr/local/lib/libzimg.so*", + "usr/share/doc/libzimg2/**" ], "licence": "WTFPL", "copyright": [ - "Copyright (c) 2014-2024, sekrit-twc" + "Copyright (c) 2014-2020, sekrit-twc" ], - "source": "https://github.com/sekrit-twc/zimg.git at commit f819b14e8f39d1282400b0d9543e8ef73c1b2bbd, tag release-3.0.6", + "debian_source_file": "usr/share/doc/libzimg2/SOURCE", "texts": [ { - "name": "WTFPL.txt", - "fetched": "zimg-3.0.6-COPYING", - "label": "zimg 3.0.6 licence text (WTFPL v2)" + "artifact": "usr/share/doc/libzimg2/copyright", + "label": "Debian copyright file of libzimg2" } - ] + ], + "source": "the Debian source package named in usr/share/doc/libzimg2/SOURCE, in the source image" }, { "id": "notices", diff --git a/tools/rc1-tester/tests/test_dockerfile_runtime_libs.py b/tools/rc1-tester/tests/test_dockerfile_runtime_libs.py index eb6f888c56..3f2f67fad8 100644 --- a/tools/rc1-tester/tests/test_dockerfile_runtime_libs.py +++ b/tools/rc1-tester/tests/test_dockerfile_runtime_libs.py @@ -1,13 +1,20 @@ # Copyright 2026 Lusoris # SPDX-License-Identifier: EUPL-1.2 -"""Every GPU runtime stage of the tester image finds the libraries its build stage stages. +"""Every image stage that copies a Debian library ships that package's record. The GPU build stages of `docker/Dockerfile.tester` copy the shared libraries the `vmaf` binary links (libzimg since zimg became a default dependency) into `/opt/vmafx/lib`. The matching runtime stage must copy that directory and put it on `LD_LIBRARY_PATH`, or the stage's `ldd` check stops the image build with `libzimg.so.2 => not found`. The GPU images build only on a push to master, so -this test is the pull-request coverage of the contract. +this test is the pull-request coverage of that contract. + +The same stages copy `libzimg.so*` out of Debian's `libzimg2` package, so they +also carry the record the licence manifest reads (issue #3241 for the tester +images, issue #3242 for the production ones): the package's copyright file plus +a `dpkg-query` `SOURCE` line. One table names every copying builder and its +published runtime stages, so a new copy site cannot land without the record +(HISS-19). """ from __future__ import annotations @@ -64,11 +71,51 @@ def test_the_stage_reader_refuses_an_unknown_stage() -> None: DEBIAN_COPIES = {"libzimg": "libzimg2", "libgomp": "libgomp1"} - -def _uncredited_debian_copies(stage: str) -> list[str]: +# Every image stage that copies a Debian library into a tree a published image is +# built from, as (file, stage, copy verb). The verb is `cp -a` in the tester +# image's build stages (whole SONAME chain followed) and `cp -d` in the +# production builders (the symlinks are copied as symlinks). Issue #3242's scope +# note of 2026-11-10 named six production builder lines; `Dockerfile.go-server` +# and `docker/Dockerfile.node` are the two the issue text had missed, and the +# node image inherits `production-cli-image`'s zimg component like the rest. +COPY_SITES = ( + ("docker/Dockerfile.tester", "python-venv", "cp -a"), + ("docker/Dockerfile.tester", "sycl-build", "cp -a"), + ("docker/Dockerfile.tester", "cuda-build", "cp -a"), + ("docker/Dockerfile.tester", "hip-build", "cp -a"), + ("docker/Dockerfile.production", "builder", "cp -d"), + ("docker/Dockerfile.controller", "vmaf-builder", "cp -d"), + ("docker/Dockerfile.production-gpu", "builder-cuda13", "cp -d"), + ("docker/Dockerfile.production-gpu", "builder-rocm10", "cp -d"), + ("Dockerfile.go-server", "vmaf-builder", "cp -d"), + ("docker/Dockerfile.node", "vmaf-builder", "cp -d"), +) + +# The published runtime stages that install the record, each with the base of its +# FROM chain: the record may come from the stage itself or from a stage it is +# FROM (node-sycl-assembled is FROM node-assembled, which is FROM runtime-base). +RUNTIME_STAGES = ( + ("docker/Dockerfile.tester", "runtime"), + ("docker/Dockerfile.tester", "sycl-runtime"), + ("docker/Dockerfile.tester", "cuda-runtime"), + ("docker/Dockerfile.tester", "hip-runtime"), + ("docker/Dockerfile.production", "cli-assembled"), + ("docker/Dockerfile.production", "server-assembled"), + ("docker/Dockerfile.controller", "controller-assembled"), + ("docker/Dockerfile.production-gpu", "cuda13-assembled"), + ("docker/Dockerfile.production-gpu", "rocm10-assembled"), + ("docker/Dockerfile.production-gpu", "oneapi2026-assembled"), + ("Dockerfile.go-server", "go-server-assembled"), + ("docker/Dockerfile.node", "node-assembled"), + ("docker/Dockerfile.node", "node-sycl-assembled"), +) + + +def _copy_gaps(stage: str, verb: str) -> list[str]: """Debian packages whose library the stage copies without recording its SOURCE and copyright.""" + pattern = rf"{re.escape(verb)} /usr/lib/\*-linux-gnu\*?/(lib\w+)\.so" missing = [] - for lib in sorted(set(re.findall(r"cp -a /usr/lib/\*-linux-gnu\*?/(lib\w+)\.so", stage))): + for lib in sorted(set(re.findall(pattern, stage))): package = DEBIAN_COPIES.get(lib, lib) source = rf"dpkg-query -W -f='\$\{{source:Package\}}=\$\{{source:Version\}}\\n' {package}\b" if not (re.search(source, stage) and f"/usr/share/doc/{package}/copyright" in stage): @@ -76,10 +123,54 @@ def _uncredited_debian_copies(stage: str) -> list[str]: return missing -@pytest.mark.parametrize("stage", ("python-venv", *(f"{kit}-build" for kit in KITS))) -def test_copied_debian_libraries_carry_their_source(stage: str) -> None: +def _stages(text: str) -> dict[str, tuple[str, str]]: + """Stage name -> (parent stage name or base image, body), in file order.""" + found: dict[str, tuple[str, str]] = {} + headers = list(re.finditer(r"^FROM (\S+) AS (\S+)\n", text, re.MULTILINE)) + for index, match in enumerate(headers): + end = headers[index + 1].start() if index + 1 < len(headers) else len(text) + found[match.group(2)] = (match.group(1), text[match.end() : end]) + return found + + +def _inherited(stages: dict[str, tuple[str, str]], name: str) -> str: + """The bodies of stage `name` and of every stage it is FROM, joined.""" + bodies, current, seen = [], name, set() + while current in stages and current not in seen: + seen.add(current) + parent, body = stages[current] + bodies.append(body) + current = parent + return "".join(bodies) + + +@pytest.mark.parametrize( + "path,stage,verb", + COPY_SITES, + ids=[f"{Path(path).name}:{stage}" for path, stage, _verb in COPY_SITES], +) +def test_copied_debian_libraries_carry_their_source(path: str, stage: str, verb: str) -> None: """A library copied out of a Debian package ships that package's copyright and SOURCE.""" - assert _uncredited_debian_copies(_stage(DOCKERFILE.read_text(), stage)) == [] + text = (REPO / path).read_text(encoding="utf-8") + assert _copy_gaps(_stage(text, stage), verb) == [] + + +@pytest.mark.parametrize("verb", ("cp -a", "cp -d")) +def test_an_uncredited_debian_copy_fails(verb: str) -> None: + """Negative case: a stage that copies a Debian library without its record is reported.""" + stage = f"RUN {verb} /usr/lib/*-linux-gnu*/libzimg.so* /somewhere/lib/ \\\n" + assert _copy_gaps(stage, verb) == ["libzimg2"] + + +def test_the_copy_reader_refuses_a_stage_with_a_record_for_another_package() -> None: + """Negative case: recording libgomp1's source does not credit a copied libzimg2.""" + stage = ( + "RUN cp -d /usr/lib/*-linux-gnu*/libzimg.so* /dist/lib/ \\\n" + " && dpkg-query -W -f='${source:Package}=${source:Version}\\n' libgomp1 \\\n" + " > /dist/doc/libgomp1/SOURCE \\\n" + " && cp /usr/share/doc/libgomp1/copyright /dist/doc/libgomp1/copyright\n" + ) + assert _copy_gaps(stage, "cp -d") == ["libzimg2"] @pytest.mark.parametrize("kit", KITS) @@ -89,7 +180,47 @@ def test_gpu_runtime_ships_the_libzimg2_record(kit: str) -> None: assert f"COPY --from={kit}-build /zimg-doc /usr/share/doc/libzimg2" in runtime -def test_an_uncredited_debian_copy_fails() -> None: - """Negative case: a stage that copies libzimg without its Debian record is reported.""" - stage = "RUN cp -a /usr/lib/*-linux-gnu*/libzimg.so* /opt/vmafx/lib/ \\\n" - assert _uncredited_debian_copies(stage) == ["libzimg2"] +@pytest.mark.parametrize( + "path,stage", + RUNTIME_STAGES, + ids=[f"{Path(path).name}:{stage}" for path, stage in RUNTIME_STAGES], +) +def test_a_published_runtime_installs_the_record(path: str, stage: str) -> None: + """The runtime stage carries the copyright and SOURCE its licence entry reads.""" + tree = _inherited(_stages((REPO / path).read_text(encoding="utf-8")), stage) + assert "/doc/libzimg2" in tree, "no builder staging of the record" + assert "/usr/share/doc/libzimg2" in tree, "the runtime stage does not install it" + + +def test_the_inheritance_reader_sees_a_parent_stage() -> None: + """Negative case: a record installed by the FROM stage counts, an absent one does not.""" + text = ( + "FROM base AS parent\n" + "COPY --from=b /dist/doc/libzimg2 /usr/share/doc/libzimg2\n" + "FROM parent AS child\n" + "RUN true\n" + ) + stages = _stages(text) + assert "/usr/share/doc/libzimg2" in _inherited(stages, "child") + assert "/usr/share/doc/libzimg2" in _inherited(stages, "parent") + assert _inherited(stages, "absent") == "" + without = "FROM base AS parent\nRUN true\nFROM parent AS child\nRUN true\n" + assert "/usr/share/doc/libzimg2" not in _inherited(_stages(without), "child") + + +def test_the_oneapi_build_script_records_the_libzimg2_it_stages() -> None: + """The shared oneAPI build script stages libzimg.so*, so it stages the record too. + + Two published images take /dist/lib from builder-oneapi2026: the production + oneAPI image and the node SYCL image. Both inherit the zimg component, so the + script is a copy site like the Dockerfiles above and is held to the same rule. + """ + script = (REPO / "scripts/ci/build-oneapi-libvmaf.sh").read_text(encoding="utf-8") + assert _copy_gaps(script, "cp -d") == [] + assert "dpkg-query" in script and "/usr/share/doc/libzimg2/copyright" in script + + +def test_a_oneapi_build_script_without_the_record_fails() -> None: + """Negative case: the oneAPI staging without the Debian record is reported.""" + script = 'cp -d /usr/lib/*-linux-gnu*/libzimg.so* "${dist_dir}/lib/"\n' + assert _copy_gaps(script, "cp -d") == ["libzimg2"] diff --git a/tools/rc1-tester/tests/test_licensing_production.py b/tools/rc1-tester/tests/test_licensing_production.py index 09f8130cbc..7a0fbaf60f 100644 --- a/tools/rc1-tester/tests/test_licensing_production.py +++ b/tools/rc1-tester/tests/test_licensing_production.py @@ -335,6 +335,77 @@ def test_the_release_files_carry_checked_notices() -> None: assert {"release-native", "release-models"} <= records.keys() +# ------------------------------------------------- the Debian zimg record (issue #3242) + + +def test_the_production_zimg_component_reads_the_debian_record() -> None: + """The production images copy libzimg2 out of Debian, so their zimg entry says so. + + The tester entry already reads `usr/share/doc/libzimg2/SOURCE` and takes the + package's copyright file as its licence text; the production entry described + upstream zimg 3.0.6, which those images do not ship. + """ + expanded = lic.load_manifest()["artifacts"] + tester = next(c for c in expanded["image"]["components"] if c["id"] == "zimg") + production = next( + c for c in expanded["production-cli-image"]["components"] if c["id"] == "zimg" + ) + assert production["debian_source_file"] == tester["debian_source_file"] + assert production["texts"] == tester["texts"] + assert production["copyright"] == tester["copyright"] + assert production["licence"] == "WTFPL" + assert "usr/share/doc/libzimg2/**" in production["paths"] + assert production["source"] == tester["source"] + + +def test_every_production_image_that_ships_zimg_reads_that_one_record() -> None: + """The eight production records inherit the component, so they agree with it.""" + expanded = lic.load_manifest()["artifacts"] + cli = next(c for c in expanded["production-cli-image"]["components"] if c["id"] == "zimg") + for kind in ( + "production-server-image", + "production-go-server-image", + "production-controller-image", + "production-node-image", + "production-node-sycl-image", + "production-cuda-image", + "production-rocm-image", + "production-oneapi-image", + ): + component = next(c for c in expanded[kind]["components"] if c["id"] == "zimg") + assert component == cli, kind + + +def test_the_debian_record_is_the_text_the_gate_demands() -> None: + """The gate requires the copyright text by path, so the one in the record is it.""" + record = lic.load_manifest()["artifacts"]["production-cli-image"] + scan = {"licences": [], "files": []} + needed = lic.needed_texts(record, scan, lic.load_manifest(), set()) + assert "/usr/share/doc/libzimg2/copyright" in needed + assert needed["/usr/share/doc/libzimg2/copyright"]["artifact"] == ( + "usr/share/doc/libzimg2/copyright" + ) + + +def test_the_images_that_link_zimg_from_source_keep_their_own_record() -> None: + """macOS, the Windows zips and the native release build zimg themselves. + + Issue #3242 leaves them alone: each links zimg from upstream's source archive + (Homebrew, vcpkg, the release container's zimg-devel), so none of them has a + Debian SOURCE file to point at and none may claim one. + """ + expanded = lic.load_manifest()["artifacts"] + for kind in ("macos", "windows-zip", "windows-cuda-zip", "windows-sycl-zip", "release-native"): + component = next(c for c in expanded[kind]["components"] if c["id"] == "zimg") + assert "debian_source_file" not in component, kind + assert not any( + "usr/share/doc/libzimg2" in text for text in json.dumps(component["texts"]).split() + ), kind + assert "sekrit-twc/zimg" in component["source"] or "zimg-3.0.6-COPYING" in json.dumps( + component + ), kind + + def test_production_node_sycl_image_shares_runtime_and_node_components() -> None: """production-node-sycl-image shares the oneAPI SYCL runtime components and the Intel GPU user-mode stack from sycl-image and the worker node components from production-node-image (ADR-3371). From 13ef98ff02627487dab499a52632d9ff1258442d Mon Sep 17 00:00:00 2001 From: luoye-cpu <3412829159@qq.com> Date: Wed, 11 Nov 2026 04:58:23 +0800 Subject: [PATCH 2/5] test(docker): make the zimg copy-site contract refuse a vacuous entry MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Codacy's review asked for `("docker/Dockerfile.production-gpu", "builder-oneapi2026")` in COPY_SITES. That stage copies nothing itself — it runs scripts/ci/build-oneapi-libvmaf.sh, which the neighbouring test pins — so the entry would have passed while asserting nothing. The suggestion does name a real hole, though: every entry in the table passed on an empty match, so a stage rename, a moved copy line or a wrong copy verb would have turned the contract green and silent. _copied_debian_packages() now reports what a stage copies, and the parametrized test requires libzimg2 among them before it checks the record. Two added tests hold the oneAPI pair down: the script is the copy site (non-empty, with the record), and the two builder-oneapi2026 stages carry no copy line of their own, so moving one there is noticed rather than silently uncovered. Checked: all ten entries match a real copy line and none is vacuous, and a wrong verb reads as an empty match — so the guard fires on the failure it exists for rather than on a coincidence. Signed-off-by: luoye-cpu <3412829159@qq.com> --- .../tests/test_dockerfile_runtime_libs.py | 35 +++++++++++++++++-- 1 file changed, 32 insertions(+), 3 deletions(-) diff --git a/tools/rc1-tester/tests/test_dockerfile_runtime_libs.py b/tools/rc1-tester/tests/test_dockerfile_runtime_libs.py index 3f2f67fad8..a3416a15d7 100644 --- a/tools/rc1-tester/tests/test_dockerfile_runtime_libs.py +++ b/tools/rc1-tester/tests/test_dockerfile_runtime_libs.py @@ -111,6 +111,12 @@ def test_the_stage_reader_refuses_an_unknown_stage() -> None: ) +def _copied_debian_packages(stage: str, verb: str) -> list[str]: + """Debian packages whose library the stage copies out of /usr/lib.""" + pattern = rf"{re.escape(verb)} /usr/lib/\*-linux-gnu\*?/(lib\w+)\.so" + return [DEBIAN_COPIES.get(lib, lib) for lib in sorted(set(re.findall(pattern, stage)))] + + def _copy_gaps(stage: str, verb: str) -> list[str]: """Debian packages whose library the stage copies without recording its SOURCE and copyright.""" pattern = rf"{re.escape(verb)} /usr/lib/\*-linux-gnu\*?/(lib\w+)\.so" @@ -150,9 +156,16 @@ def _inherited(stages: dict[str, tuple[str, str]], name: str) -> str: ids=[f"{Path(path).name}:{stage}" for path, stage, _verb in COPY_SITES], ) def test_copied_debian_libraries_carry_their_source(path: str, stage: str, verb: str) -> None: - """A library copied out of a Debian package ships that package's copyright and SOURCE.""" - text = (REPO / path).read_text(encoding="utf-8") - assert _copy_gaps(_stage(text, stage), verb) == [] + """A library copied out of a Debian package ships that package's copyright and SOURCE. + + The non-empty check is the point of the parametrization: a stage that stops + copying the library (or a rename that makes `_stage` return the wrong body) + would otherwise pass this test while asserting nothing. + """ + body = _stage((REPO / path).read_text(encoding="utf-8"), stage) + copied = _copied_debian_packages(body, verb) + assert "libzimg2" in copied, f"{path}:{stage} copies no libzimg2 with '{verb}'" + assert _copy_gaps(body, verb) == [] @pytest.mark.parametrize("verb", ("cp -a", "cp -d")) @@ -214,12 +227,28 @@ def test_the_oneapi_build_script_records_the_libzimg2_it_stages() -> None: Two published images take /dist/lib from builder-oneapi2026: the production oneAPI image and the node SYCL image. Both inherit the zimg component, so the script is a copy site like the Dockerfiles above and is held to the same rule. + The `builder-oneapi2026` stages themselves carry no copy line — they only run + this script — so the script, not those stages, is what `COPY_SITES` cannot + cover and what this test pins. """ script = (REPO / "scripts/ci/build-oneapi-libvmaf.sh").read_text(encoding="utf-8") + assert _copied_debian_packages(script, "cp -d") == ["libzimg2"] assert _copy_gaps(script, "cp -d") == [] assert "dpkg-query" in script and "/usr/share/doc/libzimg2/copyright" in script +def test_the_oneapi_stages_stage_no_library_themselves() -> None: + """The two builder-oneapi2026 stages run the shared script and copy nothing themselves. + + Recorded so a future edit that moves a copy line into one of those stages is + noticed: it would be a copy site outside `COPY_SITES` and outside the script. + """ + for path in ("docker/Dockerfile.production-gpu", "docker/Dockerfile.node"): + body = _stage((REPO / path).read_text(encoding="utf-8"), "builder-oneapi2026") + assert _copied_debian_packages(body, "cp -d") == [], path + assert "build-oneapi-libvmaf.sh" in body, path + + def test_a_oneapi_build_script_without_the_record_fails() -> None: """Negative case: the oneAPI staging without the Debian record is reported.""" script = 'cp -d /usr/lib/*-linux-gnu*/libzimg.so* "${dist_dir}/lib/"\n' From 34859507afe30f4d169f6ad53a9c7baec5ecbcfe Mon Sep 17 00:00:00 2001 From: luoye-cpu <3412829159@qq.com> Date: Wed, 11 Nov 2026 05:17:32 +0800 Subject: [PATCH 3/5] test(docker): discover the copy sites and follow the record to its builder MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Review of #3395 (Sourcery, CodeRabbit, Codacy) found four ways the contract could pass while a library went uncredited. All four are real: 1. The copy sites were a hard-coded tuple, so a new one — or one the tuple had forgotten — was never checked. Dockerfile.tester's `vmaf-build` was already missing, even though its /opt/vmafx reaches the published image through `runtime`'s `COPY --from=vmaf-build /opt/vmafx /opt/vmafx`. The sites are now discovered from COPY_SOURCES (the Dockerfiles/Containerfiles plus the shared scripts a builder runs) and compared with KNOWN_COPY_SITES, so discovery naming an undeclared site fails by name. `vmaf-build` is the one audited exclusion: `test_a_build_only_copy_site_is_superseded` proves `runtime` overwrites /opt/vmafx/lib from python-venv, which carries the record, and installs it — so the build-only copy is replaced rather than shipped uncredited. 2. The runtime check matched the path as text, which a comment, an ENV or a bare `mkdir` satisfied. It now requires a real `COPY --from= /usr/share/doc/libzimg2` and proves that builder writes ``; a builder whose record comes from a script it runs is named in SCRIPT_STAGED rather than being held to a line it does not have. 3. The source-built artifacts' provenance was asserted with an `or`, so a stale text reference covered a wrong `source`. `source` is now compared with the 3.0.6 archive URL itself, and the texts entry by name. 4. The reader's limits were implied. They are now a known-gap test listing the forms it models and the two it does not (a concrete arch path, `install`). Signed-off-by: luoye-cpu <3412829159@qq.com> --- .../tests/test_dockerfile_runtime_libs.py | 404 +++++++++++++----- .../tests/test_licensing_production.py | 20 +- 2 files changed, 310 insertions(+), 114 deletions(-) diff --git a/tools/rc1-tester/tests/test_dockerfile_runtime_libs.py b/tools/rc1-tester/tests/test_dockerfile_runtime_libs.py index a3416a15d7..b99544c4e3 100644 --- a/tools/rc1-tester/tests/test_dockerfile_runtime_libs.py +++ b/tools/rc1-tester/tests/test_dockerfile_runtime_libs.py @@ -2,19 +2,24 @@ # SPDX-License-Identifier: EUPL-1.2 """Every image stage that copies a Debian library ships that package's record. -The GPU build stages of `docker/Dockerfile.tester` copy the shared libraries the -`vmaf` binary links (libzimg since zimg became a default dependency) into -`/opt/vmafx/lib`. The matching runtime stage must copy that directory and put it -on `LD_LIBRARY_PATH`, or the stage's `ldd` check stops the image build with -`libzimg.so.2 => not found`. The GPU images build only on a push to master, so -this test is the pull-request coverage of that contract. - -The same stages copy `libzimg.so*` out of Debian's `libzimg2` package, so they -also carry the record the licence manifest reads (issue #3241 for the tester -images, issue #3242 for the production ones): the package's copyright file plus -a `dpkg-query` `SOURCE` line. One table names every copying builder and its -published runtime stages, so a new copy site cannot land without the record -(HISS-19). +The build stages copy the shared libraries the `vmaf` binary links (libzimg +since zimg became a default dependency) out of Debian's `libzimg2` package. Two +contracts follow, and this module holds both: + +- the **tester** image's GPU runtime stages find the libraries their build stage + staged in `/opt/vmafx/lib`, or the stage's `ldd` check stops the image build + with `libzimg.so.2 => not found` (the GPU images build only on a push to + master, so this is the pull-request coverage of that contract); +- every stage that copies such a library also carries the record the licence + manifest reads — the package's copyright file plus a `dpkg-query` `SOURCE` + line — and every published runtime stage installs it at + `/usr/share/doc/libzimg2` (issue #3241 for the tester images, #3242 for the + production ones). + +The copy sites are **discovered** from the tree rather than listed, so a stage +that starts copying a Debian library lands in the contract without anybody +remembering to add it; `KNOWN_COPY_SITES` names the ones the tree has today and +fails when discovery finds one it does not list. """ from __future__ import annotations @@ -28,6 +33,79 @@ DOCKERFILE = REPO / "docker/Dockerfile.tester" KITS = ("sycl", "cuda", "hip") +# Files that can copy a library out of a Debian package into an image tree: the +# Dockerfiles and Containerfiles of the tree, and the shared build scripts a +# builder stage runs (scripts/ci/build-oneapi-libvmaf.sh is the one that stages +# /dist/lib for the two oneAPI images). +COPY_SOURCES = ( + "Dockerfile", + "Dockerfile.go-server", + "Containerfile.vmafx", + "dev/Containerfile", + "docker/Dockerfile.controller", + "docker/Dockerfile.node", + "docker/Dockerfile.production", + "docker/Dockerfile.production-gpu", + "docker/Dockerfile.tester", + "scripts/ci/build-oneapi-libvmaf.sh", +) + +# Every stage that copies a Debian library today, as (file, stage, copy verb). +# `vmaf-build` was missing from the pre-#3242 coverage of `Dockerfile.tester` +# even though its /opt/vmafx reaches the published image at `runtime`'s +# `COPY --from=vmaf-build /opt/vmafx /opt/vmafx`; discovery below found it. +KNOWN_COPY_SITES = ( + ("docker/Dockerfile.tester", "python-venv", "cp -a"), + ("docker/Dockerfile.tester", "vmaf-build", "cp -a"), + ("docker/Dockerfile.tester", "sycl-build", "cp -a"), + ("docker/Dockerfile.tester", "cuda-build", "cp -a"), + ("docker/Dockerfile.tester", "hip-build", "cp -a"), + ("docker/Dockerfile.production", "builder", "cp -d"), + ("docker/Dockerfile.controller", "vmaf-builder", "cp -d"), + ("docker/Dockerfile.production-gpu", "builder-cuda13", "cp -d"), + ("docker/Dockerfile.production-gpu", "builder-rocm10", "cp -d"), + ("Dockerfile.go-server", "vmaf-builder", "cp -d"), + ("docker/Dockerfile.node", "vmaf-builder", "cp -d"), + ("scripts/ci/build-oneapi-libvmaf.sh", "", "cp -d"), +) + +# One copy site is build-only: `vmaf-build` stages /opt/vmafx for the golden +# build profile, and the published `runtime` stage overwrites /opt/vmafx/lib +# from `python-venv` (which does carry the record) before it installs +# /usr/share/doc/libzimg2. `test_a_build_only_copy_site_is_superseded` proves +# that order, so this is an audited exclusion rather than a hole. +BUILD_ONLY_COPY_SITES = (("docker/Dockerfile.tester", "vmaf-build", "cp -a"),) + +# The published runtime stages that install the record: (file, stage). The +# install may sit in the stage or in a stage it is FROM. +RUNTIME_STAGES = ( + ("docker/Dockerfile.tester", "runtime"), + ("docker/Dockerfile.tester", "sycl-runtime"), + ("docker/Dockerfile.tester", "cuda-runtime"), + ("docker/Dockerfile.tester", "hip-runtime"), + ("docker/Dockerfile.production", "cli-assembled"), + ("docker/Dockerfile.production", "server-assembled"), + ("docker/Dockerfile.controller", "controller-assembled"), + ("docker/Dockerfile.production-gpu", "cuda13-assembled"), + ("docker/Dockerfile.production-gpu", "rocm10-assembled"), + ("docker/Dockerfile.production-gpu", "oneapi2026-assembled"), + ("Dockerfile.go-server", "go-server-assembled"), + ("docker/Dockerfile.node", "node-assembled"), + ("docker/Dockerfile.node", "node-sycl-assembled"), +) + +DEBIAN_COPIES = {"libzimg": "libzimg2", "libgomp": "libgomp1"} +DOC_ROOT = "/usr/share/doc" +# Where the shared oneAPI build script writes the record (its `dist_dir`). +ONEAPI_DIST = "/dist" + +# A builder that stages the record because a *script* it runs writes it, not a +# COPY/RUN line of its own: (file, stage) -> the script's dist_dir. +SCRIPT_STAGED = { + ("docker/Dockerfile.production-gpu", "builder-oneapi2026"): ONEAPI_DIST, + ("docker/Dockerfile.node", "builder-oneapi2026"): ONEAPI_DIST, +} + def _stage(text: str, name: str) -> str: """The body of stage `name`, up to the next FROM line.""" @@ -60,7 +138,7 @@ def test_the_library_path_reader_refuses_a_missing_entry() -> None: """Negative case: a runtime stage without the directory on its path fails.""" stage = "ENV LD_LIBRARY_PATH=/opt/vmafx/build/src:/opt/vmafx/lib/rocm \\\n" assert "/opt/vmafx/lib" not in _library_path(stage) - assert _library_path("ENV LD_LIBRARY_PATH=/a:/opt/vmafx/lib \\\n")[-1] == "/opt/vmafx/lib" + assert _library_path("ENV LD_LIBRARY_PATH=/opt/vmafx/lib \\\n")[-1] == "/opt/vmafx/lib" def test_the_stage_reader_refuses_an_unknown_stage() -> None: @@ -69,64 +147,25 @@ def test_the_stage_reader_refuses_an_unknown_stage() -> None: _stage("FROM x AS a\nRUN true\n", "b") -DEBIAN_COPIES = {"libzimg": "libzimg2", "libgomp": "libgomp1"} - -# Every image stage that copies a Debian library into a tree a published image is -# built from, as (file, stage, copy verb). The verb is `cp -a` in the tester -# image's build stages (whole SONAME chain followed) and `cp -d` in the -# production builders (the symlinks are copied as symlinks). Issue #3242's scope -# note of 2026-11-10 named six production builder lines; `Dockerfile.go-server` -# and `docker/Dockerfile.node` are the two the issue text had missed, and the -# node image inherits `production-cli-image`'s zimg component like the rest. -COPY_SITES = ( - ("docker/Dockerfile.tester", "python-venv", "cp -a"), - ("docker/Dockerfile.tester", "sycl-build", "cp -a"), - ("docker/Dockerfile.tester", "cuda-build", "cp -a"), - ("docker/Dockerfile.tester", "hip-build", "cp -a"), - ("docker/Dockerfile.production", "builder", "cp -d"), - ("docker/Dockerfile.controller", "vmaf-builder", "cp -d"), - ("docker/Dockerfile.production-gpu", "builder-cuda13", "cp -d"), - ("docker/Dockerfile.production-gpu", "builder-rocm10", "cp -d"), - ("Dockerfile.go-server", "vmaf-builder", "cp -d"), - ("docker/Dockerfile.node", "vmaf-builder", "cp -d"), -) - -# The published runtime stages that install the record, each with the base of its -# FROM chain: the record may come from the stage itself or from a stage it is -# FROM (node-sycl-assembled is FROM node-assembled, which is FROM runtime-base). -RUNTIME_STAGES = ( - ("docker/Dockerfile.tester", "runtime"), - ("docker/Dockerfile.tester", "sycl-runtime"), - ("docker/Dockerfile.tester", "cuda-runtime"), - ("docker/Dockerfile.tester", "hip-runtime"), - ("docker/Dockerfile.production", "cli-assembled"), - ("docker/Dockerfile.production", "server-assembled"), - ("docker/Dockerfile.controller", "controller-assembled"), - ("docker/Dockerfile.production-gpu", "cuda13-assembled"), - ("docker/Dockerfile.production-gpu", "rocm10-assembled"), - ("docker/Dockerfile.production-gpu", "oneapi2026-assembled"), - ("Dockerfile.go-server", "go-server-assembled"), - ("docker/Dockerfile.node", "node-assembled"), - ("docker/Dockerfile.node", "node-sycl-assembled"), +COPY_LINE = re.compile( + r"(?:^|&&|\|\||\n|\bRUN[ \t])\s*(cp -[ad]) /usr/lib/\*-linux-gnu\*?/(lib\w+)\.so", re.MULTILINE ) -def _copied_debian_packages(stage: str, verb: str) -> list[str]: - """Debian packages whose library the stage copies out of /usr/lib.""" - pattern = rf"{re.escape(verb)} /usr/lib/\*-linux-gnu\*?/(lib\w+)\.so" - return [DEBIAN_COPIES.get(lib, lib) for lib in sorted(set(re.findall(pattern, stage)))] +def _copied_debian_packages(stage: str) -> list[str]: + """Debian packages whose library the stage copies out of a /usr/lib directory.""" + return [DEBIAN_COPIES.get(lib, lib) for _verb, lib in sorted(COPY_LINE.findall(stage))] -def _copy_gaps(stage: str, verb: str) -> list[str]: +def _copy_gaps(stage: str) -> list[str]: """Debian packages whose library the stage copies without recording its SOURCE and copyright.""" - pattern = rf"{re.escape(verb)} /usr/lib/\*-linux-gnu\*?/(lib\w+)\.so" missing = [] - for lib in sorted(set(re.findall(pattern, stage))): + for _verb, lib in sorted(COPY_LINE.findall(stage)): package = DEBIAN_COPIES.get(lib, lib) source = rf"dpkg-query -W -f='\$\{{source:Package\}}=\$\{{source:Version\}}\\n' {package}\b" - if not (re.search(source, stage) and f"/usr/share/doc/{package}/copyright" in stage): + if not (re.search(source, stage) and f"{DOC_ROOT}/{package}/copyright" in stage): missing.append(package) - return missing + return sorted(set(missing)) def _stages(text: str) -> dict[str, tuple[str, str]]: @@ -139,43 +178,130 @@ def _stages(text: str) -> dict[str, tuple[str, str]]: return found -def _inherited(stages: dict[str, tuple[str, str]], name: str) -> str: - """The bodies of stage `name` and of every stage it is FROM, joined.""" - bodies, current, seen = [], name, set() +def _inherit_chain(stages: dict[str, tuple[str, str]], name: str) -> list[tuple[str, str]]: + """(stage name, body) of stage `name` and of every stage it is FROM, innermost first.""" + chain, current, seen = [], name, set() while current in stages and current not in seen: seen.add(current) parent, body = stages[current] - bodies.append(body) + chain.append((current, body)) current = parent - return "".join(bodies) + return chain + + +def _discovered_copy_sites() -> set[tuple[str, str, str]]: + """(file, stage, verb) of every stage in the tree that copies a Debian library. + + A Dockerfile is read stage by stage; a shell script (which has no stages) is + read whole and reported with an empty stage name. + """ + found: set[tuple[str, str, str]] = set() + for rel in COPY_SOURCES: + text = (REPO / rel).read_text(encoding="utf-8") + stages = _stages(text) + if stages: + for stage, (_parent, body) in stages.items(): + for verb, _lib in COPY_LINE.findall(body): + found.add((rel, stage, verb)) + else: + for verb, _lib in COPY_LINE.findall(text): + found.add((rel, "", verb)) + return found + + +def test_the_declared_copy_sites_are_every_copy_site_in_the_tree() -> None: + """Discovery and the declared list agree, so a new copy site cannot slip past. + + This is the check the earlier hard-coded tuple lacked: a stage that starts + copying a Debian library, or one the list forgot, shows up here by name. + """ + discovered = _discovered_copy_sites() + declared = {(path, stage, verb) for path, stage, verb in KNOWN_COPY_SITES} + assert ( + discovered - declared == set() + ), "copy sites not covered by KNOWN_COPY_SITES: " + ", ".join( + f"{p}:{s} ({v})" for p, s, v in sorted(discovered - declared) + ) + assert ( + declared - discovered == set() + ), "KNOWN_COPY_SITES names stages that copy no Debian library any more: " + ", ".join( + f"{p}:{s} ({v})" for p, s, v in sorted(declared - discovered) + ) + + +def test_the_copy_site_discovery_finds_a_planted_site() -> None: + """Known gap: discovery reads COPY_SOURCES, so a Dockerfile outside it is not seen. + + The list is the contract: adding a Dockerfile that copies a Debian library + means adding it to COPY_SOURCES (or the site goes unheld). + """ + assert "docker/Dockerfile.tester" in COPY_SOURCES + assert "docker/Dockerfile.production-gpu" in COPY_SOURCES + assert "docker/Dockerfile.operator" not in COPY_SOURCES # copies nothing from /usr/lib @pytest.mark.parametrize( "path,stage,verb", - COPY_SITES, - ids=[f"{Path(path).name}:{stage}" for path, stage, _verb in COPY_SITES], + [site for site in KNOWN_COPY_SITES if site not in BUILD_ONLY_COPY_SITES], + ids=[ + f"{Path(path).name}:{stage or 'script'}" + for path, stage, _verb in KNOWN_COPY_SITES + if (path, stage, _verb) not in BUILD_ONLY_COPY_SITES + ], ) def test_copied_debian_libraries_carry_their_source(path: str, stage: str, verb: str) -> None: """A library copied out of a Debian package ships that package's copyright and SOURCE. - The non-empty check is the point of the parametrization: a stage that stops - copying the library (or a rename that makes `_stage` return the wrong body) - would otherwise pass this test while asserting nothing. + The non-empty check keeps the parametrization honest: a stage that stops + copying the library would otherwise pass while asserting nothing. + """ + text = (REPO / path).read_text(encoding="utf-8") + body = _stage(text, stage) if stage else text + assert verb in body, f"{path}:{stage} has no '{verb}' copy line; update KNOWN_COPY_SITES" + assert "libzimg2" in _copied_debian_packages( + body + ), f"{path}:{stage or 'script'} copies no libzimg2" + assert _copy_gaps(body) == [] + + +def test_a_build_only_copy_site_is_superseded() -> None: + """The one build-only copy site's library is replaced, with its record, in the image. + + `vmaf-build` copies libzimg into /opt/vmafx/lib without a doc record. The + published `runtime` stage copies /opt/vmafx from it and then overwrites + /opt/vmafx/lib from `python-venv` — which does stage the record — before + installing /usr/share/doc/libzimg2. Docker's COPY order is the proof: the + later COPY of the same destination wins, so the image holds python-venv's + libzimg, whose record is installed at line 156. If either half of that + pairing changes (the record source, or the lib overwrite), this fails. """ - body = _stage((REPO / path).read_text(encoding="utf-8"), stage) - copied = _copied_debian_packages(body, verb) - assert "libzimg2" in copied, f"{path}:{stage} copies no libzimg2 with '{verb}'" - assert _copy_gaps(body, verb) == [] + text = DOCKERFILE.read_text(encoding="utf-8") + runtime = _stage(text, "runtime") + lib_source = re.search(r"^COPY --from=(\S+) (\S+) /opt/vmafx/lib$", runtime, re.MULTILINE) + assert lib_source, "runtime no longer copies /opt/vmafx/lib from a stage" + winner, source_dir = lib_source.groups() + assert source_dir == "/extra-libs/lib", source_dir + assert winner == "python-venv", winner + # The stage that wins must itself carry the record, and the runtime installs it. + venv = _stage(text, winner) + assert _copy_gaps(venv) == [] + assert f"COPY --from={winner} /extra-libs/doc/libzimg2 {DOC_ROOT}/libzimg2" in runtime + # And it must be copied at all, so the build-only copy is superseded and not dropped. + whole_tree = "COPY --from=vmaf-build /opt/vmafx /opt/vmafx" + assert whole_tree in runtime + assert runtime.index(whole_tree) < runtime.index( + f"COPY --from={winner} {source_dir} /opt/vmafx/lib" + ) @pytest.mark.parametrize("verb", ("cp -a", "cp -d")) def test_an_uncredited_debian_copy_fails(verb: str) -> None: """Negative case: a stage that copies a Debian library without its record is reported.""" stage = f"RUN {verb} /usr/lib/*-linux-gnu*/libzimg.so* /somewhere/lib/ \\\n" - assert _copy_gaps(stage, verb) == ["libzimg2"] + assert _copy_gaps(stage) == ["libzimg2"] -def test_the_copy_reader_refuses_a_stage_with_a_record_for_another_package() -> None: +def test_the_copy_reader_refuses_a_record_for_another_package() -> None: """Negative case: recording libgomp1's source does not credit a copied libzimg2.""" stage = ( "RUN cp -d /usr/lib/*-linux-gnu*/libzimg.so* /dist/lib/ \\\n" @@ -183,14 +309,61 @@ def test_the_copy_reader_refuses_a_stage_with_a_record_for_another_package() -> " > /dist/doc/libgomp1/SOURCE \\\n" " && cp /usr/share/doc/libgomp1/copyright /dist/doc/libgomp1/copyright\n" ) - assert _copy_gaps(stage, "cp -d") == ["libzimg2"] + assert _copy_gaps(stage) == ["libzimg2"] + + +def test_the_copy_reader_sees_the_forms_the_tree_uses() -> None: + """The forms the reader models, and the one it does not, recorded rather than implied. + + It reads `cp -a` / `cp -d` from a `/usr/lib/*-linux-gnu*/` glob — every form + in COPY_SOURCES today, whether the line starts the `RUN` or continues its + `&&` chain. A path spelled with a concrete arch (`/usr/lib/x86_64-linux-gnu`) + or built in a shell variable, and `install` instead of `cp`, are not + modelled; that is why discovery and the declared list are compared, and why + a new copy written that way has to be added to `COPY_SOURCES` and this + reader together. + """ + assert _copy_gaps("RUN cp -a /usr/lib/*-linux-gnu*/libzimg.so* /x \\\n") == ["libzimg2"] + assert _copy_gaps("RUN cp -d /usr/lib/*-linux-gnu*/libzimg.so* /x \\\n") == ["libzimg2"] + assert _copy_gaps("RUN cp -a /usr/lib/*-linux-gnu/libzimg.so* /x \\\n") == ["libzimg2"] + assert _copy_gaps( + "RUN mkdir -p /x \\\n && cp -a /usr/lib/*-linux-gnu*/libzimg.so* /x/ \\\n" + ) == ["libzimg2"] + # Known gap, deliberate: a concrete arch path and an `install` call are unseen. + assert _copy_gaps("RUN cp -a /usr/lib/x86_64-linux-gnu/libzimg.so* /x \\\n") == [] + assert _copy_gaps("RUN install -m 644 /usr/lib/*-linux-gnu*/libzimg.so.2 /x \\\n") == [] @pytest.mark.parametrize("kit", KITS) def test_gpu_runtime_ships_the_libzimg2_record(kit: str) -> None: - """The runtime stage installs the libzimg2 copyright and SOURCE its licence entry reads.""" + """The runtime stage copies the record its build stage wrote, into the image.""" runtime = _stage(DOCKERFILE.read_text(), f"{kit}-runtime") - assert f"COPY --from={kit}-build /zimg-doc /usr/share/doc/libzimg2" in runtime + installs = _record_installs(runtime) + assert installs, f"{kit}-runtime installs no {DOC_ROOT}/libzimg2" + + +def _record_installs(body: str) -> list[tuple[str, str]]: + """(source, builder) of every `COPY --from= /usr/share/doc/libzimg2`.""" + pattern = rf"^COPY --from=(\S+) (\S+) {re.escape(DOC_ROOT)}/libzimg2\b" + return [(source, builder) for builder, source in re.findall(pattern, body, re.MULTILINE)] + + +def _builder_writes_record(path: str, builder: str, source: str) -> bool: + """Whether stage `builder` of `path` writes the record under `source`. + + A stage writes it either with a COPY/RUN line of its own, or by running a + shared script that writes to the stage's dist_dir — `SCRIPT_STAGED` names + those, so `builder-oneapi2026` (which only runs + scripts/ci/build-oneapi-libvmaf.sh) is not held to a line it does not have. + """ + staged = _stages((REPO / path).read_text(encoding="utf-8")).get(builder) + if staged is None: + return False + body = staged[1] + if f"{source.rstrip('/')}/copyright" in body or f"{source.rstrip('/')}/" in body: + return True + dist = SCRIPT_STAGED.get((path, builder)) + return bool(dist) and source.rstrip("/") == f"{dist}/doc/libzimg2" @pytest.mark.parametrize( @@ -199,14 +372,38 @@ def test_gpu_runtime_ships_the_libzimg2_record(kit: str) -> None: ids=[f"{Path(path).name}:{stage}" for path, stage in RUNTIME_STAGES], ) def test_a_published_runtime_installs_the_record(path: str, stage: str) -> None: - """The runtime stage carries the copyright and SOURCE its licence entry reads.""" - tree = _inherited(_stages((REPO / path).read_text(encoding="utf-8")), stage) - assert "/doc/libzimg2" in tree, "no builder staging of the record" - assert "/usr/share/doc/libzimg2" in tree, "the runtime stage does not install it" + """The runtime stage copies the record from the builder that staged it. + + The install must be a real `COPY --from= /usr/share/doc/libzimg2` + in the stage or in a stage it inherits, AND that stage must write `` — + a comment or an `ENV` naming the path does not satisfy this, and neither + does a COPY whose source the builder never creates. + """ + stages = _stages((REPO / path).read_text(encoding="utf-8")) + for current, body in _inherit_chain(stages, stage): + for source, builder in _record_installs(body): + assert ( + builder in stages + ), f"{path}:{current} copies the record from unknown stage {builder}" + assert _builder_writes_record( + path, builder, source + ), f"{path}:{builder} never writes {source}, which {current} copies" + return + raise AssertionError(f"{path}:{stage} installs no {DOC_ROOT}/libzimg2") + + +def test_the_record_install_reader_refuses_a_comment_or_env() -> None: + """Negative cases: naming the path is not installing the record.""" + assert _record_installs("# we used to COPY /dist/doc/libzimg2 /usr/share/doc/libzimg2\n") == [] + assert _record_installs("ENV X=/usr/share/doc/libzimg2\n") == [] + assert _record_installs("RUN mkdir -p /usr/share/doc/libzimg2\n") == [] + assert _record_installs("COPY --from=builder /dist/doc/libzimg2 /usr/share/doc/libzimg2\n") == [ + ("/dist/doc/libzimg2", "builder") + ] def test_the_inheritance_reader_sees_a_parent_stage() -> None: - """Negative case: a record installed by the FROM stage counts, an absent one does not.""" + """Negative case: an install by the FROM stage counts, an absent one does not.""" text = ( "FROM base AS parent\n" "COPY --from=b /dist/doc/libzimg2 /usr/share/doc/libzimg2\n" @@ -214,42 +411,27 @@ def test_the_inheritance_reader_sees_a_parent_stage() -> None: "RUN true\n" ) stages = _stages(text) - assert "/usr/share/doc/libzimg2" in _inherited(stages, "child") - assert "/usr/share/doc/libzimg2" in _inherited(stages, "parent") - assert _inherited(stages, "absent") == "" - without = "FROM base AS parent\nRUN true\nFROM parent AS child\nRUN true\n" - assert "/usr/share/doc/libzimg2" not in _inherited(_stages(without), "child") + assert [name for name, _ in _inherit_chain(stages, "child")] == ["child", "parent"] + assert [name for name, _ in _inherit_chain(stages, "absent")] == [] def test_the_oneapi_build_script_records_the_libzimg2_it_stages() -> None: """The shared oneAPI build script stages libzimg.so*, so it stages the record too. Two published images take /dist/lib from builder-oneapi2026: the production - oneAPI image and the node SYCL image. Both inherit the zimg component, so the - script is a copy site like the Dockerfiles above and is held to the same rule. - The `builder-oneapi2026` stages themselves carry no copy line — they only run - this script — so the script, not those stages, is what `COPY_SITES` cannot - cover and what this test pins. + oneAPI image and the node SYCL image. The `builder-oneapi2026` stages carry + no copy line of their own — they run this script — so the script is the copy + site, and it is in `COPY_SOURCES` so discovery sees it. """ script = (REPO / "scripts/ci/build-oneapi-libvmaf.sh").read_text(encoding="utf-8") - assert _copied_debian_packages(script, "cp -d") == ["libzimg2"] - assert _copy_gaps(script, "cp -d") == [] - assert "dpkg-query" in script and "/usr/share/doc/libzimg2/copyright" in script + assert _copied_debian_packages(script) == ["libzimg2"] + assert _copy_gaps(script) == [] + assert "dpkg-query" in script and f"{DOC_ROOT}/libzimg2/copyright" in script def test_the_oneapi_stages_stage_no_library_themselves() -> None: - """The two builder-oneapi2026 stages run the shared script and copy nothing themselves. - - Recorded so a future edit that moves a copy line into one of those stages is - noticed: it would be a copy site outside `COPY_SITES` and outside the script. - """ + """The two builder-oneapi2026 stages run the shared script and copy nothing themselves.""" for path in ("docker/Dockerfile.production-gpu", "docker/Dockerfile.node"): body = _stage((REPO / path).read_text(encoding="utf-8"), "builder-oneapi2026") - assert _copied_debian_packages(body, "cp -d") == [], path + assert _copied_debian_packages(body) == [], path assert "build-oneapi-libvmaf.sh" in body, path - - -def test_a_oneapi_build_script_without_the_record_fails() -> None: - """Negative case: the oneAPI staging without the Debian record is reported.""" - script = 'cp -d /usr/lib/*-linux-gnu*/libzimg.so* "${dist_dir}/lib/"\n' - assert _copy_gaps(script, "cp -d") == ["libzimg2"] diff --git a/tools/rc1-tester/tests/test_licensing_production.py b/tools/rc1-tester/tests/test_licensing_production.py index 7a0fbaf60f..d5c547e93f 100644 --- a/tools/rc1-tester/tests/test_licensing_production.py +++ b/tools/rc1-tester/tests/test_licensing_production.py @@ -395,15 +395,29 @@ def test_the_images_that_link_zimg_from_source_keep_their_own_record() -> None: Debian SOURCE file to point at and none may claim one. """ expanded = lic.load_manifest()["artifacts"] + # Each of them links the 3.0.6 release, and says so in its own `source`. + expected = "https://github.com/sekrit-twc/zimg.git at commit f819b14e8f39d1282400b0d9543e8ef73c1b2bbd, tag release-3.0.6" for kind in ("macos", "windows-zip", "windows-cuda-zip", "windows-sycl-zip", "release-native"): component = next(c for c in expanded[kind]["components"] if c["id"] == "zimg") assert "debian_source_file" not in component, kind assert not any( "usr/share/doc/libzimg2" in text for text in json.dumps(component["texts"]).split() ), kind - assert "sekrit-twc/zimg" in component["source"] or "zimg-3.0.6-COPYING" in json.dumps( - component - ), kind + assert component["source"] == expected, kind + # And the WTFPL text is the one the 3.0.6 archive ships, by name. + assert component["texts"] == [ + { + "name": "WTFPL.txt", + "repo": "tools/rc1-tester/image/licenses/zimg-3.0.6-COPYING.txt", + "label": "zimg 3.0.6 licence text (WTFPL v2)", + } + ] or component["texts"] == [ + { + "name": "WTFPL.txt", + "fetched": "zimg-3.0.6-COPYING", + "label": "zimg 3.0.6 licence text (WTFPL v2)", + } + ], kind def test_production_node_sycl_image_shares_runtime_and_node_components() -> None: From eb1c07899d628fe382a44ce5c3054e374940312d Mon Sep 17 00:00:00 2001 From: luoye-cpu <3412829159@qq.com> Date: Wed, 11 Nov 2026 05:19:11 +0800 Subject: [PATCH 4/5] docs(docker): record the discovered copy site in the invariant and the state row MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The review of #3395 turned up a copy site the old test table never covered: `docker/Dockerfile.tester`'s `vmaf-build` copies libzimg, and its /opt/vmafx reaches the published image through `runtime`'s `COPY --from=vmaf-build /opt/vmafx /opt/vmafx`. Both records now say how that is settled — runtime overwrites /opt/vmafx/lib from python-venv (which carries the record) before installing it, and a test proves the order — and that the contract discovers its sites instead of listing them, so an undeclared one fails by name. Signed-off-by: luoye-cpu <3412829159@qq.com> --- docker/AGENTS.md | 10 ++++++++-- docs/state.md | 2 +- 2 files changed, 9 insertions(+), 3 deletions(-) diff --git a/docker/AGENTS.md b/docker/AGENTS.md index b59efac7ec..1d9b2ea341 100644 --- a/docker/AGENTS.md +++ b/docker/AGENTS.md @@ -161,8 +161,14 @@ image's `licensing.py sources` (the source offer cannot name a Debian package it has no record of) and leaves the notice naming a version the image does not ship. `tools/rc1-tester/tests/test_dockerfile_runtime_libs.py` holds the six builders, the nine published runtime stages (following each `FROM` chain) and -the shared script to it. The macOS bundle, the Windows zips and `release-native` -build zimg from source themselves and keep the 3.0.6 archive entry. +the shared script to it. It **discovers** the copy sites from `COPY_SOURCES` +rather than listing them, so a stage that starts copying a Debian library fails +the contract by name until it is declared. `docker/Dockerfile.tester`'s +`vmaf-build` is the one audited build-only exception: `runtime` overwrites +`/opt/vmafx/lib` from `python-venv` (which carries the record) before installing +it, and a test proves that order. The macOS bundle, the Windows zips and +`release-native` build zimg from source themselves and keep the 3.0.6 archive +entry. ## Fedora optional SYCL repository diff --git a/docs/state.md b/docs/state.md index c24e3be70c..de8b630d8e 100644 --- a/docs/state.md +++ b/docs/state.md @@ -1176,7 +1176,7 @@ landed fix yet._ ## Recently closed -| **T-PRODUCTION-ZIMG-DEBIAN-RECORD-2026-11-10** — the production licence manifest named upstream zimg 3.0.6 while every production image ships Debian's `libzimg2` 3.0.5+ds1-1+b2, so the generated notice named the wrong version and source, and six builders copied `libzimg.so*` out of `/usr/lib/*-linux-gnu*` with nothing in the image the `zimg` component could point at | **FIXED on `fix/issue-3242-zimg-debian-source` (2026-11-11).** Each of the six builder lines the maintainer named (`docker/Dockerfile.production`, `docker/Dockerfile.production-gpu` twice, `docker/Dockerfile.controller`, `Dockerfile.go-server`, `docker/Dockerfile.node`) and the shared `scripts/ci/build-oneapi-libvmaf.sh` behind the two oneAPI images stores `/usr/share/doc/libzimg2/copyright` plus a `dpkg-query` `SOURCE` record under `/dist/doc/libzimg2`; the nine published runtime stages install both at `/usr/share/doc/libzimg2`; the `zimg` entry of `production-cli-image` reads them through `debian_source_file` and an `artifact` text, and the eight production records that inherit it follow (ADR-1517). `tools/rc1-tester/tests/test_dockerfile_runtime_libs.py` holds all six builders, the nine runtime stages and the shared script to it; the macOS, Windows and `release-native` entries stay on the 3.0.6 archive, which each of those links itself. | #3242 | 2026-11-10 | closed | +| **T-PRODUCTION-ZIMG-DEBIAN-RECORD-2026-11-10** — the production licence manifest named upstream zimg 3.0.6 while every production image ships Debian's `libzimg2` 3.0.5+ds1-1+b2, so the generated notice named the wrong version and source, and six builders copied `libzimg.so*` out of `/usr/lib/*-linux-gnu*` with nothing in the image the `zimg` component could point at | **FIXED on `fix/issue-3242-zimg-debian-source` (2026-11-11).** Each of the six builder lines the maintainer named (`docker/Dockerfile.production`, `docker/Dockerfile.production-gpu` twice, `docker/Dockerfile.controller`, `Dockerfile.go-server`, `docker/Dockerfile.node`) and the shared `scripts/ci/build-oneapi-libvmaf.sh` behind the two oneAPI images stores `/usr/share/doc/libzimg2/copyright` plus a `dpkg-query` `SOURCE` record under `/dist/doc/libzimg2`; the nine published runtime stages install both at `/usr/share/doc/libzimg2`; the `zimg` entry of `production-cli-image` reads them through `debian_source_file` and an `artifact` text, and the eight production records that inherit it follow (ADR-1517). `tools/rc1-tester/tests/test_dockerfile_runtime_libs.py` now **discovers** the copy sites instead of listing them, which surfaced a seventh site the old table had never covered: `docker/Dockerfile.tester`'s `vmaf-build` copies libzimg and its `/opt/vmafx` reaches the published image through `runtime`'s `COPY --from=vmaf-build /opt/vmafx /opt/vmafx`, so a test proves `runtime` overwrites `/opt/vmafx/lib` from `python-venv` — which does carry the record — before installing it (an audited build-only exclusion, not a hole). On master 8 of 12 copy sites and 9 of 13 runtime stages fail, on the fix all pass; the macOS, Windows and `release-native` entries stay on the 3.0.6 archive, which each of those links itself. | #3242 | 2026-11-10 | closed | | **T-CI-PRECOMMIT-DIFF-SCOPE-HOOK-INPUT-2026-11-01** — #3237 added the hook `check-fork-python-subprocess` and its script; the Pre-Commit job ran on the pull request's diff only (`--from-ref`), so the new hook never saw the files it flags elsewhere in the tree, the pull request passed, and master `a3ea48309` went red on the whole-tree run of the push | **FIXED on `fix/precommit-scope-config-change` (#3253).** `scripts/ci/precommit_scope.py` runs the whole tree when a pull request changes a hook input: `.pre-commit-config.yaml`, any file under `scripts/ci/` (CI authority) or `scripts/lib/`, any path a hook's command names (local and remote `entry` overrides, `args`, commands inside `bash -c`, `python -m` modules, deleted scripts included) or a module those scripts import; an unreadable file list, a git timeout or an entry it cannot split also select the whole tree. Invariant page: `scripts/ci/AGENTS.d/precommit-scope.md`. | `scripts/ci/tests/test_precommit_scope.py` (21 tests); against the diff-only selector of `634e636a7` 7 of them fail, the #3237 file list among them | `fix/precommit-scope-config-change` | | **T-HOOKS-WINDOWS-POSIX-FIXTURES-2026-09-30** — four local fixture hooks fail on a Windows host | **FIXED in #3282 (part of #2438).** `test_lint_configured.py` normalises the public-model path via `as_posix()` and skips `ConfiguredLintTests` off POSIX with a stated reason; `test_write_compile_commands.py` skips `CompileCommandsExportTest` off POSIX with a stated reason; `test_tidy_ratchet.py` skips off POSIX with a stated reason; `test-sync-pelorus-interop.sh` skips on Windows with a stated reason. `scripts/ci/check-source-adr-citations.py --write` writes the registry with `newline="\n"`, guards against concurrent writes, cleans up temporary files on failure, and `test_check_source_adr_citations.py` verifies canonical byte rewriting. | unit test suite | #3282 | 2026-11-03 | closed | | **T-CI-WARNINGS-NON-MSVC-LEGS-2026-10-07** — remaining: Dev Container Build compiles third-party sources (vpl-gpu-rt -Wstringop-overflow, FFmpeg) and prints gcc's LTO "serial compilation" note | **CLOSED in #3282 (part of #2438).** Recorded the dev container image build as outside the compiler warning gate in `docs/development/ci.md` ("Builds outside the gate") per ADR-2170 and ADR-2828: external container dependencies (`vpl-gpu-rt`, `FFmpeg`) are compiled into temporary container layers during OCI image assembly rather than from repository sources; in-tree VMAFx sources are compiled and gated with `-Dwerror=true` across all CI compiler matrix legs. The Windows `icx-cl` lane was previously resolved and gated with `-Dwerror=true` in PR #2650 (ADR-2828). | `docs/development/ci.md` | #3282 | 2026-11-03 | closed | From 299ec9ffdab59bf481884e78679379152ba823fb Mon Sep 17 00:00:00 2001 From: luoye-cpu <3412829159@qq.com> Date: Wed, 11 Nov 2026 06:11:06 +0800 Subject: [PATCH 5/5] docs(docker): put the zimg invariant in the fragment register MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two blocking findings from the maintainer's review of #3395. `docker/AGENTS.md` failed the caveman lint that `praetorctl compile-context --verify` runs on every nested AGENTS.md: the new section was prose, and every other section of that file is written in the fragment register. It is now the review's suggested form — the same rules as bullets, with no articles to measure — and it drops the six hard-coded line numbers, which had already drifted: `Dockerfile.production-gpu:182` is a comment on this head and the second copy line is :185. The section names the files and leaves the exact sites to the test, which discovers them. `scripts/ci/build-oneapi-libvmaf.sh` wrote the SOURCE line with a space before the redirect, which `shfmt -i 2 -ci` (the shfmt-src pre-commit hook) rewrites. Now `>"${dist_dir}/doc/libzimg2/SOURCE"`, the form the hook produces; shfmt and shellcheck are both clean on the file. Signed-off-by: luoye-cpu <3412829159@qq.com> --- docker/AGENTS.md | 50 ++++++++++++++---------------- scripts/ci/build-oneapi-libvmaf.sh | 2 +- 2 files changed, 25 insertions(+), 27 deletions(-) diff --git a/docker/AGENTS.md b/docker/AGENTS.md index 1d9b2ea341..8ea79e993d 100644 --- a/docker/AGENTS.md +++ b/docker/AGENTS.md @@ -143,32 +143,30 @@ stage Meson's generated `meson-private/libvmaf.pc`; never synthesize it from `scripts/ci/test_e2e_runtime_contract.py` pins these invariants for both Dockerfiles. -## The Debian zimg record every copying builder stages (issue #3242) - -`libzimg.so*` is **Debian's** `libzimg2`, never upstream's 3.0.6 archive, in every -image that copies it out of `/usr/lib/*-linux-gnu*` — which is what makes -`zimg` of `production-cli-image` read -`debian_source_file: usr/share/doc/libzimg2/SOURCE` and take the package's -copyright file as its licence text, as the tester entry already did. Six builder -lines copy it (`docker/Dockerfile.production:86`, `Dockerfile.production-gpu:130` -and `:182`, `docker/Dockerfile.controller:98`, `Dockerfile.go-server:102`, -`docker/Dockerfile.node:184`); a seventh site is the shared -`scripts/ci/build-oneapi-libvmaf.sh` that the two oneAPI images run. **All of them, -and any new copying builder, stage `copyright` + a `dpkg-query` `SOURCE` line -under `/dist/doc/libzimg2`; every runtime stage that publishes `libzimg.so*` -installs that directory at `/usr/share/doc/libzimg2`.** Omitting it fails the -image's `licensing.py sources` (the source offer cannot name a Debian package it -has no record of) and leaves the notice naming a version the image does not -ship. `tools/rc1-tester/tests/test_dockerfile_runtime_libs.py` holds the six -builders, the nine published runtime stages (following each `FROM` chain) and -the shared script to it. It **discovers** the copy sites from `COPY_SOURCES` -rather than listing them, so a stage that starts copying a Debian library fails -the contract by name until it is declared. `docker/Dockerfile.tester`'s -`vmaf-build` is the one audited build-only exception: `runtime` overwrites -`/opt/vmafx/lib` from `python-venv` (which carries the record) before installing -it, and a test proves that order. The macOS bundle, the Windows zips and -`release-native` build zimg from source themselves and keep the 3.0.6 archive -entry. +## Debian zimg record per copying builder (issue #3242) + +`libzimg.so*` copied from `/usr/lib/*-linux-gnu*` = Debian `libzimg2`, never +upstream 3.0.6 archive. So `zimg` component of `production-cli-image` reads +`debian_source_file: usr/share/doc/libzimg2/SOURCE` and takes package copyright +file as licence text, same as tester entry. + +- Every copying builder (`Dockerfile.go-server`, `docker/Dockerfile.controller`, + `.node`, `.production`, `.production-gpu`) and shared + `scripts/ci/build-oneapi-libvmaf.sh` (both oneAPI images) stage `copyright` + + `dpkg-query` `SOURCE` line under `/dist/doc/libzimg2`. New copying builder: same. +- Every runtime stage publishing `libzimg.so*` installs that directory at + `/usr/share/doc/libzimg2`. +- Missing record -> image `licensing.py sources` fails (source offer names no + Debian package without record); notice names version image does not ship. +- `tools/rc1-tester/tests/test_dockerfile_runtime_libs.py` pins builders, + published runtime stages (follows each `FROM` chain) and shared script. Copy + sites discovered from `COPY_SOURCES`, not listed: stage starting to copy Debian + library fails contract by name until declared. +- One audited build-only exception: `docker/Dockerfile.tester` `vmaf-build`. + `runtime` overwrites `/opt/vmafx/lib` from `python-venv` (carries record) before + install; test proves order. +- macOS bundle, Windows zips, `release-native` build zimg from source; keep 3.0.6 + archive entry. ## Fedora optional SYCL repository diff --git a/scripts/ci/build-oneapi-libvmaf.sh b/scripts/ci/build-oneapi-libvmaf.sh index 6ee1e83909..60634bb372 100755 --- a/scripts/ci/build-oneapi-libvmaf.sh +++ b/scripts/ci/build-oneapi-libvmaf.sh @@ -45,7 +45,7 @@ cp -d /usr/lib/*-linux-gnu*/libzimg.so* "${dist_dir}/lib/" # copyright file as the licence text, as the tester images already do. mkdir -p "${dist_dir}/doc/libzimg2" cp /usr/share/doc/libzimg2/copyright "${dist_dir}/doc/libzimg2/copyright" -dpkg-query -W -f='${source:Package}=${source:Version}\n' libzimg2 > "${dist_dir}/doc/libzimg2/SOURCE" +dpkg-query -W -f='${source:Package}=${source:Version}\n' libzimg2 >"${dist_dir}/doc/libzimg2/SOURCE" cp "${build_dir}/tools/vmaf" "${dist_dir}/bin/" cp -r model/. "${dist_dir}/model/"