[BUG]: Released agent image carries 58 HIGH+ findings from a stale base and a leaked build toolchain

[lockwobr]

Version

agent v6.4.2 (ghcr.io/nvidia/nodewright/agent:latest, built 2026-05-18, digest sha256:7cd80f5ef351266dc08c3979e802f9dc936a1ed9fd02e199233e7968a3a0de3e)

What happened?

Scanning the published agent image with grype reports 117 fixable findings, of which 58 are HIGH or CRITICAL (6 Critical, 52 High). None of them are inherent to the agent. They come from two independent causes, either of which can be fixed on its own.

Cause 1: the released image is built on a four-month-old base (55 of the 58).

The image labels record org.opencontainers.image.base.name=nvcr.io/nvidia/distroless/python:3.13-v4.0.6. The newest tag in that repository is 3.13-v4.1.4. Scanning the two bases directly:

base	fixable total	HIGH+	libssl3t64
3.13-v4.0.6	109	55	3.5.6-1~deb13u1
3.13-v4.1.4	4	0	not present

The four findings remaining on v4.1.4 are Medium/Low CPython issues whose only fix is an unreleased 3.15, so they sit below a HIGH+ cutoff.

scripts/latest-distroless.sh already resolves the newest base at build time, so this needs no code change. It needs a rebuild and release. The bulk of the 55 is 18 libssl3t64 plus 18 openssl-provider-fips, with libexpat1, the krb5 libraries, and the CPython binary making up the rest.

Cause 2: the runtime image ships the build toolchain (the remaining 3).

containers/agent.Dockerfile builds a venv and copies all of it into the runtime stage:

• Line 43 runs make venv, which creates ./venv and installs hatch into it.
• Line 47 runs python3 -m venv venv && ./venv/bin/pip install .../skyhook_agent*.whl. python -m venv on an existing directory does not clear it, so the wheel is installed alongside hatch rather than into a clean environment.
• Line 67 copies /code/venv/lib/python3.13/site-packages wholesale into /usr/local/lib/python3.13/site-packages.

Syft reports 50 Python packages in the released image. Among them: hatch, hatchling, virtualenv, keyring, secretstorage, cryptography, httpx, httpcore, rich, pexpect, userpath, trove-classifiers, tomli-w, click, pluggy, platformdirs, packaging.

The agent declares exactly one runtime dependency, jsonschema (agent/skyhook-agent/pyproject.toml). Everything above is build tooling.

The three remaining HIGH+ follow directly from it, via hatch -> keyring -> secretstorage -> cryptography 48.0.0:

Sev	ID	Fixed in
High	GHSA-537c-gmf6-5ccf	48.0.1
High	GHSA-jwv3-5hgf-82ww	49.0.0
High	GHSA-g6cj-pr64-35w5	50.0.0

This matters beyond the CVE count. The agent runs as USER 0:0 and chroots the host filesystem, so anything in its image is attack surface on every managed node.

How to reproduce

grype ghcr.io/nvidia/nodewright/agent:latest --only-fixed -o json \
  | jq '[.matches[] | select(.vulnerability.severity=="High" or .vulnerability.severity=="Critical")] | length'

syft ghcr.io/nvidia/nodewright/agent:latest -o json \
  | jq -r '.artifacts[] | select(.type=="python") | .name' | sort

Suggested fix

Two separable changes:

1. Rebuild and release the agent so latest-distroless.sh picks up the current base. No source change required.
2. Install the wheel into a venv that does not contain the build toolchain, so only runtime dependencies reach the final stage. Building the venv in a distinct path from the hatch one is the smallest fix; verifying the resulting package list against the declared dependencies would keep it from regressing.

Additional context

Found while implementing #627 (image vulnerability scanning). Counts come from a vulnerability database six days stale at the time of scanning, so exact numbers will move slightly; the causes and the base comparison will not.

This issue is why #627 lands report-only: switching on a HIGH+ release gate before these are fixed would block every agent release.

[lockwobr]

Important for whoever triages this after the next agent release: a release resolves 55 of the 58 findings but does not resolve this issue. The two causes have different remedies, and only one of them is a release.

Cause 1, the stale base (55 findings): fixed by a rebuild, no code change. scripts/latest-distroless.sh already resolves the newest base at build time, so the next agent build picks up 3.13-v4.1.4, which measures 0 HIGH+ against 3.13-v4.0.6's 55. Nothing in the repo needs editing for this half.

Cause 2, the build toolchain in the runtime image (3 findings): still present on main today. containers/agent.Dockerfile is unchanged:

RUN make venv                                    # line 43: installs hatch into ./venv
RUN python3 -m venv venv && ./venv/bin/pip install .../skyhook_agent*.whl   # line 47
COPY --from=builder /code/venv/lib/python${PYTHON_VERSION}/site-packages \
     /usr/local/lib/python${PYTHON_VERSION}/site-packages                    # line 67

python -m venv on an existing directory does not clear it, so the wheel installs alongside hatch and line 67 copies the whole tree. The released image carries 50 Python packages where the agent declares one runtime dependency (jsonschema). cryptography arrives via hatch -> keyring -> secretstorage, and is the sole source of the 3 remaining HIGH+.

The CVE count is the smaller half of this. The agent runs as USER 0:0 and chroots the host filesystem, so hatch, virtualenv, httpx, pexpect and the rest are attack surface on every managed node, independent of whether any of them currently has a published CVE.

So: after the next release, expect this issue's finding count to drop from 58 to 3. That is progress, not resolution. Do not close it until the venv is built somewhere the build toolchain is not, and the released image's Python package list matches the declared dependencies.