Repository navigation
Pull every published image by name, including the profiled connectors - #68
Merged
Merged
Conversation
The Release run for v0.2.4 failed on the image scan for all six connectors. Their shared python:3.12-slim base carries a Debian snapshot whose zlib1g (1:1.3.dfsg+really1.3.1-1+b1) has two HIGH findings with a fix already published as +deb13u1, and the gate fails a fixable HIGH rather than shipping it. backend/Dockerfile already upgrades its base layer for exactly this reason; the connectors never did. The upgrade runs before the packages each image installs, so a pinned version such as dnstwist=0~20250130-1 is still what apt applies last and a rebuild cannot move it. Verified by building all six images locally (zlib1g becomes +deb13u1, dnstwist still reports 20250130) and by running Trivy 0.69.0 with the workflow's own flags on connector-hibp: exit 0. 🤖 Generated with Codebuff Co-Authored-By: Codebuff <noreply@codebuff.com>
The 0.2.4 release run built and promoted all eight images and then failed in "Start the published images" before the stack was raised: four connectors were reported missing on the runner although the release had published them. They are behind Compose profiles (`shodan`, `hibp`, `netlas`, `netlas-endpoints`, `shodan-endpoints`), and `docker compose pull` only pulls services of active profiles. It fetched six of the eight images without saying so, and the digest guard, which checks all eight, failed on the rest. The guard was right; the pull was wrong. The job now names the matrix's images once in `RELEASE_IMAGES`, pulls each by name, and the guard reads that list instead of keeping a second copy that can drift. `docker compose pull` still runs afterwards for the datastores. Verified by running the guard script extracted from the workflow against a stubbed `docker`: it passes for all eight and still fails, naming the image, when one is absent. The list matches the build matrix, and the release gates (47 tests) pass. 🤖 Generated with Codebuff Co-Authored-By: Codebuff <noreply@codebuff.com>
The smoke job now names the images it pulls in `RELEASE_IMAGES`, and the digest guard reads that list. Nothing stopped the list from drifting from the matrix it copies: the 0.2.4 run failed because the pull and the guard each kept a list of their own, and only one of them matched what Compose had fetched. `validate_published_image_list` reads the matrix and the job's own env block and reports an image missing from the list, an image the matrix does not produce, a repeated image, a job with no list at all, and a list no step reads. It is wired into the existing `check_release_images.py` entry point, which CI already runs before the build and the tagged workflow runs in `verify-version`, so the gate needs no new invocation. The tests mutate the shipped workflow: dropping `connector-hibp` from the list, adding an unpublished image, repeating one, renaming the variable, hardcoding the pull, and removing the job each have to be reported. 🤖 Generated with Codebuff Co-Authored-By: Codebuff <noreply@codebuff.com>
The pull step named the eight images itself, which is a second copy of what `docker-compose.yml` already says and one more place for a new connector to be forgotten. `docker compose --profile '*' config --images` resolves the image of every service instead, profiled connectors included, and that set is piped to `docker pull`. The wildcard is the point: the four connectors that failed at 0.2.4 sit behind `shodan`, `hibp`, `netlas`, `netlas-endpoints` and `shodan-endpoints`, and without it Compose lists six of the ten images. `RELEASE_IMAGES` stays, because the guard after the pull reads it and the matrix gate keeps it equal to the build matrix; only the pull stops keeping its own copy. A test pins the wildcard so a revert to plain `docker compose pull` is caught in the suite rather than on a release. Checked locally at 0.2.4: the pipeline resolves the eight published images plus `postgres:17-alpine` and `redis:7-alpine`; the matrix gate reports no failures; 23 tests in the file pass. 🤖 Generated with Codebuff Co-Authored-By: Codebuff <noreply@codebuff.com>
The release notes are published from this file, and v0.2.4 has not been published yet: both fixes ship in it, so they belong in its section rather than in Unreleased. One entry covers the connector base layer the image scan demanded, the other the pull that skipped the profiled connectors and the gate that now keeps its list equal to the build matrix. 🤖 Generated with Codebuff Co-Authored-By: Codebuff <noreply@codebuff.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The 0.2.4 release run (#19) built, scanned and promoted all eight images, then failed in
Start the published imagesbefore the stack was ever raised:All four were published — the registry lists
0.2.4, 0.2.4-candidate, latestfor each. Butdocker compose pullonly pulls services of active profiles, and indocker-compose.ymlthese four sit behindshodan,hibp,netlas,netlas-endpointsandshodan-endpoints. The step pulled six of the eight images and said nothing; the digest guard, which checks all eight, failed. The guard was right, the pull was wrong.The job now names the matrix's images once in
RELEASE_IMAGES, pulls each by name, and the guard reads that same list rather than its own hardcoded copy.docker compose pullstill runs afterwards for the datastores the stack starts with.Verification
docker, passes for all eight and still fails namingconnector-hibpwhen one is absent.RELEASE_IMAGESequals the build matrix exactly (8 images).test_release_publish_order_gate.py,test_release_refs_gate.py,test_release_image_matrix.py,test_version_consistency_gate.py: 47 passed.Note for the release
This branch also carries the connector base-layer commit the run needed (
a266fe3, which cannot be pushed tomaindirectly — the branch is protected). After merge,v0.2.4has to move onto the merge commit again and the release re-run. Moving the tag is still safe only because no GitHub Release has been published for it yet.