Skip to content

Pull every published image by name, including the profiled connectors - #68

Merged
dyussekeyev merged 5 commits into
mainfrom
fix/smoke-pull-profiled-connectors
Nov 10, 2026
Merged

dyussekeyev merged 5 commits into
mainfrom
fix/smoke-pull-profiled-connectors

Conversation

@dyussekeyev

Copy link
Copy Markdown
Contributor

The 0.2.4 release run (#19) built, scanned and promoted all eight images, then failed in Start the published images before the stack was ever raised:

connector-shodan:0.2.4: not present on this runner
connector-hibp:0.2.4: not present on this runner
connector-intelligence:0.2.4: not present on this runner
connector-exposure-endpoints:0.2.4: not present on this runner

All four were published — the registry lists 0.2.4, 0.2.4-candidate, latest for each. But docker compose pull only pulls services of active profiles, and in docker-compose.yml these four sit behind shodan, hibp, netlas, netlas-endpoints and shodan-endpoints. The step pulled six of the eight images and said nothing; the digest guard, which checks all eight, failed. The guard was right, the pull was wrong.

The job now names the matrix's images once in RELEASE_IMAGES, pulls each by name, and the guard reads that same list rather than its own hardcoded copy. docker compose pull still runs afterwards for the datastores the stack starts with.

Verification

  • The guard script, extracted from the workflow and run against a stubbed docker, passes for all eight and still fails naming connector-hibp when one is absent.
  • RELEASE_IMAGES equals the build matrix exactly (8 images).
  • test_release_publish_order_gate.py, test_release_refs_gate.py, test_release_image_matrix.py, test_version_consistency_gate.py: 47 passed.

Note for the release

This branch also carries the connector base-layer commit the run needed (a266fe3, which cannot be pushed to main directly — the branch is protected). After merge, v0.2.4 has to move onto the merge commit again and the release re-run. Moving the tag is still safe only because no GitHub Release has been published for it yet.

dyussekeyev and others added 5 commits November 11, 2026 00:34
The Release run for v0.2.4 failed on the image scan for all six connectors.
Their shared python:3.12-slim base carries a Debian snapshot whose zlib1g
(1:1.3.dfsg+really1.3.1-1+b1) has two HIGH findings with a fix already
published as +deb13u1, and the gate fails a fixable HIGH rather than
shipping it. backend/Dockerfile already upgrades its base layer for exactly
this reason; the connectors never did.

The upgrade runs before the packages each image installs, so a pinned
version such as dnstwist=0~20250130-1 is still what apt applies last and a
rebuild cannot move it. Verified by building all six images locally
(zlib1g becomes +deb13u1, dnstwist still reports 20250130) and by running
Trivy 0.69.0 with the workflow's own flags on connector-hibp: exit 0.

🤖 Generated with Codebuff
Co-Authored-By: Codebuff <noreply@codebuff.com>
The 0.2.4 release run built and promoted all eight images and then failed in
"Start the published images" before the stack was raised: four connectors were
reported missing on the runner although the release had published them.

They are behind Compose profiles (`shodan`, `hibp`, `netlas`,
`netlas-endpoints`, `shodan-endpoints`), and `docker compose pull` only pulls
services of active profiles. It fetched six of the eight images without saying
so, and the digest guard, which checks all eight, failed on the rest. The guard
was right; the pull was wrong.

The job now names the matrix's images once in `RELEASE_IMAGES`, pulls each by
name, and the guard reads that list instead of keeping a second copy that can
drift. `docker compose pull` still runs afterwards for the datastores.

Verified by running the guard script extracted from the workflow against a
stubbed `docker`: it passes for all eight and still fails, naming the image,
when one is absent. The list matches the build matrix, and the release gates
(47 tests) pass.

🤖 Generated with Codebuff
Co-Authored-By: Codebuff <noreply@codebuff.com>
The smoke job now names the images it pulls in `RELEASE_IMAGES`, and the digest
guard reads that list. Nothing stopped the list from drifting from the matrix it
copies: the 0.2.4 run failed because the pull and the guard each kept a list of
their own, and only one of them matched what Compose had fetched.

`validate_published_image_list` reads the matrix and the job's own env block and
reports an image missing from the list, an image the matrix does not produce, a
repeated image, a job with no list at all, and a list no step reads. It is wired
into the existing `check_release_images.py` entry point, which CI already runs
before the build and the tagged workflow runs in `verify-version`, so the gate
needs no new invocation.

The tests mutate the shipped workflow: dropping `connector-hibp` from the list,
adding an unpublished image, repeating one, renaming the variable, hardcoding
the pull, and removing the job each have to be reported.

🤖 Generated with Codebuff
Co-Authored-By: Codebuff <noreply@codebuff.com>
The pull step named the eight images itself, which is a second copy of what
`docker-compose.yml` already says and one more place for a new connector to be
forgotten. `docker compose --profile '*' config --images` resolves the image of
every service instead, profiled connectors included, and that set is piped to
`docker pull`.

The wildcard is the point: the four connectors that failed at 0.2.4 sit behind
`shodan`, `hibp`, `netlas`, `netlas-endpoints` and `shodan-endpoints`, and
without it Compose lists six of the ten images.

`RELEASE_IMAGES` stays, because the guard after the pull reads it and the matrix
gate keeps it equal to the build matrix; only the pull stops keeping its own
copy. A test pins the wildcard so a revert to plain `docker compose pull` is
caught in the suite rather than on a release.

Checked locally at 0.2.4: the pipeline resolves the eight published images plus
`postgres:17-alpine` and `redis:7-alpine`; the matrix gate reports no failures;
23 tests in the file pass.

🤖 Generated with Codebuff
Co-Authored-By: Codebuff <noreply@codebuff.com>
The release notes are published from this file, and v0.2.4 has not been
published yet: both fixes ship in it, so they belong in its section rather than
in Unreleased. One entry covers the connector base layer the image scan demanded,
the other the pull that skipped the profiled connectors and the gate that now
keeps its list equal to the build matrix.

🤖 Generated with Codebuff
Co-Authored-By: Codebuff <noreply@codebuff.com>
@dyussekeyev
dyussekeyev merged commit 3040a9f into main Nov 10, 2026
12 checks passed
@dyussekeyev
dyussekeyev deleted the fix/smoke-pull-profiled-connectors branch November 11, 2026 14:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant