Repository navigation
fix(ci): stop container image releases from failing silently - #2159
Merged
Merged
Conversation
muscariello
force-pushed
the
ci/harden-release-images
branch
from
October 5, 2026 18:36
338a5ef to
e99a6f2
Compare
The slim-v3.0.0 and slim-v3.0.1 image pushes failed with "installation not allowed to Write organization package". agntcy#2032 added a top-level `permissions: contents: read` to the reusable build workflow, and for a reusable workflow that block is the whole list: `packages: write` from the caller became none. agntcy#2155 put the grant back, but the shape that invited the mistake was still there, and nothing exercised the push path between releases, so the breakage surfaced weeks later at release time. - reusable-docker-build-push: keep the top-level block empty and put the grants on the job, next to the steps that need them. Take the token from `github.token` -- a reusable workflow already gets GITHUB_TOKEN with the caller job's grants -- instead of passing it as a secret, and drop that input. `push` and `ref` become explicit inputs rather than being inferred from the triggering event. Read every pushed tag back from the registry, so a green run means pullable images. - ci: push the `main` tag on merges to main. The same login-build-push-verify path a release takes now runs on every merge, on a moving tag that does not accumulate, so a broken grant fails on the commit that caused it. This also covers the gap that ci never passed `github-token`, so its login would have failed on first push. - release-images: add workflow_dispatch to rebuild an existing tag with the current workflow, since re-running a tag run replays the workflow as it was at the tag. `latest` is opt-in for those manual rebuilds. Open an issue when a release image build fails. Drop the unused `attestations: write` grant. Signed-off-by: Luca Muscariello <muscariello@ieee.org>
muscariello
force-pushed
the
ci/harden-release-images
branch
from
October 5, 2026 18:46
e99a6f2 to
d85e4a0
Compare
muscariello
requested review from
janosSarusiKis,
lgecse,
micpapal and
msardara
October 5, 2026 18:53
msardara
approved these changes
Oct 5, 2026
github-merge-queue
Bot
removed this pull request from the merge queue due to failed status checks
Oct 5, 2026
github-merge-queue
Bot
removed this pull request from the merge queue due to failed status checks
Oct 5, 2026
This was referenced Oct 6, 2026
muscariello
added a commit
to agntcy/slim-staging
that referenced
this pull request
Oct 6, 2026
….1 (#18) ## Summary Point the three SLIM charts at the v3.0.1 images, which are now published and publicly pullable: | image | anonymous pull | |---|---| | `ghcr.io/agntcy/slim:3.0.1` | HTTP 200 | | `ghcr.io/agntcy/slim/control-plane:3.0.1` | HTTP 200 | | `ghcr.io/agntcy/slim/channel-manager:3.0.1` | HTTP 200 | These were missing until today: the `slim-v3.0.0` and `slim-v3.0.1` image builds had failed on a dropped `packages: write` grant (agntcy/slim#2155, hardened in agntcy/slim#2159), so the v3 charts could not be cut. They have now been rebuilt. All three charts default `image.tag` to `.Chart.AppVersion`, so bumping `appVersion` is the whole change. Chart `version` is left to release-please via `.github/release-manifest.json`. Note that `slim-control-plane` was two releases behind its own images — `appVersion: 2.1.0` while 2.3.0 was current — which forced downstream deployments to pin `image.tag` by hand to get a current image. That is no longer necessary. ## Compatibility v3 is a major bump, so I checked the config surface the charts render: - **`topology.links`** (with `domain` / `neighbors`) is still a supported control-plane topology mode. v3 added `segments` and `segments-template` **alongside** it, not in its place, so existing config-managed values keep working. - The data-plane and channel-manager config schemas the charts emit are unchanged. ## Test plan - [x] `helm lint` clean on all three charts - [x] `helm template` renders the expected images: - `ghcr.io/agntcy/slim:3.0.1` - `ghcr.io/agntcy/slim/control-plane:3.0.1` - `ghcr.io/agntcy/slim/channel-manager:3.0.1` - [x] All three images confirmed anonymously pullable from ghcr - [ ] Deploy to dev and confirm the control plane, data plane and channel manager come up and peer ## Unrelated observation `agntcy/slim` main carries changelog entries for a **3.0.2** release dated 2026-09-30 for both `slim` and `control-plane`, but there is no 3.0.2 tag, release or image, and the workspace version is still `3.0.1`. Looks like leftover from the failed release runs that day. Not a blocker here — 3.0.1 is genuinely the latest — but the next release may try to produce 3.0.2 again. Signed-off-by: Luca Muscariello <muscariello@ieee.org>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Root cause
The
slim-v3.0.0andslim-v3.0.1images were never published. Both pushes failed withdenied: installation not allowed to Write organization package.#2032 ("Scorecard: add explicit top-level permissions to workflows missing one") added this to
reusable-docker-build-push.yaml:For a reusable workflow that top-level block is the complete list: every scope it leaves out becomes
none, however much the caller grants.release-images.yamlstill grantedpackages: write, and it was silently dropped.Nothing caught it.
docker/login-actionsucceeds without write access, so the failure lands at the very end of a ~20 minute build. actionlint and zizmor do not model this interaction. And because images are only pushed from release tags, no run exercised the push path between the change and the release.#2155 restored the grant. This PR removes the shape that invited the mistake, and makes the push path run often enough to fail on the commit that breaks it.
Changes
reusable-docker-build-push.yamlgithub.tokeninstead of being passed as a secret, and thegithub-tokeninput is gone. A reusable workflow already receivesGITHUB_TOKENwith the caller job's grants, so the hand-off was never load-bearing — only the caller'spackages: writeis.pushandrefare explicit inputs rather than inferred fromgithub.event_name/github.ref.docker buildx imagetools inspect, so a green run means pullable images.ci.yamlmainnow push themaintag. This runs the same login→build→push→verify path a release takes, so a broken grant or credential fails on the merge that caused it instead of weeks later. The tag moves rather than accumulating, so it costs one tag of storage.ci.yamlnever passedgithub-token, so the login step would have failed with an empty password the first time it tried to push. Taking the token fromgithub.tokenfixes that by construction.release-images.yamlworkflow_dispatchwith ataginput rebuilds an existing release tag using the workflow on the selected branch. Re-running a tag run replays the workflow as it was at the tag, so today there is no way to rebuildslim-v3.0.1with the fix. The tag is validated and passed throughenv.latestis opt-in for manual rebuilds, so rebuilding an older tag does not movelatestback onto it.report-failurejob opens an issue (or comments on an open one) when a release image build fails, since the failure comes after crates, tags and the GitHub release are already out.attestations: write, which nothing uses (provenance: false, no attest step).An earlier revision of this PR added a Python lint check that re-implemented GitHub's permission rules. Dropped in favour of the
:mainpush, which tests the real thing instead of a model of it.Test plan
actionlintandzizmor --offlineclean on all changed workflowsslim-v3.0.1,slim-control-plane-v3.0.1,slim-channel-manager-v3.0.1,control-plane-v2.0.0-alpha.12, crate tags such asslim-config-v0.16.5); malformed input rejectedci-ci / Build SLIM docker imageshould pushghcr.io/agntcy/slim:mainand pass the verify step — this is the real test of the push pathgh workflow run ci-release-images -f tag=slim-v3.0.1 -f latest=truegh workflow run ci-release-images -f tag=slim-control-plane-v3.0.1 -f latest=truegh workflow run ci-release-images -f tag=slim-channel-manager-v3.0.1 -f latest=trueNote:
ci.yamlbuilds only theslimbake target, so the merge-time check covers that image. The control-plane and channel-manager images still build only at release, but they share this workflow, so a permission regression fails onmainfirst.Not in this PR:
slimctl aarch64-pc-windows-msvcinrelease-rust.yamlfails on cosign ("unsupported architecture ARM64"), which skips the Homebrew cask job.