Skip to content

fix(ci): stop container image releases from failing silently - #2159

Merged
muscariello merged 1 commit into
agntcy:mainfrom
muscariello:ci/harden-release-images
Oct 6, 2026
Merged

muscariello merged 1 commit into
agntcy:mainfrom
muscariello:ci/harden-release-images

Conversation

@muscariello

@muscariello muscariello commented Oct 5, 2026 •

Copy link
Copy Markdown
Member

Root cause

The slim-v3.0.0 and slim-v3.0.1 images were never published. Both pushes failed with denied: installation not allowed to Write organization package.

#2032 ("Scorecard: add explicit top-level permissions to workflows missing one") added this to reusable-docker-build-push.yaml:

permissions:
  contents: read

For a reusable workflow that top-level block is the complete list: every scope it leaves out becomes none, however much the caller grants. release-images.yaml still granted packages: write, and it was silently dropped.

Nothing caught it. docker/login-action succeeds without write access, so the failure lands at the very end of a ~20 minute build. actionlint and zizmor do not model this interaction. And because images are only pushed from release tags, no run exercised the push path between the change and the release.

#2155 restored the grant. This PR removes the shape that invited the mistake, and makes the push path run often enough to fail on the commit that breaks it.

Changes

reusable-docker-build-push.yaml

  • Top-level block stays empty; the grants live on the job, next to the steps that need them.
  • The token comes from github.token instead of being passed as a secret, and the github-token input is gone. A reusable workflow already receives GITHUB_TOKEN with the caller job's grants, so the hand-off was never load-bearing — only the caller's packages: write is.
  • push and ref are explicit inputs rather than inferred from github.event_name/github.ref.
  • After pushing, every tag is read back with docker buildx imagetools inspect, so a green run means pullable images.

ci.yaml

  • Merges to main now push the main tag. This runs the same login→build→push→verify path a release takes, so a broken grant or credential fails on the merge that caused it instead of weeks later. The tag moves rather than accumulating, so it costs one tag of storage.
  • This also closes a latent gap: ci.yaml never passed github-token, so the login step would have failed with an empty password the first time it tried to push. Taking the token from github.token fixes that by construction.

release-images.yaml

  • workflow_dispatch with a tag input rebuilds an existing release tag using the workflow on the selected branch. Re-running a tag run replays the workflow as it was at the tag, so today there is no way to rebuild slim-v3.0.1 with the fix. The tag is validated and passed through env.
  • latest is opt-in for manual rebuilds, so rebuilding an older tag does not move latest back onto it.
  • New report-failure job opens an issue (or comments on an open one) when a release image build fails, since the failure comes after crates, tags and the GitHub release are already out.
  • Dropped attestations: write, which nothing uses (provenance: false, no attest step).

An earlier revision of this PR added a Python lint check that re-implemented GitHub's permission rules. Dropped in favour of the :main push, which tests the real thing instead of a model of it.

Test plan

  • actionlint and zizmor --offline clean on all changed workflows
  • Tag regex and target/version parsing checked locally against real tags (slim-v3.0.1, slim-control-plane-v3.0.1, slim-channel-manager-v3.0.1, control-plane-v2.0.0-alpha.12, crate tags such as slim-config-v0.16.5); malformed input rejected
  • On merge, ci-ci / Build SLIM docker image should push ghcr.io/agntcy/slim:main and pass the verify step — this is the real test of the push path
  • Then rebuild the missing v3.0.1 images:
    • gh workflow run ci-release-images -f tag=slim-v3.0.1 -f latest=true
    • gh workflow run ci-release-images -f tag=slim-control-plane-v3.0.1 -f latest=true
    • gh workflow run ci-release-images -f tag=slim-channel-manager-v3.0.1 -f latest=true

Note: ci.yaml builds only the slim bake target, so the merge-time check covers that image. The control-plane and channel-manager images still build only at release, but they share this workflow, so a permission regression fails on main first.

Not in this PR: slimctl aarch64-pc-windows-msvc in release-rust.yaml fails on cosign ("unsupported architecture ARM64"), which skips the Homebrew cask job.

@muscariello
muscariello requested a review from a team as a code owner October 5, 2026 18:36
@muscariello
muscariello force-pushed the ci/harden-release-images branch from 338a5ef to e99a6f2 Compare October 5, 2026 18:36
The slim-v3.0.0 and slim-v3.0.1 image pushes failed with "installation
not allowed to Write organization package". agntcy#2032 added a top-level
`permissions: contents: read` to the reusable build workflow, and for a
reusable workflow that block is the whole list: `packages: write` from
the caller became none. agntcy#2155 put the grant back, but the shape that
invited the mistake was still there, and nothing exercised the push path
between releases, so the breakage surfaced weeks later at release time.

- reusable-docker-build-push: keep the top-level block empty and put the
  grants on the job, next to the steps that need them. Take the token
  from `github.token` -- a reusable workflow already gets GITHUB_TOKEN
  with the caller job's grants -- instead of passing it as a secret, and
  drop that input. `push` and `ref` become explicit inputs rather than
  being inferred from the triggering event. Read every pushed tag back
  from the registry, so a green run means pullable images.
- ci: push the `main` tag on merges to main. The same
  login-build-push-verify path a release takes now runs on every merge,
  on a moving tag that does not accumulate, so a broken grant fails on
  the commit that caused it. This also covers the gap that ci never
  passed `github-token`, so its login would have failed on first push.
- release-images: add workflow_dispatch to rebuild an existing tag with
  the current workflow, since re-running a tag run replays the workflow
  as it was at the tag. `latest` is opt-in for those manual rebuilds.
  Open an issue when a release image build fails. Drop the unused
  `attestations: write` grant.

Signed-off-by: Luca Muscariello <muscariello@ieee.org>
@muscariello
muscariello added this pull request to the merge queue Oct 5, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Oct 5, 2026
@muscariello
muscariello added this pull request to the merge queue Oct 5, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Oct 5, 2026
@muscariello
muscariello added this pull request to the merge queue Oct 6, 2026
Merged via the queue into agntcy:main with commit 3b6a3f6 Oct 6, 2026
18 checks passed
@muscariello
muscariello deleted the ci/harden-release-images branch October 6, 2026 07:29
muscariello added a commit to agntcy/slim-staging that referenced this pull request Oct 6, 2026
….1 (#18)

## Summary

Point the three SLIM charts at the v3.0.1 images, which are now
published and publicly pullable:

| image | anonymous pull |
|---|---|
| `ghcr.io/agntcy/slim:3.0.1` | HTTP 200 |
| `ghcr.io/agntcy/slim/control-plane:3.0.1` | HTTP 200 |
| `ghcr.io/agntcy/slim/channel-manager:3.0.1` | HTTP 200 |

These were missing until today: the `slim-v3.0.0` and `slim-v3.0.1`
image builds had failed on a dropped `packages: write` grant
(agntcy/slim#2155, hardened in agntcy/slim#2159), so the v3 charts could
not be cut. They have now been rebuilt.

All three charts default `image.tag` to `.Chart.AppVersion`, so bumping
`appVersion` is the whole change. Chart `version` is left to
release-please via `.github/release-manifest.json`.

Note that `slim-control-plane` was two releases behind its own images —
`appVersion: 2.1.0` while 2.3.0 was current — which forced downstream
deployments to pin `image.tag` by hand to get a current image. That is
no longer necessary.

## Compatibility

v3 is a major bump, so I checked the config surface the charts render:

- **`topology.links`** (with `domain` / `neighbors`) is still a
supported control-plane topology mode. v3 added `segments` and
`segments-template` **alongside** it, not in its place, so existing
config-managed values keep working.
- The data-plane and channel-manager config schemas the charts emit are
unchanged.

## Test plan

- [x] `helm lint` clean on all three charts
- [x] `helm template` renders the expected images:
  - `ghcr.io/agntcy/slim:3.0.1`
  - `ghcr.io/agntcy/slim/control-plane:3.0.1`
  - `ghcr.io/agntcy/slim/channel-manager:3.0.1`
- [x] All three images confirmed anonymously pullable from ghcr
- [ ] Deploy to dev and confirm the control plane, data plane and
channel manager come up and peer

## Unrelated observation

`agntcy/slim` main carries changelog entries for a **3.0.2** release
dated 2026-09-30 for both `slim` and `control-plane`, but there is no
3.0.2 tag, release or image, and the workspace version is still `3.0.1`.
Looks like leftover from the failed release runs that day. Not a blocker
here — 3.0.1 is genuinely the latest — but the next release may try to
produce 3.0.2 again.

Signed-off-by: Luca Muscariello <muscariello@ieee.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants