Skip to content

ci(release): stop a missed stable release from looking exactly like a normal merge - #195

Merged
castrojo merged 2 commits into
mainfrom
claude/release-trigger-hardening
Sep 22, 2026
Merged

castrojo merged 2 commits into
mainfrom
claude/release-trigger-hardening

Conversation

@hanthor

@hanthor hanthor commented Sep 19, 2026

Copy link
Copy Markdown
Member

The problem

execute-release.yml decides whether a push to main is the testing→main promotion by matching the head commit subject:

[[ "$MESSAGE" =~ ^chore:\ promote\ testing\ to\ main ]]

The promotion branch carries a commit with exactly that subject, so squashing the single-commit promotion PR inherits it and the check works. A merge commit does not:

Merge pull request #181 from projectbluefin/auto/promote-testing-to-main

And #181's own footer recommends precisely that: gh pr merge <pr-number> --repo projectbluefin/utah --merge --admin. An edited squash subject misses too.

The bad part is not that it can miss — it is that missing is invisible. check-trigger succeeds, execute is skipped, the workflow reports green, main advances, and no :stable tag moves. That is byte for byte what an ordinary merge to main looks like, so nothing distinguishes "the release was never cut" from "there was no release today". The only way to notice is to go and check whether the tag moved.

This is observable on the real thing right now. Run 35454830793, the push of #158 to main:

Execute stable release   completed success
  check-trigger   success
  variants        success
  execute         skipped

Correct there — #158 is not a promotion. And identical to what a silently-failed promotion would produce.

The change

Both parts live in scripts/release-trigger.py so they are testable rather than inline shell.

1. A commit that arrived from auto/promote-testing-to-main is a promotion whatever its subject says. Found by asking which pull requests contain the commit. The lookup is deliberately fail-open to not-a-promotion: a missing token, a rate limit, or a commit with no PR returns an empty list rather than an error, so the worst case is exactly the behaviour we have today — never a spurious release.

2. Every outcome annotates itself. A recognised promotion says so. One recognised by branch rather than subject emits a ::warning and names squashing as the remedy, so the merge-method problem surfaces the first time it happens instead of silently. A push that is not a promotion emits a ::notice stating that no :stable tag was moved and how to cut one. A bare skipped job is the thing being fixed, so silence is not an acceptable outcome on any path.

Only the subject line is read, so a commit body quoting the phrase cannot trigger a release.

Old versus new on the case that motivated it:

OLD logic, merge commit of the promotion PR:
  is-promotion unset -> execute SKIPPED, no :stable, workflow green

NEW logic, same commit, arriving from the promotion branch:
  is-promotion = True
  ::warning title=promotion subject did not match::This commit came from
  auto/promote-testing-to-main, so it is a promotion, but its subject does
  not start with 'chore: promote testing to main'...

Tests

14 tests, driving the real decision function and the real script rather than reading them as text:

  • the merge-commit case above, recognised by branch and warning
  • the " (#181)" suffix GitHub appends to squash subjects
  • a commit body quoting the phrase does not trigger a release
  • a failed gh lookup degrades to not-a-promotion rather than crashing
  • every outcome produces at least one annotation (the silence being fixed)
  • GITHUB_OUTPUT actually receives is-promotion=true/false
  • the workflow runs the script, has pull-requests: read and the env it needs, and the inline regex has not come back

255 tests pass. check_workflow_outputs.py and check-download-integrity.py clean.

Scope note

release-trigger.py is CI-only and is deliberately not added to the Justfile's test -f scripts/... list — that list covers scripts installed into the image, and this one never is.

Not fixed here: the footer recommending --merge is generated by reusable-promote-squash.yml in projectbluefin/actions, outside this repository. This change makes that advice harmless rather than correct; fixing the advice itself needs a PR there.

Found while answering "what's the strategy for promote to stable?" — Utah has never cut either tag (:testing and :stable are both 404 for all four images, checked with an anonymous GHCR pull token), so this would have been a first-release trap rather than a regression.

🤖 Generated with Claude Code

https://claude.ai/code/session_01W6BYLUD6KBzF8rBFXKjP7M


Generated by Claude Code

execute-release.yml decided whether a push to main was the promotion by matching
the head commit subject:

    [[ "$MESSAGE" =~ ^chore:\ promote\ testing\ to\ main ]]

The promotion branch carries a commit with exactly that subject, so squashing the
single-commit promotion PR inherits it and the check works. A merge commit does
not:

    Merge pull request #181 from projectbluefin/auto/promote-testing-to-main

and the promotion PR body recommends precisely that -- `gh pr merge --merge
--admin`. An edited squash subject misses too.

The bad part is not that it can miss. It is that missing is invisible.
check-trigger succeeds, execute is skipped, the workflow reports green, main
advances, and no :stable tag moves. That is byte-for-byte what an ordinary merge
to main looks like, so nothing distinguishes a release that was never cut from a
day with no release. Confirmed on the real thing: run 35454830793 on main reports
`Execute stable release: success` with `execute: skipped`, which is correct
there, and would look the same if a genuine promotion had silently failed to
trigger.

Two changes, both in scripts/release-trigger.py so they can be tested:

A commit that arrived from auto/promote-testing-to-main is a promotion whatever
its subject says, found by asking which pull requests contain the commit. The
lookup is fail-open to not-a-promotion: no token, a rate limit or a commit with
no PR returns an empty list rather than an error, so the worst case is the
behaviour we already had.

And every outcome annotates itself. A recognised promotion says so; one
recognised by branch rather than subject warns, and names squashing as the
remedy; a push that is not a promotion states that no :stable tag was moved and
how to cut one. A bare skipped job is the thing being fixed, so silence is not
an acceptable outcome for any path.

Only the subject line is read, so a commit body quoting the phrase cannot
trigger a release.

14 tests drive the real decision function and the real script, including the
merge-commit case, the " (#181)" suffix GitHub appends, a failed gh lookup, and
that the inline regex has not come back. 255 tests pass; check_workflow_outputs
and check-download-integrity clean.

Not fixed here: the promotion PR footer that recommends --merge lives in
projectbluefin/actions (reusable-promote-squash.yml), outside this repository.
This makes that advice harmless rather than correct.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W6BYLUD6KBzF8rBFXKjP7M
@hanthor

hanthor commented Sep 19, 2026

Copy link
Copy Markdown
Member Author

kernel_cache is red on 11b2e1f, and it is not this PR's failure. Recording that rather than silently re-running.

The job died pulling the base image, before any build logic ran:

Copying blob sha256:3021df79d94319c0f737a2fec0e4614c88cb3ff5bdf1de949b86ea02e62222cd
Error: creating build container: unable to copy from source
docker://quay.io/hummingbird-community/bootc-os@sha256:c5539f9e…:
copying system image from manifest list:
##[error]Process completed with exit code 125

A registry transfer failure from quay.io during blob copy, exit 125 from the container runtime. This PR changes three files — .github/workflows/execute-release.yml, scripts/release-trigger.py and tests/test_release_trigger.py — none of which is in the kernel path, touches Containerfile.kernel, or affects which base image is pulled. kernel_cache was green on main for the last several runs, so it is not broken at the base either.

I tried to re-run the failed job and the API refused with "This workflow is already running" — other jobs in the run are still in flight. I will re-run once it completes; that is the one re-run this warrants, and a second failure I will treat as real rather than as another transfer error.

One note for whoever looks at the log: the string denied: installation not allowed to Write organization package appears in it, which looks alarming and is not an error here. It is the explanatory comment inside build.yml's kernel_cache step being echoed by bash -x — the comment added in #184 describing the fork-push problem. The actual failure is the quay.io transfer above.


Generated by Claude Code

@hivecommons-hive

Copy link
Copy Markdown
Contributor

kernel_cache failed here for a transient registry reason, not because of this diff.

Step "Build the kernel cache image if it is not published yet" died pulling the base image from Quay's CDN:

Error: creating build container: unable to copy from source
docker://quay.io/hummingbird-community/bootc-os@sha256:c5539f9e...:
copying system image from manifest list: writing blob: storing blob to file ...:
happened during read: unexpected EOF
##[error]Process completed with exit code 125

unexpected EOF mid-blob is a CDN/network truncation. kernel_cache is green on main, and this branch (claude/release-trigger-hardening) touches release-trigger logic, which cannot affect a base-image pull.

A re-run should clear it. I don't have gh run rerun in my permitted command surface, so this needs a maintainer to hit re-run on run 35455668951 — there is no code change to make here.

— hive: agent=scanner backend=copilot model=claude-opus-5 copilot=1.0.78

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

security

The PR-lookup trusts a fork-controlled branch name. pull_request_head_refs extracts .[].head.ref (scripts/release-trigger.py:49) from repos/{repo}/commits/{sha}/pulls and decide treats any match of auto/promote-testing-to-main as a promotion (scripts/release-trigger.py:82). head.ref is just the branch name — it is attacker-chosen for fork PRs, and the endpoint associates a merge/squash commit on main with the PR it came from. So an otherwise-innocuous fork PR whose branch happens to be named auto/promote-testing-to-main, once merged, sets is-promotion=true and runs execute (contents: write, packages: write — .github/workflows/execute-release.yml), cutting an unintended :stable release. Reviewers don't normally vet head branch names, so this crosses the fork trust boundary the subject-prefix check never did.

Fix: constrain the jq to same-repo heads, e.g. .[] | select(.head.repo.full_name == "<repo>") | .head.ref (or compare head.label against projectbluefin:auto/promote-testing-to-main), and add a test with a fork ref.

No findings from: correctness, intent-alignment, style, docs-currency.

— hive: agent=reviewer backend=copilot model=claude-fable-5 copilot=1.0.78

head.ref is attacker-chosen on a fork pull request, and GitHub associates the
merge or squash commit on main with the pull request it came from. A fork
branch named auto/promote-testing-to-main therefore made an innocuous merge
look exactly like the promotion, and the execute job cuts a :stable release
with contents: write and packages: write.

Ask the API for the head repository alongside the head ref and keep only refs
whose head.repo.full_name is this repository, so a fork branch name is not a
release trigger. The filter is in Python rather than the jq string so tests
cover it: a fork branch with the promotion name, a deleted fork reporting a
null repo, and the real promotion branch surviving alongside a spoofed one.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed — no findings from correctness, security, intent-alignment, style, docs-currency.

— hive: agent=reviewer backend=copilot model=claude-fable-5 copilot=1.0.78

@castrojo castrojo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed diff and CI runs: hardens release trigger logic to prevent missed stable releases from looking like normal merges. All CI checks pass. Awaiting review from an independent maintainer to satisfy policy.

@Danathar Danathar left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed head 1adcd1c in a local clone. scripts/release-trigger.py keeps the old subject-prefix rule and workflow_dispatch rule, and adds a branch check that only counts pull request heads whose head.repo.full_name equals the repository, so a fork branch named auto/promote-testing-to-main cannot trigger a release. I ran the same gh api commits//pulls query with the PR's jq against the current tip of main and it reports the fork's full name (mrbobbytables/utah), which the filter excludes, so the trust boundary works against the real API. A failed lookup returns an empty list and the push is classified as not a promotion with a notice, which matches today's behaviour. python3 -m unittest tests.test_release_trigger passes (17 tests), the whole suite passes via tests/run_suite.py, and the check-trigger job has the checkout, GH_TOKEN and pull-requests: read permission the script needs. The branch merges cleanly with current main.

@castrojo castrojo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Second review: verified release-trigger.py correctly gates on branch origin and informs annotations without leaking permissions.

@castrojo
castrojo merged commit 3dbaf9c into main Sep 22, 2026
14 checks passed
@castrojo
castrojo deleted the claude/release-trigger-hardening branch September 22, 2026 18:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants