Skip to content

πŸ› release: tag the built commit instead of deferring when v5 advances - #17083

Merged
clubanderson merged 1 commit into
v5from
ci/tag-built-ancestor
Nov 7, 2026
Merged

clubanderson merged 1 commit into
v5from
ci/tag-built-ancestor

Conversation

@clubanderson

Copy link
Copy Markdown
Member

Problem

v5.186.4 (2026-10-18) is still the newest tag after ~1,700 commits. #17015 removed the release: none deadlock, but the next layer stalled too: Tagged Release's precheck deferred whenever v5's tip had moved past the commit docker.yml built ("v5 is already at …"). With merges landing every few minutes and a multi-arch build taking ~15–40 min, no run ever saw an unmoved tip, every successor deferred again, and the hourly backstop applied the same tip-only test ("standing down: no successful publishing run for v5 tip"). Meanwhile GitHub's concurrency queue (one pending run per group) was cancelling most docker.yml push builds outright (runs 42400016149, 42401056583, 42403292505), so tip images were rarely published at all.

Fix

tagged-release.yml

  • precheck: release the built SHA when (a) it is an ancestor of v5's tip and (b) the newest v5.* tag is an ancestor of it (no release landed after it; derived version does not already exist). Only (a)/(b) failing defers now, with ::warning::.
  • push_v5: the version tag names the release commit (built SHA + compiled changelog) β€” never the merge commit, so the tag's tree is exactly what the retagged images were built from. The release reaches v5 by fast-forward push when the tip is unmoved, otherwise as a merge-commit PR. 409 "Base branch was modified" is retried within the existing deadline; 409 "Head branch was modified" hard-fails; a CONFLICTING PR (mergeable_state=dirty) hard-fails at once. The post-merge docker.yml dispatch passes release_version only when the merge adds nothing but the changelog (otherwise git describe β†’ v5.x.y-N-gSHA, which is accurate).
  • backstop (decide): picks the newest successfully built ancestor of the tip within the last 100 commits (vars.RELEASE_BACKSTOP_WINDOW), instead of requiring the tip itself.
  • RACES / BACKSTOP header comments rewritten.

docker.yml: push builds get a per-run concurrency group (never cancelled, never queued behind each other); PR builds keep per-PR cancel-in-progress. Out-of-order completion is already handled by publish-image-tags.sh's run-number generation guard.

Invariants kept: never tag a commit whose images weren't published (tag = release commit parented on the built SHA; Tag immutable version images still retags <sha> digests); never tag twice (existing-tag refusal + the new last-tag ancestry check + the tagged-release-v5 concurrency group).

Tests / docs

  • src/scripts/test-release-push-retry.sh: base-moved is retried and still lands with the tag on the release commit; base-moved past the deadline hard-fails; head-moved hard-fails; conflicting PR hard-fails; moved-tip path skips the direct push, releases, omits release_version. All cases pass locally; actionlint clean on both workflows.
  • src/docs/releases.md backstop/tag-target/VERSION sections updated.
  • TRUSTED_SCRIPTS_TREE re-pinned.
  • changelog.d/fixed-release-tag-built-ancestor.md.

@kubestellar-prow kubestellar-prow Bot added the dco-signoff: yes Indicates the PR's author has signed the DCO. label Nov 7, 2026
@kubestellar-prow

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign clubanderson for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@kubestellar-prow kubestellar-prow Bot added the size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. label Nov 7, 2026
@github-actions

github-actions Bot commented Nov 7, 2026

Copy link
Copy Markdown
Contributor

Open PR overlap check

Base branch: v5

Conflicts are checked with every PR merged onto the current tip of v5, so a PR that only needs a rebase is not reported as conflicting with its siblings.

PR Title Relation Files
#17074 ✨ release: make v6 an official release line (v6.x.y tags) CONFLICT .github/scripts/integration-governance.cjs
.github/workflows/tagged-release.yml

@github-actions github-actions Bot added the conflicts-with-open-pr Open PR has a merge conflict with another open PR label Nov 7, 2026
Tagged Release deferred whenever v5's tip had moved past the commit
docker.yml built, expecting the successor run to release. With merges
landing faster than a multi-arch build no run ever saw an unmoved tip,
every successor deferred again, and the hourly backstop applied the same
tip-only test β€” v5 went from v5.186.4 (2026-10-18) to ~1,700 untagged
commits.

- precheck: release the built SHA when it is an ancestor of v5's tip and
  newer than the last v5.* tag; defer only when it is off-branch or a
  release already landed after it (version collision).
- push_v5: the version tag names the release commit (built SHA + changelog),
  never the merge commit; 409 base-moved is retried, head-moved and a
  CONFLICTING PR hard-fail; the post-merge docker dispatch passes
  release_version only when the merge adds nothing but the changelog.
- backstop: pick the newest successfully built ancestor of the tip instead
  of requiring the tip itself.
- docker.yml: push builds get a per-run concurrency group β€” GitHub's queue
  holds one pending run per group and cancels the rest, which is what was
  killing most v5 builds.
- test-release-push-retry.sh covers the new state machine; releases.md and
  the RACES/BACKSTOP headers describe the new rule; TRUSTED_SCRIPTS_TREE
  re-pinned.

Signed-off-by: Andrew Anderson <andy@clubanderson.com>
@clubanderson
clubanderson force-pushed the ci/tag-built-ancestor branch from cd9b8b8 to acc633e Compare November 7, 2026 19:32
@hivecommons-hive

Copy link
Copy Markdown
Contributor
acc633e

ℹ️ Sentinel notice β€” informational; author is trusted (owner), merge is not blocked

Hive flagged this PR (author @clubanderson, head acc633ecd648) because it matches behaviors that can override security controls, escalate privileges or damage the codebase. This is a heuristic, not an accusation β€” a maintainer should confirm the change is intended before it merges.

  • sensitive_path β€” changes 2 sensitive path(s) (Touches a sensitive path (OWNERS, workflows, policies, hive config, security docs, …))
    • .github/workflows/docker.yml
    • .github/workflows/tagged-release.yml
  • ci_gate_weakening β€” CI configuration is deleted, softened or has gate checks removed (Disables or softens CI gates (continue-on-error, || true, removed checks))
    • .github/workflows/tagged-release.yml

Hive did not add the sentinel-alert label because this author is trusted. The finding remains recorded for audit and the dashboard Security tab; set sentinel.trusted_authors_block: true to require the blocking label for trusted authors too.

@clubanderson
clubanderson merged commit dc4f20c into v5 Nov 7, 2026
86 of 87 checks passed
@kubestellar-prow
kubestellar-prow Bot deleted the ci/tag-built-ancestor branch November 7, 2026 19:45
@github-actions

github-actions Bot commented Nov 7, 2026

Copy link
Copy Markdown
Contributor

Thank you for your contribution! Your PR has been merged.

We'd love to hear how your experience was: share feedback

clubanderson added a commit that referenced this pull request Nov 7, 2026
…7128)

A `${{ github.repository }}` mention in a bash comment inside the
push_line step made GitHub evaluate the whole script as an expression;
after the #17083 growth it passed the 21000-character limit, so every
event since bde4411 has reported 'Invalid workflow file … Exceeded max
expression length 21000' for tagged-release.yml and no release has been
considered on v5 or v6. Reword the comment; the script's value comes from
$GITHUB_REPOSITORY already.

Signed-off-by: Andrew Anderson <andy@clubanderson.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

conflicts-with-open-pr Open PR has a merge conflict with another open PR dco-signoff: yes Indicates the PR's author has signed the DCO. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant