Skip to content

🐛 fix(ci): run pinned PR verifier on native ARM64 - #16290

Merged
clubanderson merged 2 commits into
v5from
fix/pr-verifier-native-arm64-20261102
Nov 2, 2026
Merged

clubanderson merged 2 commits into
v5from
fix/pr-verifier-native-arm64-20261102

Conversation

@clubanderson

Copy link
Copy Markdown
Member

Verified cause and publisher ownership

The optional PR Type provider on #16188 failed before validating metadata: run 41499101162/job 125461658689 logged linux/arm64 image versus linux/amd64/v4 host, followed by exec /verifypr: exec format error. The required wrapper was intentionally advisory; its green result was not proof of child-check success. This is not established as the dominant backlog cause.

Public OCI inspection verified the existing pinned ghcr.io/hivecommons/pr-verifier@sha256:8c2b63817c4bfa7de510ab699b154637f27a42f87826c98083903daf2a539943 is a SINGLE linux/arm64 manifest, not an index. Tag v1 resolves to that exact digest. Labels identify upstream kubebuilder-release-tools v0.4.3 and source kubestellar/infra; hivecommons/infra mirrors the same consumer pin. The original upstream GCR v0.4.3 image is retired/404. No unpublished image, guessed digest, or mutable-tag repoint is proposed.

Normal source fix

  • Retain the verified digest/parser version and reuse Hive's existing native ARM64 runner route. Same-repository heads may use HIVE_ARM64_RUNNER; forks always use GitHub-hosted ubuntu-24.04-arm, never the self-hosted override. General runner-pool variables/configuration remain unchanged.
  • Validate actual runner architecture and pulled image platform before executing. Preserve existing pull_request_target trust boundaries, token permissions, advisory policy, current-head checks and read-only mounts.
  • Warn explicitly when execution fails; green wrapper is not child approval. Add seven mocked preflight regressions to the already-CI-wired advisory contract and document ownership/delivery.

Validation and rollout boundary

One bounded independent read-only review found no significant issues. No local test/build/linter/repository guard was run, per AGENTS; CI is the executable verdict. The production pull_request_target verifier uses the BASE workflow, so this PR cannot itself prove the proposed workflow executed live: contract CI validates routing/preflight, and native provider behavior must be observed after legitimate source delivery on a natural event. Existing stale child checks are not overwritten or forced successful.

No manual base refresh/rerun/title edit/label/reconcile/lease release, individual backlog merge, App permission grant, alternate credential, deployment/restart, or capacity/budget change accompanies this source PR. #16277 recovery/operator handoff and #16288 reservation safety remain separate.

Signed-off-by: Andrew Anderson <andy@clubanderson.com>
Signed-off-by: Andrew Anderson <andy@clubanderson.com>
@clubanderson
clubanderson requested a review from hanthor as a code owner November 2, 2026 19:12
@kubestellar-prow kubestellar-prow Bot added the dco-signoff: yes Indicates the PR's author has signed the DCO. label Nov 2, 2026
@kubestellar-prow

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign nicholasjackson for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@kubestellar-prow kubestellar-prow Bot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Nov 2, 2026
@clubanderson
clubanderson removed the request for review from Danathar November 2, 2026 19:13
@github-actions

github-actions Bot commented Nov 2, 2026

Copy link
Copy Markdown
Contributor

Open PR overlap check

Base branch: v5

Conflicts are checked with every PR merged onto the current tip of v5, so a PR that only needs a rebase is not reported as conflicting with its siblings.

No conflicting open PRs or same-file overlaps were found.

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Read the diff against v5 and independently checked the premise. Looks correct to me.

Verified:

  • ghcr.io/hivecommons/pr-verifier@sha256:8c2b6381… is a single application/vnd.docker.distribution.manifest.v2+json (no manifests list) whose config blob says os=linux arch=arm64; tag v1 returns the same docker-content-digest. So the exec format error on the amd64 pool is real and QEMU-less docker run cannot fix it — native ARM64 is the only option without republishing.
  • .github/workflows/pr-verifier.yml:20 — the selector reuses the existing vars.HIVE_ARM64_RUNNER JSON convention (docker.yml:284, src/docs/ci-runner-labels.md:133), and the fork branch is a literal ubuntu-24.04-arm, so a pull_request_target from a fork can never land on a repository-variable runner. permissions: is unchanged.
  • pr-verifier.yml:46-56 — preflight runs under the default bash -e, so a failed docker pull or a non-linux/arm64 inspect result exits non-zero and continue-on-error routes it to the "unavailable or incompatible" warning rather than running the verifier. The seven mocked cases in src/deploy/test_pr_verifier_advisory.sh:152-160 cover exactly those branches (arch gate before pull, platform gate after), and test-native-queue-workflows.py:71 now finds the verifier step by name instead of assuming it is last — needed because Report verifier failure is now the final step.

One low-severity docs note: src/docs/ci-runner-labels.md:133-134 still describes HIVE_ARM64_RUNNER as "only an explicit hosted-runner fallback" for QEMU builds, while this PR's comment, test (test_pr_verifier_advisory.sh:89) and hivecommons-migration.md call it a "self-hosted override". Since the fleet is amd64 (LKE), same-repo verifier runs will in practice be GitHub-hosted too — which also means the "Required on v5 … survive a GitHub-hosted outage" rationale the old comment carried no longer holds for this check. That's a deliberate trade-off given the image, but worth one sentence in ci-runner-labels.md so the two docs agree on what the variable is for.

— hive: agent=reviewer-2 backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive

hivecommons-hive Bot commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor

Hive status — updated in place; edits do not notify. Full history is in the Activity Monitor.

Review findings from reviewer (folded here because this head already carries its hive review):

Conclusion: The diagnosis is right and the code does what it says; the one thing I cannot settle is a runner-availability trade-off on a workflow file, so I am routing that to an approver rather than judging it.

What I verified independently (before reading the body):

  • The pinned ghcr.io/hivecommons/pr-verifier@sha256:8c2b6381… really is a single linux/arm64 manifest (application/vnd.docker.distribution.manifest.v2+json, config blob os=linux architecture=arm64), not an index. So on the amd64 pool the pull "succeeds" and the run dies with exec format error — which is also why the optional PR Type check is red on 🌱 build(deps): update github.com/charmbracelet/x/ansi to 0.11.9 in /src #16188.
  • .github/workflows/pr-verifier.yml:46-56 — the arch/platform preflight lives inside the continue-on-error: true pull step, so a mismatch still yields the explicit "SKIPPED, not passed" warning and a green advisory wrapper rather than a hard failure. Consistent with the existing 🐛 bug: PR Verifier's fatal image pull fails the check on registry access, defeating the continue-on-error on the verification itself #6085 design.
  • .github/workflows/pr-verifier.yml:21 — fork heads (head.repo.full_name != github.repository) always land on the string 'ubuntu-24.04-arm', never on fromJSON(vars.HIVE_ARM64_RUNNER), so the pull_request_target isolation for forks is preserved.
  • src/deploy/test_pr_verifier_advisory.sh:137-178 executes the real run: script under a mocked docker for 7 arch/platform/pull-exit cases; .github/scripts/test-native-queue-workflows.py:71-73 is updated to find the verifier step by name instead of steps[-1] (correct now that a "Report verifier failure" step follows it).

The open point (medium, not a defect in the diff):

  • Old pr-verifier.yml:19-21 (origin/v5) carried: "Required on v5: keep on the self-hosted pool so a GitHub-hosted runner outage does not make otherwise-mergeable PRs impossible to merge." The new selector at line 21 defaults to GitHub-hosted ubuntu-24.04-arm whenever HIVE_ARM64_RUNNER is unset, and src/docs/ci-runner-labels.md:133-134 says to set that variable "only as an explicit hosted-runner fallback". So in the documented default configuration this job moves off self-hosted — the dependency the removed comment existed to avoid. I could not read branch protection (403 for the App) to confirm the job is still required. If it is, either set HIVE_ARM64_RUNNER to a self-hosted ARM64 label set before merging, or accept the hosted dependency knowingly; if it is not required, this is moot and the change looks correct to me.

@clubanderson this PR is held at stage/071-needs-author and the hive will not merge it yet.

Why:

  • the review swarm returned requires_human: it has a question for you, not a fix for the hive
  • the reviewer asked you: @hanthor should decide whether the required-on-v5 verify / verify PR contents job may move from the self-hosted pool to GitHub-hosted ubuntu-24.04-arm by default (it only stays self-hosted when HIVE_ARM64_RUNNER is set, which src/docs/ci-runner-labels.md:133 describes as an explicit fallback, not the norm).

Next: the hive does not push to your branch. Push a new head (or reply here); CI and the review swarm re-run automatically and the PR returns to the normal path.

@hivecommons-hive

Copy link
Copy Markdown
Contributor
e413f7e

ℹ️ Sentinel notice — informational; author is trusted (owner), merge is not blocked

Hive flagged this PR (author @clubanderson, head e413f7e35f8d) because it matches behaviors that can override security controls, escalate privileges or damage the codebase. This is a heuristic, not an accusation — a maintainer should confirm the change is intended before it merges.

  • sensitive_path — changes 2 sensitive path(s) (Touches a sensitive path (OWNERS, workflows, policies, hive config, security docs, …))
    • .github/workflows/pr-verifier.yml
    • src/deploy/test_pr_verifier_advisory.sh

Hive did not add the sentinel-alert label because this author is trusted. The finding remains recorded for audit and the dashboard Security tab; set sentinel.trusted_authors_block: true to require the blocking label for trusted authors too.

@hivecommons-hive

Copy link
Copy Markdown
Contributor

Needs a decision, not a merge:

  • the reviewer swarm returned requires_human @e413f7e
  • the review found: the reviewer marked this change high-risk
  • reviewer question: @hanthor should decide whether the required-on-v5 verify / verify PR contents job may move from the self-hosted pool to GitHub-hosted ubuntu-24.04-arm by default (it only stays self-hosted when HIVE_ARM64_RUNNER is set, which src/docs/ci-runner-labels.md:133 describes as an explicit fallback, not the norm).
  • the PR edits CI workflow / action files: awaiting owner approval — a hive owner / trusted merger's GitHub Approve review on the current head (or a dashboard /hive decision approve) releases the fence and the hive merges it; approval on an earlier head does not count, so re-approve after a push

Head: verdict requires_human @e413f7e, 76/76 checks green, DCO yes, soak 10m0s elapsed.

How to release it:

  • A hive owner / trusted merger submits a GitHub Approve review on the current head (or approves from the Hive dashboard, which posts /hive decision approve on their behalf); that explicit approval releases the workflow-path fence and the lane merges on its next pass once CI, soak and the other gates are clear. Approval on an earlier head does not carry over: re-approve after a push.
  • After reading the verdict, a trusted merger can submit a GitHub Approve review for the current head and the lane clears the hold on its next pass; /hive approve replies on a PR are acknowledged but not consumed. An ordinary contributor cannot release their own PR.

Clear the cause and the hive re-evaluates on its next pass; the stage moves to stage/090-ready-for-human-merge, stage/095-pre-merge-soak or the merge queue on its own. Approval does not bypass CI, hold labels or other merge gates.

@clubanderson

Copy link
Copy Markdown
Member Author

Decision (owner): approved — GitHub-hosted ubuntu-24.04-arm is the default for verify PR contents; self-hosted only when HIVE_ARM64_RUNNER is set, per ci-runner-labels.md. Proceed to merge when green.

@clubanderson
clubanderson merged commit f3527d5 into v5 Nov 2, 2026
90 of 93 checks passed
@kubestellar-prow
kubestellar-prow Bot deleted the fix/pr-verifier-native-arm64-20261102 branch November 2, 2026 22:58
@github-actions

github-actions Bot commented Nov 2, 2026

Copy link
Copy Markdown
Contributor

Thank you for your contribution! Your PR has been merged.

We'd love to hear how your experience was: share feedback

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

dco-signoff: yes Indicates the PR's author has signed the DCO. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant