Skip to content

ci: add non-blocking testsuite reporting to post-testing-e2e (#161) - #631

Merged
hanthor merged 3 commits into
mainfrom
ci/consume-testsuite
Oct 12, 2026
Merged

hanthor merged 3 commits into
mainfrom
ci/consume-testsuite

Conversation

@castrojo

@castrojo castrojo commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

Refs #13.

Summary

Adds an asynchronous, non-blocking testsuite reporting matrix job to .github/workflows/post-testing-e2e.yml.

  • Boots each flavor's exact candidate @digest in the projectbluefin/testsuite QEMU VM using pinned e2e.yml@7da50e5b6f165a4704159b0ce5a4898caff6466a (v1).
  • Suites are dynamically configured via scripts/flavors.py suites and config/flavors.json:
    • utah, utah-gaming: smoke,common
    • utah-nvidia, utah-nvidia-gaming: smoke,common,nvidia
  • Non-blocking by design: Neither promote-to-testing nor documentation depend on the testsuite job (needs: [resolve, luks, production-iso] remains intact). Promotion to :testing continues uninterrupted on passing LUKS installer checks, while testsuite results, step summaries, and desktop screenshots attach to the run for QA observability.
  • Updates documentation in docs/skills/ci-workflows.md and docs/skills/flavors.md.

Runs projectbluefin/testsuite e2e suites asynchronously against each
flavor's exact built digest in post-testing-e2e without blocking
:testing promotion or documentation generation.

Assisted-by: Gemini 2.5 Flash via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

HUMAN DECISION NEEDED — @castrojo: #161 asks for a gate (promote-to-testing must need the suite job); this PR makes it non-blocking and says "closes #161". Decide whether non-blocking satisfies #161 or #161 should stay open.

intent-alignment

  • .github/workflows/post-testing-e2e.yml:320 — promote-to-testing: needs: [resolve, luks, production-iso] is unchanged, and testsuite (:303) is not in it. Issue #161's body explicitly requires "promote-to-testing now needs it, so a single failed suite … skips promotion, leaving :testing untouched and un-bypassable." The PR body states non-blocking is "by design", so this is a deliberate deviation, not a bug — but the PR closes the issue it does not implement. Either drop the closing keyword / re-scope #161, or a maintainer confirms non-blocking is the accepted outcome.

security

  • .github/workflows/post-testing-e2e.yml:311 uses: projectbluefin/testsuite/.github/workflows/e2e.yml@v1 and :318 test_ref: v1 pin to a mutable tag while granting packages: write (:306). Repo convention is SHA-pin with a # v1 comment (promote-testing-to-main.yml:35, docs/skills/ci-workflows.md:254 "same SHA pin"). Anyone who can move the v1 tag in testsuite gets a GHCR-write token in utah. Pin both to the commit (da6a487… is current v1).

style

  • .github/workflows/post-testing-e2e.yml:55-61 — the missing digest warning is unreachable: scripts/resolve-e2e-inputs.py:34-35 already fails closed when the artifact set ≠ flavors.py images, and flavors.py suites iterates the same flavors list, so every suite image has a digest by the time this runs. Dead code in a bash step adds maintenance cost; drop it or move the check into the Python.
  • scripts/flavors.py:20 docstring labels suites as "gate" — stale given the non-blocking design.

No findings from: correctness, docs-currency.

Confidence: 2/5 (needs attention) — 1 high finding; 1 medium finding

— hive: agent=reviewer backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

HUMAN DECISION NEEDED — @castrojo confirm that a non-blocking testsuite job is meant to close #161, whose stated contract is that promote-to-testing needs the suite matrix ("un-bypassable"); this PR keeps needs: [resolve, luks, production-iso] (.github/workflows/post-testing-e2e.yml:321).

security

  • .github/workflows/post-testing-e2e.yml:311,316,320 — projectbluefin/testsuite/.github/workflows/e2e.yml@v1 and test_ref: v1 are floating tags, and the call grants packages: write on utah's GITHUB_TOKEN (line 307). Every other uses: in this repo is pinned to a 40-char SHA (post-testing-e2e.yml:30, same across build.yml etc.). A moved v1 tag runs arbitrary workflow code with write access to the very ghcr.io/projectbluefin/utah* packages this workflow is validating. Pin both to a SHA (# v1 comment), as #161 itself did (4c053fd).

intent-alignment

  • #161 asks for a gate; this PR delivers reporting only and says so in its title/body. Deviation is explicit but still closes the issue — needs a maintainer call (see header). No other scope creep.

docs-currency

  • docs/skills/ci-workflows.md:407-419 — the new section was inserted inside ## Verification, so the pre-existing "Front-matter presence checks feed grep -q… SIGPIPE" paragraph (line 417) now reads as part of "Testsuite reporting". Move the new heading below that paragraph.

style

  • scripts/flavors.py:20 — docstring usage line ends with stray word gate (leftover from the gating variant) and breaks the column alignment of the table above it.

No findings from: correctness. (flavors.py suites + tests pass at head; resolve-e2e-inputs.py:34 fails closed so the missing warning branch at post-testing-e2e.yml:55-61 is effectively unreachable, not wrong.)

Confidence: 3/5 (needs attention) — 2 medium findings

— hive: agent=reviewer backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hanthor hanthor left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Request changes at 60cd5c4. Pin both the reusable testsuite workflow and test_ref to reviewed commit SHAs. The current mutable v1 references run with packages:write, contrary to this repository's pinned workflow convention. Keep #13 open: this is reporting, not a promotion gate. The claim that it closes PR #161 also needs reconciliation with #161's gate intent; reporting alone does not establish flavor-specific runtime acceptance. Move the existing frontmatter/SIGPIPE guidance out of the new reporting section.

Reviewed with Codex.

@hivecommons-hive

Copy link
Copy Markdown
Contributor
60cd5c4

ℹ️ Sentinel notice — informational; author is trusted (owner), merge is not blocked

Hive flagged this PR (author @castrojo, head 60cd5c4ae451) because it matches behaviors that can override security controls, escalate privileges or damage the codebase. This is a heuristic, not an accusation — a maintainer should confirm the change is intended before it merges.

  • sensitive_path — changes 1 sensitive path(s) (Touches a sensitive path (OWNERS, workflows, policies, hive config, security docs, …))
    • .github/workflows/post-testing-e2e.yml
  • permission_escalation — widens workflow permissions or uses pull_request_target (Widens GitHub Actions permissions or switches to pull_request_target)
    • .github/workflows/post-testing-e2e.yml

Hive did not add the sentinel-alert label because this author is trusted. The finding remains recorded for audit and the dashboard Security tab; set sentinel.trusted_authors_block: true to require the blocking label for trusted authors too.

Addresses review: pin reusable workflow and test_ref to reviewed commit SHA
(7da50e5b6f165a4704159b0ce5a4898caff6466a), update docstring and reconcile
documentation placement.
@castrojo

Copy link
Copy Markdown
Contributor Author

Pushed commit c73db94 addressing review feedback:

  • Pinned projectbluefin/testsuite/.github/workflows/e2e.yml and test_ref to reviewed commit SHA 7da50e5b6f165a4704159b0ce5a4898caff6466a.
  • Replaced mutable v1 refs.
  • Kept #13 open (re-targeted summary to Refs #13, removed claim closing #161).
  • Reconciled docstring in scripts/flavors.py and moved the documentation block out of the frontmatter/SIGPIPE note in docs/skills/ci-workflows.md.

@hivecommons-hive

Copy link
Copy Markdown
Contributor
c73db94

ℹ️ Sentinel notice — informational; author is trusted (owner), merge is not blocked

Hive flagged this PR (author @castrojo, head c73db94fae79) because it matches behaviors that can override security controls, escalate privileges or damage the codebase. This is a heuristic, not an accusation — a maintainer should confirm the change is intended before it merges.

  • sensitive_path — changes 1 sensitive path(s) (Touches a sensitive path (OWNERS, workflows, policies, hive config, security docs, …))
    • .github/workflows/post-testing-e2e.yml
  • permission_escalation — widens workflow permissions or uses pull_request_target (Widens GitHub Actions permissions or switches to pull_request_target)
    • .github/workflows/post-testing-e2e.yml

Hive did not add the sentinel-alert label because this author is trusted. The finding remains recorded for audit and the dashboard Security tab; set sentinel.trusted_authors_block: true to require the blocking label for trusted authors too.

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

HUMAN DECISION NEEDED — @castrojo decide whether reporting-only satisfies #161/#13, which both ask the suite job to gate :testing promotion (promote-to-testing would need it).

Reviewed at c73db94. This PR adds a job that boots each built image in a virtual machine and runs the project's test suite, then publishes the results — but it does not stop a failing image from being promoted. The two issues it points at ask for exactly that stop.

intent-alignment

  • Title cites #161 and body says "Refs #13". #161's text: "promote-to-testing now needs it, so a single failed suite … skips promotion, leaving :testing untouched and un-bypassable"; #13's title is "gate testing tags on exact-digest VM E2E". The diff deliberately keeps promote-to-testing at needs: [resolve, luks, production-iso] (.github/workflows/post-testing-e2e.yml:321), so the suite job cannot block promotion. The body is honest about this, but it leaves both issues open; a maintainer should confirm non-blocking is the intended first step rather than a scope change.

docs-currency

  • README.md:54 (post-testing-e2e.yml:309), README.md:90 and docs/skills/docs-parity.md:87 (post-testing-e2e.yml:286-309) cite the promote block by line number. After this PR's 34 inserted lines, promote-to-testing starts at :320 and the skopeo copy is at :343. Those citations should be bumped (or made line-free) in the same change.

style

  • .github/workflows/post-testing-e2e.yml:55-61 — the missing digest warning is unreachable. scripts/resolve-e2e-inputs.py:34-35 already raises "build artifacts do not cover the configured flavor set" when any expected image lacks a digest, and expected (resolve-e2e-inputs.py:48-49, from flavors.py images) iterates the same flavors list as flavors.py suites (scripts/flavors.py:68). set -e aborts the step before line 55 runs. Drop the block, or move the check into the script if a soft warning is really wanted.

security

  • Info only: packages: write on the testsuite job (post-testing-e2e.yml:307) hands the utah GITHUB_TOKEN to third-party workflow code with write access to utah's own GHCR packages (utah:testing etc.). It is required — the callee's e2e job declares packages: write (testsuite e2e.yml:199) for a continue-on-error screenshot push — and both uses and test_ref are pinned to the full v1 SHA, so the exposure is bounded. Noting it so the pin is kept on future bumps.

No findings from: correctness. (actionlint clean; tests/test_flavors.py 18 passed at head; matrix context in with: and fromJSON(...)[matrix.image] are valid; nvidia suite dir exists at the pinned testsuite SHA; compose_image: false is tolerated by the callee's e2e if:.)

Confidence: 3/5 (needs attention) — 1 medium finding; 1 perspective requires a human decision

— hive: agent=reviewer backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hanthor hanthor left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Request changes. The requested review feedback is still outstanding: pin both the reusable testsuite workflow and test_ref to reviewed immutable SHAs, and resolve scope with #13/#161. The current job reports results but does not gate promote-to-testing, while those issues ask for promotion to stop on failed flavor-specific runtime suites. Please either implement that gate with tests against a Utah digest or retitle/rescope this as reporting-only after maintainer agreement, correct the README line references, and remove the unreachable missing-digest warning. Keep the write-token workflow pin fixed at a full SHA.

@hanthor

hanthor commented Oct 12, 2026

Copy link
Copy Markdown
Member

Maintainer decision: merge this as an incremental, non-blocking QA-reporting improvement. It exercises each candidate flavor digest and publishes suite diagnostics; it does not satisfy the full #13 acceptance criteria or gate :testing promotion. #13 remains open for the missing enforced promotion gate and runnable NVIDIA evidence. The title/body already describe this change as non-blocking and reference, not close, #13.

@hanthor hanthor left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved as explicitly scoped incremental reporting only. The requested gate for full #13 is not claimed here; issue #13 remains open. CI/image matrix is green.

@hanthor
hanthor merged commit 02c33a9 into main Oct 12, 2026
18 checks passed
@hanthor
hanthor deleted the ci/consume-testsuite branch October 12, 2026 18:55
@github-actions github-actions Bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Oct 12, 2026
@github-actions

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: castrojo, hanthor

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants