Repository navigation
ci: add non-blocking testsuite reporting to post-testing-e2e (#161) - #631
Conversation
Runs projectbluefin/testsuite e2e suites asynchronously against each flavor's exact built digest in post-testing-e2e without blocking :testing promotion or documentation generation. Assisted-by: Gemini 2.5 Flash via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
HUMAN DECISION NEEDED — @castrojo: #161 asks for a gate (promote-to-testing must need the suite job); this PR makes it non-blocking and says "closes #161". Decide whether non-blocking satisfies #161 or #161 should stay open.
intent-alignment
.github/workflows/post-testing-e2e.yml:320—promote-to-testing: needs: [resolve, luks, production-iso]is unchanged, andtestsuite(:303) is not in it. Issue #161's body explicitly requires "promote-to-testingnow needs it, so a single failed suite … skips promotion, leaving :testing untouched and un-bypassable." The PR body states non-blocking is "by design", so this is a deliberate deviation, not a bug — but the PR closes the issue it does not implement. Either drop the closing keyword / re-scope #161, or a maintainer confirms non-blocking is the accepted outcome.
security
.github/workflows/post-testing-e2e.yml:311uses: projectbluefin/testsuite/.github/workflows/e2e.yml@v1and:318test_ref: v1pin to a mutable tag while grantingpackages: write(:306). Repo convention is SHA-pin with a# v1comment (promote-testing-to-main.yml:35,docs/skills/ci-workflows.md:254"same SHA pin"). Anyone who can move thev1tag in testsuite gets a GHCR-write token in utah. Pin both to the commit (da6a487…is currentv1).
style
.github/workflows/post-testing-e2e.yml:55-61— themissingdigest warning is unreachable:scripts/resolve-e2e-inputs.py:34-35already fails closed when the artifact set ≠flavors.py images, andflavors.py suitesiterates the sameflavorslist, so every suite image has a digest by the time this runs. Dead code in a bash step adds maintenance cost; drop it or move the check into the Python.scripts/flavors.py:20docstring labelssuitesas "gate" — stale given the non-blocking design.
No findings from: correctness, docs-currency.
Confidence: 2/5 (needs attention) — 1 high finding; 1 medium finding
— hive: agent=reviewer backend=copilot model=claude-fable-5.1 copilot=1.0.88
There was a problem hiding this comment.
HUMAN DECISION NEEDED — @castrojo confirm that a non-blocking testsuite job is meant to close #161, whose stated contract is that promote-to-testing needs the suite matrix ("un-bypassable"); this PR keeps needs: [resolve, luks, production-iso] (.github/workflows/post-testing-e2e.yml:321).
security
.github/workflows/post-testing-e2e.yml:311,316,320—projectbluefin/testsuite/.github/workflows/e2e.yml@v1andtest_ref: v1are floating tags, and the call grantspackages: writeon utah'sGITHUB_TOKEN(line 307). Every otheruses:in this repo is pinned to a 40-char SHA (post-testing-e2e.yml:30, same across build.yml etc.). A movedv1tag runs arbitrary workflow code with write access to the veryghcr.io/projectbluefin/utah*packages this workflow is validating. Pin both to a SHA (# v1comment), as #161 itself did (4c053fd).
intent-alignment
- #161 asks for a gate; this PR delivers reporting only and says so in its title/body. Deviation is explicit but still closes the issue — needs a maintainer call (see header). No other scope creep.
docs-currency
docs/skills/ci-workflows.md:407-419— the new section was inserted inside## Verification, so the pre-existing "Front-matter presence checks feedgrep -q… SIGPIPE" paragraph (line 417) now reads as part of "Testsuite reporting". Move the new heading below that paragraph.
style
scripts/flavors.py:20— docstring usage line ends with stray wordgate(leftover from the gating variant) and breaks the column alignment of the table above it.
No findings from: correctness. (flavors.py suites + tests pass at head; resolve-e2e-inputs.py:34 fails closed so the missing warning branch at post-testing-e2e.yml:55-61 is effectively unreachable, not wrong.)
Confidence: 3/5 (needs attention) — 2 medium findings
— hive: agent=reviewer backend=copilot model=claude-fable-5.1 copilot=1.0.88
hanthor
left a comment
There was a problem hiding this comment.
Request changes at 60cd5c4. Pin both the reusable testsuite workflow and test_ref to reviewed commit SHAs. The current mutable v1 references run with packages:write, contrary to this repository's pinned workflow convention. Keep #13 open: this is reporting, not a promotion gate. The claim that it closes PR #161 also needs reconciliation with #161's gate intent; reporting alone does not establish flavor-specific runtime acceptance. Move the existing frontmatter/SIGPIPE guidance out of the new reporting section.
Reviewed with Codex.
60cd5c4
ℹ️ Sentinel notice — informational; author is trusted (owner), merge is not blockedHive flagged this PR (author @castrojo, head
Hive did not add the |
Addresses review: pin reusable workflow and test_ref to reviewed commit SHA (7da50e5b6f165a4704159b0ce5a4898caff6466a), update docstring and reconcile documentation placement.
|
Pushed commit c73db94 addressing review feedback:
|
c73db94
ℹ️ Sentinel notice — informational; author is trusted (owner), merge is not blockedHive flagged this PR (author @castrojo, head
Hive did not add the |
There was a problem hiding this comment.
HUMAN DECISION NEEDED — @castrojo decide whether reporting-only satisfies #161/#13, which both ask the suite job to gate :testing promotion (promote-to-testing would need it).
Reviewed at c73db94. This PR adds a job that boots each built image in a virtual machine and runs the project's test suite, then publishes the results — but it does not stop a failing image from being promoted. The two issues it points at ask for exactly that stop.
intent-alignment
- Title cites #161 and body says "Refs #13". #161's text: "
promote-to-testingnow needs it, so a single failed suite … skips promotion, leaving :testing untouched and un-bypassable"; #13's title is "gate testing tags on exact-digest VM E2E". The diff deliberately keepspromote-to-testingatneeds: [resolve, luks, production-iso](.github/workflows/post-testing-e2e.yml:321), so the suite job cannot block promotion. The body is honest about this, but it leaves both issues open; a maintainer should confirm non-blocking is the intended first step rather than a scope change.
docs-currency
README.md:54(post-testing-e2e.yml:309),README.md:90anddocs/skills/docs-parity.md:87(post-testing-e2e.yml:286-309) cite the promote block by line number. After this PR's 34 inserted lines,promote-to-testingstarts at:320and theskopeo copyis at:343. Those citations should be bumped (or made line-free) in the same change.
style
.github/workflows/post-testing-e2e.yml:55-61— themissingdigest warning is unreachable.scripts/resolve-e2e-inputs.py:34-35already raises"build artifacts do not cover the configured flavor set"when any expected image lacks a digest, andexpected(resolve-e2e-inputs.py:48-49, fromflavors.py images) iterates the sameflavorslist asflavors.py suites(scripts/flavors.py:68).set -eaborts the step before line 55 runs. Drop the block, or move the check into the script if a soft warning is really wanted.
security
- Info only:
packages: writeon thetestsuitejob (post-testing-e2e.yml:307) hands the utahGITHUB_TOKENto third-party workflow code with write access to utah's own GHCR packages (utah:testingetc.). It is required — the callee'se2ejob declarespackages: write(testsuite e2e.yml:199) for acontinue-on-errorscreenshot push — and bothusesandtest_refare pinned to the fullv1SHA, so the exposure is bounded. Noting it so the pin is kept on future bumps.
No findings from: correctness. (actionlint clean; tests/test_flavors.py 18 passed at head; matrix context in with: and fromJSON(...)[matrix.image] are valid; nvidia suite dir exists at the pinned testsuite SHA; compose_image: false is tolerated by the callee's e2e if:.)
Confidence: 3/5 (needs attention) — 1 medium finding; 1 perspective requires a human decision
— hive: agent=reviewer backend=copilot model=claude-fable-5.1 copilot=1.0.88
hanthor
left a comment
There was a problem hiding this comment.
Request changes. The requested review feedback is still outstanding: pin both the reusable testsuite workflow and test_ref to reviewed immutable SHAs, and resolve scope with #13/#161. The current job reports results but does not gate promote-to-testing, while those issues ask for promotion to stop on failed flavor-specific runtime suites. Please either implement that gate with tests against a Utah digest or retitle/rescope this as reporting-only after maintainer agreement, correct the README line references, and remove the unreachable missing-digest warning. Keep the write-token workflow pin fixed at a full SHA.
|
Maintainer decision: merge this as an incremental, non-blocking QA-reporting improvement. It exercises each candidate flavor digest and publishes suite diagnostics; it does not satisfy the full #13 acceptance criteria or gate :testing promotion. #13 remains open for the missing enforced promotion gate and runnable NVIDIA evidence. The title/body already describe this change as non-blocking and reference, not close, #13. |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: castrojo, hanthor The full list of commands accepted by this bot can be found here. Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
Refs #13.
Summary
Adds an asynchronous, non-blocking
testsuitereporting matrix job to.github/workflows/post-testing-e2e.yml.@digestin theprojectbluefin/testsuiteQEMU VM using pinnede2e.yml@7da50e5b6f165a4704159b0ce5a4898caff6466a(v1).scripts/flavors.py suitesandconfig/flavors.json:utah,utah-gaming:smoke,commonutah-nvidia,utah-nvidia-gaming:smoke,common,nvidiapromote-to-testingnordocumentationdepend on thetestsuitejob (needs: [resolve, luks, production-iso]remains intact). Promotion to:testingcontinues uninterrupted on passing LUKS installer checks, while testsuite results, step summaries, and desktop screenshots attach to the run for QA observability.docs/skills/ci-workflows.mdanddocs/skills/flavors.md.