Repository navigation
ci(promote): utah testing → main 2026-09-19 - #181
github-actions[bot] wants to merge 1 commit into
Conversation
|
Clarifying the branch question, and then why this gate is blocked — the two turn out to be connected.
|
| Check | Status | Details |
|---|---|---|
| Digest resolution | ❌ failed | Failed to resolve one or more digests from testing. |
| Cosign signatures | ❌ failed | Skipped because digest resolution failed. |
| E2E | ⏭️ skipped | E2E check disabled by caller. |
E2E is not missing — it is switched off deliberately (run_e2e: false in promote-testing-to-main.yml). The failing row is digest resolution, and it is failing for a concrete reason:
utah :testing -> HTTP 404
utah-nvidia :testing -> HTTP 404
utah-gaming :testing -> HTTP 404
utah-nvidia-gaming :testing -> HTTP 404
No :testing tag has ever been created for any of the four images. The images themselves are published — the E2E jobs pull utah@sha256:f3ffd108… and friends by digest — but the tag the promotion resolves does not exist.
That tag is written by post-testing-e2e.yml's promote-to-testing job, which needs [resolve, luks]. luks has never fully passed, because two of the four flavors fail; in the latest run promote-to-testing was skipped for exactly that reason.
So the chain is:
gaming flavors cannot mount btrfs →
luksfails →promote-to-testingskipped → no:testingtag → digest resolution fails → nothing reachesmain, no:stablerelease.
Which makes #162 the root blocker for the whole release pipeline, not merely a gaming-flavor bug. It needs a reviewer who is not me.
Generated by Claude Code
There was a problem hiding this comment.
HUMAN DECISION NEEDED — this promotion's own gate shows ❌ (digest resolution failed, cosign verification skipped, E2E disabled), yet the body schedules auto-merge; a human must decide whether to apply do-not-merge or re-run the gate before the merge window.
Findings (correctness perspective, head 1be0677):
-
Gate vs. auto-merge conflict (PR body). The gate-section reports "Digest resolution ❌ failed" and "Cosign signatures ❌ failed (skipped)", meaning none of the four
:testingvariants being promoted were resolved or signature-verified. The same body schedules auto-merge for the next weekly window. Promoting unverified digests is exactly what the gate exists to prevent — this needs a human call before Tuesday/Thursday 04:00 UTC. -
The code diff itself looks correct. The new
system_files/shared/usr/lib/systemd/user-preset/85-utah-desktop.preset:1-3matches whatscripts/configure-services.sh:111-119enables viasystemctl --global(guarded byuser_unit_exists, safe underset -e), andJustfile:39,46-48checks assert both. Docs atdocs/skills/desktop-contract.md:60-67accurately describe the socket-activation rationale. No regressions found in the diff.
Not verified: whether the gate failure is transient (registry hiccup) or the testing images are genuinely unsigned — only the promotion workflow logs can answer that.
— hive: agent=reviewer backend=copilot model=claude-fable-5 copilot=1.0.78
…158) Both gaming flavors fail the LUKS install at installer phase 3 with the same error, most recently in Post-Testing E2E run 35447837973: + mount -t btrfs /dev/mapper/fisherman-root /mnt/fisherman-target mount: unknown filesystem type 'btrfs'. (blkid: /dev/mapper/fisherman-root: LABEL="root" TYPE="btrfs") mkfs.btrfs succeeds, blkid reads the filesystem back correctly, and the kernel cannot mount it: the OGC kernel is a source build from upstream x86_64_defconfig, which omits CONFIG_BTRFS_FS, so no btrfs module exists. The non-gaming flavors install to a GNOME login in the same run, because only the gaming flavors ship that kernel. BTRFS_FS is enabled as a module and added to required_config, so verify_config rejects a config missing it at config time rather than after a 45-minute build and a full ISO compose. Verified in kernel_cache on ee4868d: CC [M] fs/btrfs/super.o and btrfs.ko produced. Every symbol in required_config is set somewhere in the script, so the new entry cannot pass vacuously. luks-e2e.sh also gains a modprobe btrfs pre-check in the live guest, so the next occurrence is a one-line diagnosis instead of a phase-3 installer failure that reads as a cryptsetup or filesystem problem. This unblocks promote-to-testing, which has been skipped for want of a passing luks matrix, which is why no :testing tag exists for any of the four images and why promotion PR #181 fails digest resolution. kernel_cache green and build_kernel skipped is the expected shape for a fork pull request whose diff changes the kernel cache tag (#184): the kernel is compiled and validated, and the cache is not published.
There was a problem hiding this comment.
HUMAN DECISION NEEDED — release gate is blocked (digest resolution failed, cosign verification skipped) but auto-merge is scheduled; a human must decide whether this promotion may proceed unverified.
security
- The PR body's gate table reports Digest resolution ❌ failed and Cosign signatures ❌ skipped ("Skipped signature verification because digest resolution failed"), while the body also schedules auto-merge for the next weekly window. If branch protection does not hard-require the gate check, the
:testingimages for all four variants would be promoted to stable without signature verification against the identity pinned in.github/workflows/promote-testing-to-main.yml:39-40. E2E is also disabled (run_e2e: false, line 41). Please confirm the gate is a required status check or hold the merge until digests resolve and cosign passes.
The code diff itself is sound: the new user-preset/85-utah-desktop.preset (lines 1-3), the guarded systemctl --global enable calls in scripts/configure-services.sh:111-119, the Justfile checks, and the docs/skills/desktop-contract.md:60-67 update all match promoted commit #177 with no scope creep.
No findings from: intent-alignment, style, docs-currency.
— hive: agent=reviewer backend=copilot model=claude-fable-5 copilot=1.0.78
Ghostty is a GTK4 app whose GDK surface needs a working GL context, and this VM never has one: luks-e2e.sh's VGA_ARGS is plain stdvga, no virtio-gpu, no /dev/dri. Starting with post-testing-e2e run 35521817359, every flavor failed identically: 'FAIL: fastfetch output was not visible in the desktop screenshot', an empty OCR transcript, and a screenshot of a bare desktop with no terminal on it. luks (utah) and luks (utah-nvidia) had passed as recently as run 35475091852 the day before, with the same script and the same QEMU flags, so the regression tracks something outside this repo rather than a Utah commit. installed-serial.log confirms it: the passing run's Ghostty (1.3.2-main-+5252b19) logs 'setting GDK_DISABLE=gles-api,vulkan', hits 'MESA: error: ZINK: failed to choose pdev', and still recovers to 'loaded OpenGL 4.6'. The failing run's Ghostty (1.3.2-main-+27e8b3f, pulled the next day from tuna-os's unpinned, unversioned master branch) logs only 'setting GDK_DISABLE=vulkan' -- gles-api dropped -- hits the same zink error, and this time never recovers: 'gtk_ghostty_surface: failed to initialize surface err=error.NotInitialized'. Ghostty exits without ever mapping a window, so fastfetch has nowhere to render and PR #181 (testing -> main) fails digest resolution because promote-to-testing never runs (#183). A flatpak --env override cannot fix this. Ghostty's own source (src/apprt/gtk/class/application.zig, gtk_ghostty_application scope) computes GDK_DISABLE from a hardcoded struct literal and calls setenv("GDK_DISABLE", value, 1) -- overwrite=1 -- right before gtk_init, unconditionally clobbering any inherited value. An initial version of this fix pinned a flatpak override for exactly that variable; reading the upstream source proved it a no-op and it is reverted here. luks-e2e.sh's terminal autostart entry now launches Ghostty with LIBGL_ALWAYS_SOFTWARE=1 MESA_LOADER_DRIVER_OVERRIDE=llvmpipe instead. Ghostty's setenv calls only ever touch LANG, GDK_DEBUG and GDK_DISABLE, so these two survive into the sandboxed process and force Mesa's software rasterizer directly, without depending on whichever GDK path Ghostty's own default happens to select on a given day. tests/test_iso_ci.py asserts the autostart Exec line carries both variables and neither script nor a flatpak override still references GDK_DISABLE. docs/skills/local-testing.md documents why the GDK_DISABLE route does not work, so a future contributor does not reintroduce it. Assisted-by: Claude Sonnet 5 via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Danathar <Danathar@users.noreply.github.com>
castrojo
left a comment
There was a problem hiding this comment.
Promotion PR 181 from auto-maintainer: gate checks failed in source run (digest resolution failed). Cannot land until fresh testing build generates valid digests.
Ghostty is a GTK4 app whose GDK surface needs a working GL context, and this VM never has one: luks-e2e.sh's VGA_ARGS is plain stdvga, no virtio-gpu, no /dev/dri. Starting with post-testing-e2e run 35521817359, every flavor failed identically: 'FAIL: fastfetch output was not visible in the desktop screenshot', an empty OCR transcript, and a screenshot of a bare desktop with no terminal on it. luks (utah) and luks (utah-nvidia) had passed as recently as run 35475091852 the day before, with the same script and the same QEMU flags, so the regression tracks something outside this repo rather than a Utah commit. installed-serial.log confirms it: the passing run's Ghostty (1.3.2-main-+5252b19) logs 'setting GDK_DISABLE=gles-api,vulkan', hits 'MESA: error: ZINK: failed to choose pdev', and still recovers to 'loaded OpenGL 4.6'. The failing run's Ghostty (1.3.2-main-+27e8b3f, pulled the next day from tuna-os's unpinned, unversioned master branch) logs only 'setting GDK_DISABLE=vulkan' -- gles-api dropped -- hits the same zink error, and this time never recovers: 'gtk_ghostty_surface: failed to initialize surface err=error.NotInitialized'. Ghostty exits without ever mapping a window, so fastfetch has nowhere to render and PR #181 (testing -> main) fails digest resolution because promote-to-testing never runs (#183). A flatpak --env override cannot fix this. Ghostty's own source (src/apprt/gtk/class/application.zig, gtk_ghostty_application scope) computes GDK_DISABLE from a hardcoded struct literal and calls setenv("GDK_DISABLE", value, 1) -- overwrite=1 -- right before gtk_init, unconditionally clobbering any inherited value. An initial version of this fix pinned a flatpak override for exactly that variable; reading the upstream source proved it a no-op and it is reverted here. luks-e2e.sh's terminal autostart entry now launches Ghostty with LIBGL_ALWAYS_SOFTWARE=1 MESA_LOADER_DRIVER_OVERRIDE=llvmpipe instead. Ghostty's setenv calls only ever touch LANG, GDK_DEBUG and GDK_DISABLE, so these two survive into the sandboxed process and force Mesa's software rasterizer directly, without depending on whichever GDK path Ghostty's own default happens to select on a given day. tests/test_iso_ci.py asserts the autostart Exec line carries both variables and neither script nor a flatpak override still references GDK_DISABLE. docs/skills/local-testing.md documents why the GDK_DISABLE route does not work, so a future contributor does not reintroduce it. Assisted-by: Claude Sonnet 5 via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Danathar <Danathar@users.noreply.github.com>
… normal merge (#195) * Stop a missed stable release from looking exactly like a normal merge execute-release.yml decided whether a push to main was the promotion by matching the head commit subject: [[ "$MESSAGE" =~ ^chore:\ promote\ testing\ to\ main ]] The promotion branch carries a commit with exactly that subject, so squashing the single-commit promotion PR inherits it and the check works. A merge commit does not: Merge pull request #181 from projectbluefin/auto/promote-testing-to-main and the promotion PR body recommends precisely that -- `gh pr merge --merge --admin`. An edited squash subject misses too. The bad part is not that it can miss. It is that missing is invisible. check-trigger succeeds, execute is skipped, the workflow reports green, main advances, and no :stable tag moves. That is byte-for-byte what an ordinary merge to main looks like, so nothing distinguishes a release that was never cut from a day with no release. Confirmed on the real thing: run 35454830793 on main reports `Execute stable release: success` with `execute: skipped`, which is correct there, and would look the same if a genuine promotion had silently failed to trigger. Two changes, both in scripts/release-trigger.py so they can be tested: A commit that arrived from auto/promote-testing-to-main is a promotion whatever its subject says, found by asking which pull requests contain the commit. The lookup is fail-open to not-a-promotion: no token, a rate limit or a commit with no PR returns an empty list rather than an error, so the worst case is the behaviour we already had. And every outcome annotates itself. A recognised promotion says so; one recognised by branch rather than subject warns, and names squashing as the remedy; a push that is not a promotion states that no :stable tag was moved and how to cut one. A bare skipped job is the thing being fixed, so silence is not an acceptable outcome for any path. Only the subject line is read, so a commit body quoting the phrase cannot trigger a release. 14 tests drive the real decision function and the real script, including the merge-commit case, the " (#181)" suffix GitHub appends, a failed gh lookup, and that the inline regex has not come back. 255 tests pass; check_workflow_outputs and check-download-integrity clean. Not fixed here: the promotion PR footer that recommends --merge lives in projectbluefin/actions (reusable-promote-squash.yml), outside this repository. This makes that advice harmless rather than correct. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01W6BYLUD6KBzF8rBFXKjP7M * fix(release): ignore fork head refs when recognising the promotion head.ref is attacker-chosen on a fork pull request, and GitHub associates the merge or squash commit on main with the pull request it came from. A fork branch named auto/promote-testing-to-main therefore made an innocuous merge look exactly like the promotion, and the execute job cuts a :stable release with contents: write and packages: write. Ask the API for the head repository alongside the head ref and keep only refs whose head.repo.full_name is this repository, so a fork branch name is not a release trigger. The filter is in Python rather than the jq string so tests cover it: a fork branch with the promotion name, a deleted fork reporting a null repo, and the real promotion branch surviving alongside a spoofed one. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: ci-maintainer <ci-maintainer@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
Closing: stale auto-promotion from 2026-09-19 with a blocked gate; promotion runs are ephemeral and should not be merged manually. |
* fix(iso): force software GL rendering for the E2E test's terminal Ghostty is a GTK4 app whose GDK surface needs a working GL context, and this VM never has one: luks-e2e.sh's VGA_ARGS is plain stdvga, no virtio-gpu, no /dev/dri. Starting with post-testing-e2e run 35521817359, every flavor failed identically: 'FAIL: fastfetch output was not visible in the desktop screenshot', an empty OCR transcript, and a screenshot of a bare desktop with no terminal on it. luks (utah) and luks (utah-nvidia) had passed as recently as run 35475091852 the day before, with the same script and the same QEMU flags, so the regression tracks something outside this repo rather than a Utah commit. installed-serial.log confirms it: the passing run's Ghostty (1.3.2-main-+5252b19) logs 'setting GDK_DISABLE=gles-api,vulkan', hits 'MESA: error: ZINK: failed to choose pdev', and still recovers to 'loaded OpenGL 4.6'. The failing run's Ghostty (1.3.2-main-+27e8b3f, pulled the next day from tuna-os's unpinned, unversioned master branch) logs only 'setting GDK_DISABLE=vulkan' -- gles-api dropped -- hits the same zink error, and this time never recovers: 'gtk_ghostty_surface: failed to initialize surface err=error.NotInitialized'. Ghostty exits without ever mapping a window, so fastfetch has nowhere to render and PR #181 (testing -> main) fails digest resolution because promote-to-testing never runs (#183). A flatpak --env override cannot fix this. Ghostty's own source (src/apprt/gtk/class/application.zig, gtk_ghostty_application scope) computes GDK_DISABLE from a hardcoded struct literal and calls setenv("GDK_DISABLE", value, 1) -- overwrite=1 -- right before gtk_init, unconditionally clobbering any inherited value. An initial version of this fix pinned a flatpak override for exactly that variable; reading the upstream source proved it a no-op and it is reverted here. luks-e2e.sh's terminal autostart entry now launches Ghostty with LIBGL_ALWAYS_SOFTWARE=1 MESA_LOADER_DRIVER_OVERRIDE=llvmpipe instead. Ghostty's setenv calls only ever touch LANG, GDK_DEBUG and GDK_DISABLE, so these two survive into the sandboxed process and force Mesa's software rasterizer directly, without depending on whichever GDK path Ghostty's own default happens to select on a given day. tests/test_iso_ci.py asserts the autostart Exec line carries both variables and neither script nor a flatpak override still references GDK_DISABLE. docs/skills/local-testing.md documents why the GDK_DISABLE route does not work, so a future contributor does not reintroduce it. Assisted-by: Claude Sonnet 5 via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Danathar <Danathar@users.noreply.github.com> * test: assert install-flatpaks.sh doesn't reintroduce GDK_DISABLE The PR body claimed the regression test covers both the autostart Exec line and iso/live/src/install-flatpaks.sh, but the assertion only checked the Exec line. Add the missing assertNotIn so a future reintroduction of the disproven flatpak override actually fails this test, matching the claim. --------- Signed-off-by: Danathar <Danathar@users.noreply.github.com> Co-authored-by: Danathar <Danathar@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
🦕 Utah testing → stable · 2026-09-19
Release checklist
❌ Gate blocked
Variants being promoted
utah:testingutah-nvidia:testingutah-gaming:testingutah-nvidia-gaming:testingChanges since last stable
1 commit ahead of stable · Compare main…testing ↗
Recent commits (showing last 1)
6519c83Follow this repository's branch-protection requirements before merging. Force:
gh pr merge <pr-number> --repo projectbluefin/utah --merge --adminDesktop Screenshot
Caution
Auto-merge scheduled for Tuesday 04:00 UTC (bluefin/dakota) / Thursday 04:00 UTC (bluefin-lts).
To block this release: add the
do-not-mergelabel to this PR before that time.Remove the label when the issue is resolved -- the next weekly window will pick it up automatically.