Skip to content

ci(promote): utah testing → main 2026-09-19 - #181

Closed
github-actions[bot] wants to merge 1 commit into
mainfrom
auto/promote-testing-to-main
Closed

github-actions[bot] wants to merge 1 commit into
mainfrom
auto/promote-testing-to-main

Conversation

@github-actions

@github-actions github-actions Bot commented Sep 19, 2026 •

Copy link
Copy Markdown
Contributor

🦕 Utah testing → stable · 2026-09-19

Auto-maintained by promote-testing-to-main.yml · Updated 2026-09-19T11:32:36Z · Run ↗

Release checklist

❌ Gate blocked

Check Status Details
Digest resolution ❌ failed Failed to resolve one or more digests from testing.
Cosign signatures ❌ failed Skipped signature verification because digest resolution failed.
E2E ⏭️ skipped E2E check disabled by caller. Caller disabled the E2E gate.

Variants being promoted

Variant Tag
utah :testing
utah-nvidia :testing
utah-gaming :testing
utah-nvidia-gaming :testing

Changes since last stable

1 commit ahead of stable · Compare main…testing ↗

Recent commits (showing last 1)
SHA Subject
6519c83 fix: enable PipeWire user units so GNOME screencast can start (#177)

Follow this repository's branch-protection requirements before merging. Force: gh pr merge <pr-number> --repo projectbluefin/utah --merge --admin

Desktop Screenshot

Caution

Auto-merge scheduled for Tuesday 04:00 UTC (bluefin/dakota) / Thursday 04:00 UTC (bluefin-lts).
To block this release: add the do-not-merge label to this PR before that time.
Remove the label when the issue is resolved -- the next weekly window will pick it up automatically.

utah desktop

@github-actions github-actions Bot added the release/blocked Release gate checks failed label Sep 19, 2026
@hanthor

hanthor commented Sep 19, 2026

Copy link
Copy Markdown
Member

Clarifying the branch question, and then why this gate is blocked — the two turn out to be connected.

main is the default branch

From the repository itself, not from inference:

default_branch : main
main       a2cfbc261  protected=True
testing    6519c83a6  protected=False

So main is the default and the protected one; testing is a downstream integration branch and is unprotected. The intended flow is: code lands on main by pull request → sync-main-to-testing.yml pushes main into testing and dispatches a build → build.yml on testing builds and publishes images → post-testing-e2e.yml validates them and tags :testing → this PR brings the commits back to main and promotes the tags.

Right now testing is one commit ahead of main, and that commit is #177:

commits on testing not on main: 1
  6519c83 fix: enable PipeWire user units so GNOME screencast can start (#177)
commits on main not on testing: 0

#177 was merged with testing as its base rather than main, so the PipeWire screencast fix currently exists only on testing. This PR is the mechanism that would carry it to main, which is worth knowing while the gate is blocked: until it merges, that fix is on an unprotected branch and nowhere else. Contributors targeting main is the safer default; I have said so on #177.

Why the gate is blocked

I need to correct something I wrote on #185 an hour ago. I said the promotion failed for want of E2E evidence. The gate's own rows say otherwise:

Check Status Details
Digest resolution ❌ failed Failed to resolve one or more digests from testing.
Cosign signatures ❌ failed Skipped because digest resolution failed.
E2E ⏭️ skipped E2E check disabled by caller.

E2E is not missing — it is switched off deliberately (run_e2e: false in promote-testing-to-main.yml). The failing row is digest resolution, and it is failing for a concrete reason:

utah                   :testing -> HTTP 404
utah-nvidia            :testing -> HTTP 404
utah-gaming            :testing -> HTTP 404
utah-nvidia-gaming     :testing -> HTTP 404

No :testing tag has ever been created for any of the four images. The images themselves are published — the E2E jobs pull utah@sha256:f3ffd108… and friends by digest — but the tag the promotion resolves does not exist.

That tag is written by post-testing-e2e.yml's promote-to-testing job, which needs [resolve, luks]. luks has never fully passed, because two of the four flavors fail; in the latest run promote-to-testing was skipped for exactly that reason.

So the chain is:

gaming flavors cannot mount btrfs → luks fails → promote-to-testing skipped → no :testing tag → digest resolution fails → nothing reaches main, no :stable release.

Which makes #162 the root blocker for the whole release pipeline, not merely a gaming-flavor bug. It needs a reviewer who is not me.


Generated by Claude Code

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

HUMAN DECISION NEEDED — this promotion's own gate shows ❌ (digest resolution failed, cosign verification skipped, E2E disabled), yet the body schedules auto-merge; a human must decide whether to apply do-not-merge or re-run the gate before the merge window.

Findings (correctness perspective, head 1be0677):

  1. Gate vs. auto-merge conflict (PR body). The gate-section reports "Digest resolution ❌ failed" and "Cosign signatures ❌ failed (skipped)", meaning none of the four :testing variants being promoted were resolved or signature-verified. The same body schedules auto-merge for the next weekly window. Promoting unverified digests is exactly what the gate exists to prevent — this needs a human call before Tuesday/Thursday 04:00 UTC.

  2. The code diff itself looks correct. The new system_files/shared/usr/lib/systemd/user-preset/85-utah-desktop.preset:1-3 matches what scripts/configure-services.sh:111-119 enables via systemctl --global (guarded by user_unit_exists, safe under set -e), and Justfile:39,46-48 checks assert both. Docs at docs/skills/desktop-contract.md:60-67 accurately describe the socket-activation rationale. No regressions found in the diff.

Not verified: whether the gate failure is transient (registry hiccup) or the testing images are genuinely unsigned — only the promotion workflow logs can answer that.

— hive: agent=reviewer backend=copilot model=claude-fable-5 copilot=1.0.78

hanthor pushed a commit that referenced this pull request Sep 19, 2026
…158)

Both gaming flavors fail the LUKS install at installer phase 3 with the same
error, most recently in Post-Testing E2E run 35447837973:

  + mount -t btrfs /dev/mapper/fisherman-root /mnt/fisherman-target
  mount: unknown filesystem type 'btrfs'.
  (blkid: /dev/mapper/fisherman-root: LABEL="root" TYPE="btrfs")

mkfs.btrfs succeeds, blkid reads the filesystem back correctly, and the kernel
cannot mount it: the OGC kernel is a source build from upstream
x86_64_defconfig, which omits CONFIG_BTRFS_FS, so no btrfs module exists. The
non-gaming flavors install to a GNOME login in the same run, because only the
gaming flavors ship that kernel.

BTRFS_FS is enabled as a module and added to required_config, so verify_config
rejects a config missing it at config time rather than after a 45-minute build
and a full ISO compose. Verified in kernel_cache on ee4868d: CC [M]
fs/btrfs/super.o and btrfs.ko produced. Every symbol in required_config is set
somewhere in the script, so the new entry cannot pass vacuously.

luks-e2e.sh also gains a modprobe btrfs pre-check in the live guest, so the next
occurrence is a one-line diagnosis instead of a phase-3 installer failure that
reads as a cryptsetup or filesystem problem.

This unblocks promote-to-testing, which has been skipped for want of a passing
luks matrix, which is why no :testing tag exists for any of the four images and
why promotion PR #181 fails digest resolution.

kernel_cache green and build_kernel skipped is the expected shape for a fork
pull request whose diff changes the kernel cache tag (#184): the kernel is
compiled and validated, and the cache is not published.

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

HUMAN DECISION NEEDED — release gate is blocked (digest resolution failed, cosign verification skipped) but auto-merge is scheduled; a human must decide whether this promotion may proceed unverified.

security

  • The PR body's gate table reports Digest resolution ❌ failed and Cosign signatures ❌ skipped ("Skipped signature verification because digest resolution failed"), while the body also schedules auto-merge for the next weekly window. If branch protection does not hard-require the gate check, the :testing images for all four variants would be promoted to stable without signature verification against the identity pinned in .github/workflows/promote-testing-to-main.yml:39-40. E2E is also disabled (run_e2e: false, line 41). Please confirm the gate is a required status check or hold the merge until digests resolve and cosign passes.

The code diff itself is sound: the new user-preset/85-utah-desktop.preset (lines 1-3), the guarded systemctl --global enable calls in scripts/configure-services.sh:111-119, the Justfile checks, and the docs/skills/desktop-contract.md:60-67 update all match promoted commit #177 with no scope creep.

No findings from: intent-alignment, style, docs-currency.

— hive: agent=reviewer backend=copilot model=claude-fable-5 copilot=1.0.78

Danathar added a commit that referenced this pull request Sep 21, 2026
Ghostty is a GTK4 app whose GDK surface needs a working GL context, and
this VM never has one: luks-e2e.sh's VGA_ARGS is plain stdvga, no
virtio-gpu, no /dev/dri. Starting with post-testing-e2e run 35521817359,
every flavor failed identically: 'FAIL: fastfetch output was not visible
in the desktop screenshot', an empty OCR transcript, and a screenshot of
a bare desktop with no terminal on it. luks (utah) and luks (utah-nvidia)
had passed as recently as run 35475091852 the day before, with the same
script and the same QEMU flags, so the regression tracks something
outside this repo rather than a Utah commit.

installed-serial.log confirms it: the passing run's Ghostty
(1.3.2-main-+5252b19) logs 'setting GDK_DISABLE=gles-api,vulkan', hits
'MESA: error: ZINK: failed to choose pdev', and still recovers to
'loaded OpenGL 4.6'. The failing run's Ghostty (1.3.2-main-+27e8b3f,
pulled the next day from tuna-os's unpinned, unversioned master branch)
logs only 'setting GDK_DISABLE=vulkan' -- gles-api dropped -- hits the
same zink error, and this time never recovers: 'gtk_ghostty_surface:
failed to initialize surface err=error.NotInitialized'. Ghostty exits
without ever mapping a window, so fastfetch has nowhere to render and
PR #181 (testing -> main) fails digest resolution because
promote-to-testing never runs (#183).

A flatpak --env override cannot fix this. Ghostty's own source
(src/apprt/gtk/class/application.zig, gtk_ghostty_application scope)
computes GDK_DISABLE from a hardcoded struct literal and calls
setenv("GDK_DISABLE", value, 1) -- overwrite=1 -- right before
gtk_init, unconditionally clobbering any inherited value. An initial
version of this fix pinned a flatpak override for exactly that variable;
reading the upstream source proved it a no-op and it is reverted here.

luks-e2e.sh's terminal autostart entry now launches Ghostty with
LIBGL_ALWAYS_SOFTWARE=1 MESA_LOADER_DRIVER_OVERRIDE=llvmpipe instead.
Ghostty's setenv calls only ever touch LANG, GDK_DEBUG and GDK_DISABLE,
so these two survive into the sandboxed process and force Mesa's
software rasterizer directly, without depending on whichever GDK path
Ghostty's own default happens to select on a given day.

tests/test_iso_ci.py asserts the autostart Exec line carries both
variables and neither script nor a flatpak override still references
GDK_DISABLE. docs/skills/local-testing.md documents why the GDK_DISABLE
route does not work, so a future contributor does not reintroduce it.

Assisted-by: Claude Sonnet 5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: Danathar <Danathar@users.noreply.github.com>

@castrojo castrojo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Promotion PR 181 from auto-maintainer: gate checks failed in source run (digest resolution failed). Cannot land until fresh testing build generates valid digests.

Danathar added a commit that referenced this pull request Sep 21, 2026
Ghostty is a GTK4 app whose GDK surface needs a working GL context, and
this VM never has one: luks-e2e.sh's VGA_ARGS is plain stdvga, no
virtio-gpu, no /dev/dri. Starting with post-testing-e2e run 35521817359,
every flavor failed identically: 'FAIL: fastfetch output was not visible
in the desktop screenshot', an empty OCR transcript, and a screenshot of
a bare desktop with no terminal on it. luks (utah) and luks (utah-nvidia)
had passed as recently as run 35475091852 the day before, with the same
script and the same QEMU flags, so the regression tracks something
outside this repo rather than a Utah commit.

installed-serial.log confirms it: the passing run's Ghostty
(1.3.2-main-+5252b19) logs 'setting GDK_DISABLE=gles-api,vulkan', hits
'MESA: error: ZINK: failed to choose pdev', and still recovers to
'loaded OpenGL 4.6'. The failing run's Ghostty (1.3.2-main-+27e8b3f,
pulled the next day from tuna-os's unpinned, unversioned master branch)
logs only 'setting GDK_DISABLE=vulkan' -- gles-api dropped -- hits the
same zink error, and this time never recovers: 'gtk_ghostty_surface:
failed to initialize surface err=error.NotInitialized'. Ghostty exits
without ever mapping a window, so fastfetch has nowhere to render and
PR #181 (testing -> main) fails digest resolution because
promote-to-testing never runs (#183).

A flatpak --env override cannot fix this. Ghostty's own source
(src/apprt/gtk/class/application.zig, gtk_ghostty_application scope)
computes GDK_DISABLE from a hardcoded struct literal and calls
setenv("GDK_DISABLE", value, 1) -- overwrite=1 -- right before
gtk_init, unconditionally clobbering any inherited value. An initial
version of this fix pinned a flatpak override for exactly that variable;
reading the upstream source proved it a no-op and it is reverted here.

luks-e2e.sh's terminal autostart entry now launches Ghostty with
LIBGL_ALWAYS_SOFTWARE=1 MESA_LOADER_DRIVER_OVERRIDE=llvmpipe instead.
Ghostty's setenv calls only ever touch LANG, GDK_DEBUG and GDK_DISABLE,
so these two survive into the sandboxed process and force Mesa's
software rasterizer directly, without depending on whichever GDK path
Ghostty's own default happens to select on a given day.

tests/test_iso_ci.py asserts the autostart Exec line carries both
variables and neither script nor a flatpak override still references
GDK_DISABLE. docs/skills/local-testing.md documents why the GDK_DISABLE
route does not work, so a future contributor does not reintroduce it.

Assisted-by: Claude Sonnet 5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: Danathar <Danathar@users.noreply.github.com>
castrojo pushed a commit that referenced this pull request Sep 22, 2026
… normal merge (#195)

* Stop a missed stable release from looking exactly like a normal merge

execute-release.yml decided whether a push to main was the promotion by matching
the head commit subject:

    [[ "$MESSAGE" =~ ^chore:\ promote\ testing\ to\ main ]]

The promotion branch carries a commit with exactly that subject, so squashing the
single-commit promotion PR inherits it and the check works. A merge commit does
not:

    Merge pull request #181 from projectbluefin/auto/promote-testing-to-main

and the promotion PR body recommends precisely that -- `gh pr merge --merge
--admin`. An edited squash subject misses too.

The bad part is not that it can miss. It is that missing is invisible.
check-trigger succeeds, execute is skipped, the workflow reports green, main
advances, and no :stable tag moves. That is byte-for-byte what an ordinary merge
to main looks like, so nothing distinguishes a release that was never cut from a
day with no release. Confirmed on the real thing: run 35454830793 on main reports
`Execute stable release: success` with `execute: skipped`, which is correct
there, and would look the same if a genuine promotion had silently failed to
trigger.

Two changes, both in scripts/release-trigger.py so they can be tested:

A commit that arrived from auto/promote-testing-to-main is a promotion whatever
its subject says, found by asking which pull requests contain the commit. The
lookup is fail-open to not-a-promotion: no token, a rate limit or a commit with
no PR returns an empty list rather than an error, so the worst case is the
behaviour we already had.

And every outcome annotates itself. A recognised promotion says so; one
recognised by branch rather than subject warns, and names squashing as the
remedy; a push that is not a promotion states that no :stable tag was moved and
how to cut one. A bare skipped job is the thing being fixed, so silence is not
an acceptable outcome for any path.

Only the subject line is read, so a commit body quoting the phrase cannot
trigger a release.

14 tests drive the real decision function and the real script, including the
merge-commit case, the " (#181)" suffix GitHub appends, a failed gh lookup, and
that the inline regex has not come back. 255 tests pass; check_workflow_outputs
and check-download-integrity clean.

Not fixed here: the promotion PR footer that recommends --merge lives in
projectbluefin/actions (reusable-promote-squash.yml), outside this repository.
This makes that advice harmless rather than correct.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W6BYLUD6KBzF8rBFXKjP7M

* fix(release): ignore fork head refs when recognising the promotion

head.ref is attacker-chosen on a fork pull request, and GitHub associates the
merge or squash commit on main with the pull request it came from. A fork
branch named auto/promote-testing-to-main therefore made an innocuous merge
look exactly like the promotion, and the execute job cuts a :stable release
with contents: write and packages: write.

Ask the API for the head repository alongside the head ref and keep only refs
whose head.repo.full_name is this repository, so a fork branch name is not a
release trigger. The filter is in Python rather than the jq string so tests
cover it: a fork branch with the promotion name, a deleted fork reporting a
null repo, and the real promotion branch surviving alongside a spoofed one.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: ci-maintainer <ci-maintainer@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@hanthor

hanthor commented Sep 22, 2026

Copy link
Copy Markdown
Member

Closing: stale auto-promotion from 2026-09-19 with a blocked gate; promotion runs are ephemeral and should not be merged manually.

@hanthor hanthor closed this Sep 22, 2026
hanthor pushed a commit that referenced this pull request Sep 22, 2026
* fix(iso): force software GL rendering for the E2E test's terminal

Ghostty is a GTK4 app whose GDK surface needs a working GL context, and
this VM never has one: luks-e2e.sh's VGA_ARGS is plain stdvga, no
virtio-gpu, no /dev/dri. Starting with post-testing-e2e run 35521817359,
every flavor failed identically: 'FAIL: fastfetch output was not visible
in the desktop screenshot', an empty OCR transcript, and a screenshot of
a bare desktop with no terminal on it. luks (utah) and luks (utah-nvidia)
had passed as recently as run 35475091852 the day before, with the same
script and the same QEMU flags, so the regression tracks something
outside this repo rather than a Utah commit.

installed-serial.log confirms it: the passing run's Ghostty
(1.3.2-main-+5252b19) logs 'setting GDK_DISABLE=gles-api,vulkan', hits
'MESA: error: ZINK: failed to choose pdev', and still recovers to
'loaded OpenGL 4.6'. The failing run's Ghostty (1.3.2-main-+27e8b3f,
pulled the next day from tuna-os's unpinned, unversioned master branch)
logs only 'setting GDK_DISABLE=vulkan' -- gles-api dropped -- hits the
same zink error, and this time never recovers: 'gtk_ghostty_surface:
failed to initialize surface err=error.NotInitialized'. Ghostty exits
without ever mapping a window, so fastfetch has nowhere to render and
PR #181 (testing -> main) fails digest resolution because
promote-to-testing never runs (#183).

A flatpak --env override cannot fix this. Ghostty's own source
(src/apprt/gtk/class/application.zig, gtk_ghostty_application scope)
computes GDK_DISABLE from a hardcoded struct literal and calls
setenv("GDK_DISABLE", value, 1) -- overwrite=1 -- right before
gtk_init, unconditionally clobbering any inherited value. An initial
version of this fix pinned a flatpak override for exactly that variable;
reading the upstream source proved it a no-op and it is reverted here.

luks-e2e.sh's terminal autostart entry now launches Ghostty with
LIBGL_ALWAYS_SOFTWARE=1 MESA_LOADER_DRIVER_OVERRIDE=llvmpipe instead.
Ghostty's setenv calls only ever touch LANG, GDK_DEBUG and GDK_DISABLE,
so these two survive into the sandboxed process and force Mesa's
software rasterizer directly, without depending on whichever GDK path
Ghostty's own default happens to select on a given day.

tests/test_iso_ci.py asserts the autostart Exec line carries both
variables and neither script nor a flatpak override still references
GDK_DISABLE. docs/skills/local-testing.md documents why the GDK_DISABLE
route does not work, so a future contributor does not reintroduce it.

Assisted-by: Claude Sonnet 5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: Danathar <Danathar@users.noreply.github.com>

* test: assert install-flatpaks.sh doesn't reintroduce GDK_DISABLE

The PR body claimed the regression test covers both the autostart Exec
line and iso/live/src/install-flatpaks.sh, but the assertion only
checked the Exec line. Add the missing assertNotIn so a future
reintroduction of the disproven flatpak override actually fails this
test, matching the claim.

---------

Signed-off-by: Danathar <Danathar@users.noreply.github.com>
Co-authored-by: Danathar <Danathar@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

release/blocked Release gate checks failed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants