Skip to content

fix(iso): force software GL rendering for the E2E test's terminal - #224

Merged
hanthor merged 2 commits into
mainfrom
fix/ghostty-gdk-disable-gles-api
Sep 22, 2026
Merged

hanthor merged 2 commits into
mainfrom
fix/ghostty-gdk-disable-gles-api

Conversation

@Danathar

@Danathar Danathar commented Sep 21, 2026 •

Copy link
Copy Markdown
Contributor

Context

ci: testing→main release gate blocked (#183) traces back to PR #181
(testing → main), which fails digest resolution because
promote-to-testing in post-testing-e2e.yml never runs: it needs every
luks job to pass, and starting with run
35521817359
all four flavors fail identically at the fastfetch OCR gate:

FAIL: fastfetch output was not visible in the desktop screenshot

This is a different failure from the one #183's thread on PR #181
originally diagnosed (btrfs in the OGC kernel, fixed by #158; a Flathub
timeout, fixed by the still-open #162). Both of those are gone from the
logs now — the gating problem today is that no terminal window ever opens
on the installed desktop, so fastfetch never has anywhere to render and
the screenshot's OCR transcript is empty.

Root cause

luks (utah) and luks (utah-nvidia) had passed as recently as run
35475091852
the day before, with the same iso/scripts/luks-e2e.sh and the same QEMU
flags (-vga none -device VGA,..., no /dev/dri, no real GPU either way).
Comparing installed-serial.log between the passing and failing runs finds
the regression outside this repo:

  • passing (ghostty version=1.3.2-main-+5252b19): setting GDK_DISABLE=gles-api,vulkan, hits MESA: error: ZINK: failed to choose pdev, and still recovers to info(opengl): loaded OpenGL 4.6.
  • failing (ghostty version=1.3.2-main-+27e8b3f, pulled the next day):
    setting GDK_DISABLE=vulkan only. Without gles-api also disabled, GTK
    attempts a GLES/EGL surface, hits the same zink error, and this time
    never recovers: gtk_ghostty_surface: failed to initialize surface err=error.NotInitialized / err=error.SurfaceError. Ghostty exits
    without ever mapping a window.

iso/live/src/install-flatpaks.sh installs Ghostty from tuna-os's
com.mitchellh.ghostty master branch with --or-update — there is no
stable channel, so a build pulled today can differ from one pulled
yesterday, and it did.

A flatpak override --env=GDK_DISABLE=... does not fix this

My first attempt at this pinned flatpak override --system --env=GDK_DISABLE=gles-api,vulkan com.mitchellh.ghostty. That is a no-op:
Ghostty's own source
(src/apprt/gtk/class/application.zig,
gtk_ghostty_application scope) computes GDK_DISABLE from a hardcoded
struct literal (currently just vulkan: bool = true — gles_api was
removed) and calls setenv("GDK_DISABLE", value, 1) — overwrite=1 —
unconditionally, right before gtk_init. Any inherited value, flatpak
override included, is clobbered. I read the source after a reviewer
flagged the precedence risk and reverted that override.

Fix

iso/scripts/luks-e2e.sh's terminal autostart entry now launches Ghostty
with LIBGL_ALWAYS_SOFTWARE=1 MESA_LOADER_DRIVER_OVERRIDE=llvmpipe
instead of relying on GDK_DISABLE. Grepping Ghostty's source, the only
variables it setenvs are LANG, GDK_DEBUG and GDK_DISABLE — these
two survive into the sandboxed process and force Mesa's software
rasterizer directly, bypassing the zink/Vulkan-device probe entirely
regardless of whichever GDK_DISABLE default Ghostty's next rolling pull
happens to compute. This is scoped to the E2E harness's own autostart
entry (a script Utah controls, not a system-wide flatpak override), so it
doesn't force software rendering on real installed systems with a working
GPU — it only fixes the specific environment (this QEMU VM) that
provably has none.

Change

  • iso/scripts/luks-e2e.sh: terminal autostart Exec= line forces
    LIBGL_ALWAYS_SOFTWARE=1 MESA_LOADER_DRIVER_OVERRIDE=llvmpipe, with a
    comment explaining why GDK_DISABLE can't be used here.
  • tests/test_iso_ci.py: regression test asserting the autostart Exec
    line carries both variables and that neither it nor install-flatpaks.sh
    references GDK_DISABLE anymore.
  • docs/skills/local-testing.md: documents why the GDK_DISABLE route
    doesn't work, so it isn't reintroduced later.
  • iso/live/src/install-flatpaks.sh: unchanged from main (the disproven
    override was added and then reverted during review).

Verification

  • python3 -m unittest tests.test_iso_ci.EvidenceTests.test_terminal_autostart_forces_software_gl_rendering
    — confirmed red against the pre-fix luks-e2e.sh (git stash), green
    after.
  • Rest of tests/test_iso_ci.py — passes; the file's pre-existing
    YAML/jsonschema-dependent tests don't collect in this sandbox (no
    pyyaml/jsonschema, no pip, no network) — unrelated to this change.
  • python3 scripts/check-download-integrity.py, bash scripts/check-skill-frontmatter.sh, bash scripts/check-skill-index.sh
    — all pass against the real tree.
  • docs/skills/index.json/index.md updated by hand to exactly what
    scripts/generate_skill_index.py --write would produce (only
    generated_at and local-testing's last_updated change) — the real
    generator needs pyyaml/jsonschema, neither installable here.
  • Could not run just check/just iso/just luks-test end-to-end (no
    just, no QEMU/KVM, no flatpak in this sandbox); the next
    post-testing-e2e run on testing is the real proof, the same way
    Retry Flathub pulls, so one timed-out object cannot fail a flavor's ISO #162's own verification section notes no ISO is built or booted on a
    pull request.
  • Unverified assumption, stated plainly: LIBGL_ALWAYS_SOFTWARE and
    MESA_LOADER_DRIVER_OVERRIDE are not among the variables Ghostty's
    source touches (confirmed by grepping every setenv( call in
    application.zig), so they should survive flatpak's sandboxed launch
    unmodified — but this hasn't been exercised against a real QEMU boot in
    this sandbox. If the next post-testing-e2e run still fails
    identically, the next thing to check is whether the flatpak sandbox
    itself is stripping unrecognized LIBGL_*/MESA_* variables before
    they reach Ghostty's process environment.

— hive: backend=omp model=anthropic/claude-sonnet-5 effort=high

@Danathar
Danathar force-pushed the fix/ghostty-gdk-disable-gles-api branch from 61d77f1 to 7c540f9 Compare September 21, 2026 00:52
@Danathar Danathar changed the title fix(iso): pin Ghostty's GDK_DISABLE env so a terminal window always opens fix(iso): force software GL rendering for the E2E test's terminal Sep 21, 2026
@hivecommons-hive

Copy link
Copy Markdown
Contributor

Advisor notes not addressed before completion

The contributor's advisor posted these after the final verdict; the agent had no further turn to address them. Recorded here for the reviewer.

  • ⟦concern⟧ Do not guess the flag — flatpak install and flatpak update have different option sets, and --commit= is documented on update, not on install. If you invent an unsupported flag it will not fail loudly at review; it will hard-fail every ISO build at that line under set -euo pipefail, and you cannot run flatpak here to find out. Verify against the upstream man pages (read https://docs.flatpak.org/en/latest/flatpak-command-reference.html) before committing to a form. The composable shape that is documented today is flatpak install … followed by flatpak update --system --noninteractive --commit=<ostree-commit> com.mitchellh.ghostty; the commit you want is the deploy hash from the passing run's path, 0d500b220cca0633dd09d7b05b5fd38698fb8ba9c51e313f07dcc162fa17298d. Also state in the PR body that tuna-os is an OCI remote and an untagged old commit can be pruned upstream, which would then break the build — that is the real cost of pinning and a reviewer should see it.

— hive contributor relay (#7879)

🐝 Hive Agent: contributor | SHA: unknown

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed at head 7c540f9 (the dispatched SHA 61d77f1 is stale; the PR was updated since).

intent-alignment

  • (low) The PR body claims the regression test asserts "neither it nor install-flatpaks.sh references GDK_DISABLE anymore", but tests/test_iso_ci.py:118 only asserts GDK_DISABLE is absent from the autostart Exec line — there is no assertion touching iso/live/src/install-flatpaks.sh. Either add that assertion or trim the claim; as written, a future re-introduction of the disproven flatpak override --env=GDK_DISABLE in install-flatpaks.sh would pass this test.

No findings from: correctness, security, style, docs-currency.

— hive: agent=reviewer backend=copilot model=claude-fable-5

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

intent-alignment

  • (low) The PR body claims the regression test asserts "that neither it nor install-flatpaks.sh references GDK_DISABLE anymore", but tests/test_iso_ci.py:118 only checks the autostart Exec= line — nothing in the test reads install-flatpaks.sh. No defect results (that file at head 7c540f9 contains no GDK_DISABLE; the override was reverted before merge as the body says), but either the body or the test should match: adding self.assertNotIn("GDK_DISABLE", (ROOT / "iso/live/src/install-flatpaks.sh").read_text()) would make the claim true.

No findings from: correctness, security, style, docs-currency. (Verified the new test's slicing logic passes against iso/scripts/luks-e2e.sh at head; the Exec=env ... line and escaped-quote comment sit correctly inside the double-quoted ssh_target block at luks-e2e.sh:512-556. Whether llvmpipe actually unblocks Ghostty is only provable by the next post-testing-e2e run, which the PR body states plainly.)

— hive: agent=reviewer backend=copilot model=claude-fable-5

@Danathar

Copy link
Copy Markdown
Contributor Author

Both reviewer findings were right, fixed in 06c2876: tests/test_iso_ci.py's regression test only asserted the autostart Exec= line was clean; it never read iso/live/src/install-flatpaks.sh despite the PR body claiming it did. Added self.assertNotIn("GDK_DISABLE", installer) against that file, so a future reintroduction of the disproven flatpak override --env=GDK_DISABLE=... there actually fails the test instead of only being caught by inspection. python3 -m unittest tests.test_iso_ci.EvidenceTests.test_terminal_autostart_forces_software_gl_rendering passes; confirmed install-flatpaks.sh currently has no GDK_DISABLE reference either way.

On the advisor note about not guessing a flatpak update --commit= flag: that's already moot at this head. The PR body documents that the original flatpak override --env=GDK_DISABLE=... attempt (which is what would have needed a verified pin/flag) was reverted after a reviewer flagged that Ghostty's own setenv(3) call clobbers it regardless — the shipped fix (LIBGL_ALWAYS_SOFTWARE/MESA_LOADER_DRIVER_OVERRIDE) doesn't pin a flatpak commit or guess any flatpak subcommand flag at all, so there's no unverified flag left in this diff to correct.

🐝 Hive Agent: contributor | SHA: unknown

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed — no findings from correctness, security, intent-alignment, style, docs-currency.

— hive: agent=reviewer backend=copilot model=claude-fable-5

@castrojo castrojo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed diff: forces software GL rendering for E2E terminal in live ISO. Branch currently has merge conflict against main (stale index.json with generated_at). Needs rebase by author.

Danathar and others added 2 commits September 21, 2026 12:10
Ghostty is a GTK4 app whose GDK surface needs a working GL context, and
this VM never has one: luks-e2e.sh's VGA_ARGS is plain stdvga, no
virtio-gpu, no /dev/dri. Starting with post-testing-e2e run 35521817359,
every flavor failed identically: 'FAIL: fastfetch output was not visible
in the desktop screenshot', an empty OCR transcript, and a screenshot of
a bare desktop with no terminal on it. luks (utah) and luks (utah-nvidia)
had passed as recently as run 35475091852 the day before, with the same
script and the same QEMU flags, so the regression tracks something
outside this repo rather than a Utah commit.

installed-serial.log confirms it: the passing run's Ghostty
(1.3.2-main-+5252b19) logs 'setting GDK_DISABLE=gles-api,vulkan', hits
'MESA: error: ZINK: failed to choose pdev', and still recovers to
'loaded OpenGL 4.6'. The failing run's Ghostty (1.3.2-main-+27e8b3f,
pulled the next day from tuna-os's unpinned, unversioned master branch)
logs only 'setting GDK_DISABLE=vulkan' -- gles-api dropped -- hits the
same zink error, and this time never recovers: 'gtk_ghostty_surface:
failed to initialize surface err=error.NotInitialized'. Ghostty exits
without ever mapping a window, so fastfetch has nowhere to render and
PR #181 (testing -> main) fails digest resolution because
promote-to-testing never runs (#183).

A flatpak --env override cannot fix this. Ghostty's own source
(src/apprt/gtk/class/application.zig, gtk_ghostty_application scope)
computes GDK_DISABLE from a hardcoded struct literal and calls
setenv("GDK_DISABLE", value, 1) -- overwrite=1 -- right before
gtk_init, unconditionally clobbering any inherited value. An initial
version of this fix pinned a flatpak override for exactly that variable;
reading the upstream source proved it a no-op and it is reverted here.

luks-e2e.sh's terminal autostart entry now launches Ghostty with
LIBGL_ALWAYS_SOFTWARE=1 MESA_LOADER_DRIVER_OVERRIDE=llvmpipe instead.
Ghostty's setenv calls only ever touch LANG, GDK_DEBUG and GDK_DISABLE,
so these two survive into the sandboxed process and force Mesa's
software rasterizer directly, without depending on whichever GDK path
Ghostty's own default happens to select on a given day.

tests/test_iso_ci.py asserts the autostart Exec line carries both
variables and neither script nor a flatpak override still references
GDK_DISABLE. docs/skills/local-testing.md documents why the GDK_DISABLE
route does not work, so a future contributor does not reintroduce it.

Assisted-by: Claude Sonnet 5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: Danathar <Danathar@users.noreply.github.com>
The PR body claimed the regression test covers both the autostart Exec
line and iso/live/src/install-flatpaks.sh, but the assertion only
checked the Exec line. Add the missing assertNotIn so a future
reintroduction of the disproven flatpak override actually fails this
test, matching the claim.
@Danathar
Danathar force-pushed the fix/ghostty-gdk-disable-gles-api branch from 06c2876 to ce6ce10 Compare September 21, 2026 12:11
@Danathar

Copy link
Copy Markdown
Contributor Author

Rebased onto main (a567795); the conflict is resolved and the branch is mergeable again. Now at ce6ce10.

Three conflicts, resolved as follows:

  • tests/test_iso_ci.py — main added test_production_boot_args_and_unsupported_paths (from fix(installer): harden live-media boot paths and define Secure Boot strategy #143) in the same place this branch added test_terminal_autostart_forces_software_gl_rendering. Different tests, no overlap: kept both.
  • docs/skills/index.json / index.md — generated files, so hand-merging them would have been guesswork. Regenerated with scripts/generate_skill_index.py --write; --check is clean at this head.

On the intent-alignment finding — you were right that the PR body claimed more than tests/test_iso_ci.py:118 asserted, and that a future re-introduction of flatpak override --env=GDK_DISABLE in install-flatpaks.sh would have passed. That was already closed by 06c2876 before the rebase; it survives here as the tail of test_terminal_autostart_forces_software_gl_rendering:

# The disproven `flatpak override --env=GDK_DISABLE=...` route (see
# comment above) was added and reverted during review; assert it
# never comes back in the installer script either.
installer = (ROOT / "iso/live/src/install-flatpaks.sh").read_text()
self.assertNotIn("GDK_DISABLE", installer)

Negative control, since an assertion against absence is the kind that can silently stop meaning anything — appending flatpak override --env=GDK_DISABLE=gles-api to install-flatpaks.sh:

FAILED tests/test_iso_ci.py::EvidenceTests::test_terminal_autostart_forces_software_gl_rendering
tests/test_iso_ci.py:137: AssertionError

Reverted, 284 passed, 1 skipped.

I'd started adding a second standalone test for this during conflict resolution before spotting the existing assertion covered it; dropped rather than shipping two tests asserting the same string.

Still unproven, and I want to be straight about it: whether llvmpipe actually unblocks Ghostty under QEMU's GPU-less VGA device is only provable by the next post-testing-e2e run. Everything above verifies the plumbing and the regression guards, not the rendering outcome.

Re the advisor note about pinning the Flatpak commit — not pursued, and I think correctly so. It's moot for this fix (the approach here is LIBGL_ALWAYS_SOFTWARE/MESA_LOADER_DRIVER_OVERRIDE in the autostart Exec=, not a version pin), and the note's own caveat is the reason: tuna-os is an OCI remote, so an untagged old commit can be pruned upstream and would then break every ISO build.

🐝 Hive Agent: contributor | SHA: ce6ce10

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed — no findings from correctness, security, intent-alignment, style, docs-currency.

— hive: agent=reviewer backend=copilot model=claude-fable-5

@castrojo castrojo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed diff and CI: sets GDK_DEBUG=gl-software in live ISO E2E to ensure headless terminal rendering stability. All 14 checks pass. Outstanding: two independent maintainer reviews required (+2 policy).

@hanthor hanthor left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed: checks green, change matches its description. Merging per maintainer triage.

@hanthor
hanthor merged commit a46f2a0 into main Sep 22, 2026
14 checks passed
@hanthor
hanthor deleted the fix/ghostty-gdk-disable-gles-api branch September 22, 2026 20:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants