Skip to content

Retry Flathub pulls, so one timed-out object cannot fail a flavor's ISO - #162

Merged
castrojo merged 8 commits into
mainfrom
claude/iso-btrfs-and-ocr
Sep 21, 2026
Merged

castrojo merged 8 commits into
mainfrom
claude/iso-btrfs-and-ocr

Conversation

@hanthor

@hanthor hanthor commented Sep 19, 2026 •

Copy link
Copy Markdown
Member

Reduced in scope. The btrfs half of this branch landed in #158 (kylerankin's, which diagnosed it first and also carries a modprobe btrfs pre-check this never had), so what remains is the utah-gaming Flathub failure plus the test coverage.

The failure

Failed to install org.mozilla.firefox: While pulling
app/org.mozilla.firefox/x86_64/stable from remote flathub: While fetching
.../88680ed7...commitmeta: [28] Timeout was reached

Three minutes forty-four into composing the ISO, with nothing wrong in the image. The Flathub pulls are the largest network operation in the whole build — Firefox alone is around 200 MB — and they had no retry, while the curl a few lines above them has had one all along.

iso/live/src/install-flatpaks.sh gains a retry_flatpak helper (three attempts, 15s then 30s backoff) and all three installs go through it. flatpak resumes a partial pull from the local repository, so a retry re-fetches only what is still missing, and every install passes --or-update, which is what makes a retry a no-op for a ref that already completed.

That last part was not uniformly true when this PR opened: the installer-local install lacked --or-update, which would have let an attempt that installed the app but still exited nonzero fail attempts 2 and 3 with "already installed" — a retry converting a flaky success into a hard failure. Caught in review, fixed in 738d64e, and now asserted by a test that fails against the previous line.

What I got wrong about this, worth recording

I originally presented this as the fix for utah-gaming and the btrfs change as the fix for utah-nvidia-gaming, as though they were independent flavor-specific defects. They are not. Post-Testing E2E run 35447837973 shows both gaming flavors failing on the identical btrfs mount error — both ship the OGC kernel. In the earlier run the Flathub timeout simply hit utah-gaming first and masked the btrfs failure underneath it.

So this is a fix for a flake that was hiding the real bug, not a co-equal second cause. It is still worth having — a timed-out object should not fail a compose — but #158 was the load-bearing change and I had the priority backwards.

Tests

tests/test_iso_ci.py, 250 in the suite. Both new classes drive the real code rather than reading it as text:

  • FlatpakRetryTests extracts the actual retry_flatpak function and runs it against a stub flatpak with sleep stubbed out, asserting a transient failure is retried to success (rc=0 attempts=3) and that a persistent one still fails (rc=1 attempts=3) — resilience must not mean swallowing a repository that is genuinely gone. Two contract assertions cover the call sites: every network install goes through the wrapper, and every wrapped install carries --or-update.
  • OgcKernelConfigGateTests drives the real verify_config against generated configs, with a subtest dropping each symbol in required_config in turn. Since BTRFS_FS is now on main, this gates main's code rather than this branch's.

Also in this PR

A comment in scripts/install-ogc-kernel.sh recording the btrfs failure verbatim: the mount error, why that list had never needed a filesystem before (the missing KMS device stopped the ISO test at phase 2, so it was written for the live-boot and LUKS paths), and that BTRFS_FS selects its own crc/zlib/lzo/zstd/raid6 helpers so naming it is enough for olddefconfig. No production change — 5a261af removed a require_config line I had added that verify_config already covers two lines later, and whose fixed diagnostic prints BPF and binder settings that are noise for a btrfs failure.

Verification

Note that no ISO is built or booted on a pull request — dispatch-iso is gated on github.ref == 'refs/heads/testing' — so CI here proves the images build and the host suite passes, not that the retry recovers a real Flathub timeout. That is only observable on testing.

🤖 Generated with Claude Code

https://claude.ai/code/session_01W6BYLUD6KBzF8rBFXKjP7M

…reen edge

The first ISO end-to-end run after #94 got much further and split the four
flavors into two different failures. Both are fixed here.

gaming and nvidia-gaming now boot the live session -- the simpledrm fix
worked -- and die one phase later, in the installer:

    mkfs.btrfs -f -L root /dev/mapper/fisherman-root          (ok)
    mount -t btrfs /dev/mapper/fisherman-root /mnt/...
    mount: unknown filesystem type 'btrfs'
    fisherman: fatal: mounting root: ... exit status 32

Utah installs to btrfs on LUKS and x86_64 defconfig has no BTRFS_FS, so
the OGC kernel can format a root volume and then not mount it. The gaming
ISO boots the OGC kernel, so this hits the installer as well as the
installed system. required_config was written for the live-boot and LUKS
paths because no gaming flavor had ever reached the phase that formats a
disk; it now also names the filesystem the disk is made of. BTRFS_FS
selects its own crc, zlib, lzo, zstd, raid6 and xor helpers, so
olddefconfig pulls the rest in, and a require_config gate catches it if
it does not. This rotates the kernel-cache tag, so the next gaming build
compiles once.

utah and nvidia go all the way to phase 6 -- installed, unlocked, logged
in, GSConnect ACTIVE, wayland session -- and fail on the screenshot gate
alone. The screenshot is perfect; the gate could not read it. From the
run's own fastfetch-ocr.txt:

    TAH-E2E-FASTFETCH
    Utah (Version: testing-20260918-2216657)
    Linux 7.1.8-100.fc43.x86_64

The marker lost its first glyph, and every fastfetch label -- OS:,
Kernel:, Uptime: -- is missing while their values read perfectly. The
leftmost column of the terminal is what tesseract does not read, so both
assertions were aimed at exactly the text that cannot survive. The marker
is now framed as ===UTAH-E2E-FASTFETCH=== so it does not start at column
0, and the second assertion matches the kernel version fastfetch prints
rather than the label in front of it. Both still prove what they did
before: an interactive shell ran our marker, and fastfetch rendered real
system information beside it. Nothing is skipped and the gate stays
fatal.

Checked against that captured OCR text rather than assumed: the old
patterns both fail on it, the new version pattern passes on it unchanged,
and the framed marker passes once the frame is there.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W6BYLUD6KBzF8rBFXKjP7M
@hanthor
hanthor requested a review from castrojo September 19, 2026 02:01

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed — no findings from this perspective.

— hive: agent=reviewer backend=copilot model=claude-fable-5 copilot=1.0.78

@Danathar

Copy link
Copy Markdown
Contributor

Heads-up: the btrfs half of this is byte-for-byte the same change as #158 (opened ~2h earlier by @kylerankin): BTRFS_FS added to required_config and --module BTRFS_FS in install-ogc-kernel.sh. The one difference is your require_config '^CONFIG_BTRFS_FS=(y|m)$' gate, which is worth keeping. Suggest one of you folds the gate into #158 and this PR drops to just the fastfetch-gate half (and please check that half against #137, which landed this morning and also changed what the OCR gate reads). Also note kernel_cache on both PRs is failing on the fork push, not the build — see my comment on #173.

hanthor and others added 2 commits September 19, 2026 11:29
#137 landed the fastfetch OCR decision as iso/scripts/fastfetch-ocr-match.sh,
which is a better shape than the inline greps this branch carried -- the
harness now delegates instead of holding the patterns itself. So both
conflicts resolve to main's side and this branch's OCR work retires with them.

What is left relative to main is the part still missing, and the part today's
end-to-end run proves is still needed: BTRFS_FS in the OGC kernel.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W6BYLUD6KBzF8rBFXKjP7M
Today's post-testing-e2e run (35432516418) is the first to get two of the
four flavors all the way through: utah and utah-nvidia installed to LUKS,
unlocked, logged in and passed the fastfetch gate. The two failures are
unrelated to each other, and only one of them is a bug in the image.

utah-gaming died composing its ISO, three minutes and forty-four seconds in:

    Error: Failed to install org.mozilla.firefox: While pulling
    app/org.mozilla.firefox/x86_64/stable from remote flathub: While fetching
    https://dl.flathub.org/repo/objects/88/680ed7...commitmeta:
    [28] Timeout was reached

Nothing was wrong with the image. The Flathub pull is the largest network
operation in the whole build -- Firefox alone is around 200 MB -- and a single
timed-out object took down the flavor's entire end-to-end run. Meanwhile the
curl fetching the installer bundle, twenty lines above, has had --retry 3 all
along for exactly this reason.

So the three flatpak installs now go through a retry_flatpak helper: three
attempts, 15s then 30s apart. Retrying is cheap because flatpak resumes a
partial pull from the local repository, so only what is still missing is
re-fetched, and every install already passes --or-update, which makes a retry
a no-op for refs that completed.

It does not swallow failures. A repository that is genuinely gone still fails
the build after three attempts, and there is a test for that as well as for
the recovery case -- both drive the real helper with a stub flatpak rather
than asserting on the script's text. A third test walks every install line in
the script and fails if one bypasses the retry, so the next install added
cannot quietly reintroduce the gap.

The other failure, utah-nvidia-gaming, is the btrfs one this branch already
fixes: the OGC kernel formats the root volume and then cannot mount it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W6BYLUD6KBzF8rBFXKjP7M
@hanthor

hanthor commented Sep 19, 2026

Copy link
Copy Markdown
Member Author

Rebased onto current main and widened slightly, because today's end-to-end run gave this PR fresh evidence and a second failure to fix.

The headline: the live installer works. Run 35432516418 is the first where any flavor completed the whole path — luks (utah) and luks (utah-nvidia) both passed: composed an ISO, booted it, installed to btrfs-on-LUKS, unlocked, reached a desktop, and satisfied the fastfetch screenshot gate. All 13 previous post-testing-e2e runs failed outright, so this is the first time the ISO has been shown to install Utah end to end.

The two remaining failures are unrelated to each other.

1. utah-nvidia-gaming — the bug this PR was opened for, now confirmed in a real run:

mkfs.btrfs -f -L root /dev/mapper/fisherman-root       (ok)
mount -t btrfs /dev/mapper/fisherman-root /mnt/fisherman-target
mount: unknown filesystem type 'btrfs'
blkid: /dev/mapper/fisherman-root: LABEL="root" TYPE="btrfs"

blkid sees btrfs; the kernel cannot mount it. The OGC kernel formats the root volume and then has no filesystem driver for it. BTRFS_FS in required_config is unchanged from the original commit and is the fix.

2. utah-gaming — not a bug in the image at all, which is why it is now fixed here too:

Failed to install org.mozilla.firefox: ... While fetching
https://dl.flathub.org/repo/objects/88/680ed7...commitmeta: [28] Timeout was reached

One timed-out object, 3m44s into composing the ISO, took down the flavor's entire run. The Flathub pull is the largest network operation in the build and had no retry — while the curl twenty lines above it has had --retry 3 all along. The three flatpak install calls now go through a retry_flatpak helper (3 attempts, 15s/30s backoff). It is cheap because flatpak resumes a partial pull from the local repository, and every install already passes --or-update.

It does not swallow failures: a repository that is genuinely gone still fails after three attempts, and there are tests for both the recovery case and the give-up case that drive the real helper with a stub flatpak rather than grepping the script. A third test walks every install line and fails if one bypasses the retry, so the next install added cannot quietly reintroduce the gap.

What the merge dropped: #137 landed the fastfetch OCR decision as iso/scripts/fastfetch-ocr-match.sh, which is a better shape than the inline greps this branch carried. Both conflicts resolved to main's side, and this branch's OCR work retires with them — the diff against main is now just the kernel config and the flatpak retries.

Suite green at 84685d6: Ran 198 tests ... OK. This rotates the kernel-cache tag, so the next gaming build compiles the kernel once (~45 min). That is the cost of a gaming flavor that can mount its own root filesystem.


Generated by Claude Code

@hanthor

hanthor commented Sep 19, 2026

Copy link
Copy Markdown
Member Author

Two notes from checking this against upstream rather than against the failure log alone.

The root cause is confirmed independently. From arch/x86/configs/x86_64_defconfig on current master:

CONFIG_VFAT_FS=y
CONFIG_MSDOS_FS=y
CONFIG_ISO9660_FS=y
BTRFS_FS       ABSENT
DM_CRYPT       ABSENT

BTRFS_FS genuinely is not in the defconfig this kernel is built from, and DM_CRYPT is not either — which is why it was already in required_config for the LUKS path. So the list was right about the paths it was written for and simply had never been asked about the filesystem the disk is made of.

I checked the next step and it needs nothing. After mounting btrfs the installer writes a vfat ESP, and VFAT_FS and MSDOS_FS are both in the defconfig, so there is no follow-on gap there and I have not added a speculative entry for it.

One thing I deliberately did not add. CONFIG_EFIVAR_FS is absent from the defconfig. The two flavors that pass do so on the Fedora kernel, which has it, so I cannot tell from today's evidence whether fisherman needs efivarfs to write a boot entry — and beyond the installer, an installed system without efivarfs cannot run bootupctl update or let bootc manage its boot entries at all. That is a plausible next failure and a plausible functional gap, but it is a different claim from the one this run proves, so it does not belong in this PR. If the gaming flavors reach phase 3 and then fail at the bootloader step, that is the first thing to look at.


Generated by Claude Code

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correctness review (head 84685d6)

The diff itself looks correct — BTRFS_FS added to both the config step (scripts/install-ogc-kernel.sh:145) and the required_config/require_config gate accepting y|m (scripts/install-ogc-kernel.sh:49, :177), and the retry_flatpak wrapper covers all three network installs with sound retry/backoff semantics verified by the new tests (tests/test_iso_ci.py:230-276).

Two body-vs-diff gaps worth fixing before merge:

  1. The title and half the body describe a change that is no longer in this PR. The fastfetch OCR marker/assertion fix ("stop the fastfetch gate reading the screen edge", the whole "utah and nvidia" section, and the OCR verification table) was dropped in merge commit ffbafbd ("keep the btrfs fix, drop the OCR half that #137 superseded"). Nothing in the diff touches the OCR gate. Conversely, the flatpak retry — a third of the diff (iso/live/src/install-flatpaks.sh:7-34) — is not mentioned in the body at all.

  2. The body claims test coverage that does not exist at head. "81 unit tests pass, including new coverage for the BTRFS_FS gate and for the OCR assertions" — there is no BTRFS reference in any file under tests/ at 84685d6. The only new tests are FlatpakRetryTests. The require_config BTRFS gate is exercised only by an actual kernel build.

Neither blocks the fix functionally; both should be corrected so the merge record matches the change.

— hive: agent=reviewer backend=copilot model=claude-fable-5 copilot=1.0.78

Review finding on 84685d6: the body claimed "new coverage for the BTRFS_FS
gate" and there was no BTRFS reference anywhere under tests/. That was true,
and it is the same mistake I have been flagging on other people's pull
requests this week -- a description asserting coverage the diff does not carry.

Rather than delete the claim, make it true. verify_config is a shell function
taking a config path, so its contract is testable without building anything:

- BTRFS_FS is in required_config;
- a config missing it is rejected, and the message names it;
- a complete config passes, whether the symbols are =y or =m, because
  BTRFS_FS is enabled with --module;
- and every symbol in required_config is genuinely checked -- a subtest drops
  each one in turn and asserts the gate catches it, so a symbol listed but
  never looked at cannot masquerade as a gate.

That last one is the case worth having. The list is the documentation of what
the gaming flavors need to boot, unlock and mount; an entry nobody verifies is
a comment with a misleading shape.

A full kernel build is still the only complete proof and still runs in CI.
This covers the part that can fail in review rather than in 45 minutes.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W6BYLUD6KBzF8rBFXKjP7M
@hanthor hanthor changed the title fix(iso): give the OGC kernel btrfs, and stop the fastfetch gate reading the screen edge fix(iso): give the OGC kernel btrfs, and retry Flathub pulls Sep 19, 2026
@hanthor

hanthor commented Sep 19, 2026

Copy link
Copy Markdown
Member Author

Both findings were right, and both were mine. Fixed in 596ae22 plus a title and body rewrite.

The body claimed test coverage that did not exist. You are right that there was no BTRFS reference anywhere under tests/ at 84685d6. This is the same mistake I have spent the week flagging on other people's PRs — a description asserting coverage the diff does not carry — so rather than delete the sentence I made it true.

verify_config is a shell function taking a config path, so its contract is testable in milliseconds even though the full proof is a 45-minute build. OgcKernelConfigGateTests drives the real function against fake configs:

  • BTRFS_FS is in required_config;
  • a config missing it is rejected by name;
  • a complete config passes whether symbols are =y or =m, since it is enabled with --module;
  • and a subtest drops each required_config symbol in turn and asserts the gate catches it.

That last case is the one worth having. The list is the documentation of what the gaming flavors need in order to boot, unlock and mount; an entry nobody verifies is a comment with the shape of a gate. Removing BTRFS_FS from the list fails two of the new tests, so they are not decorative either.

The title and body described a change that had left the PR. Also correct. The OCR half was dropped when I merged main, because #137 landed a better version of it, and the flatpak retry — a third of the diff — was never in the body at all. Title is now "give the OGC kernel btrfs, and retry Flathub pulls", and the body covers both halves with the failure output for each, plus a note explaining what was dropped and why so the merge record makes sense.

Two things I added while checking your first point, both from upstream's x86_64_defconfig rather than from the failure log: BTRFS_FS genuinely is absent there (as is DM_CRYPT, which is why that was already in the list), and VFAT_FS is present, so the ESP the installer writes immediately after mounting needs no further entry. That is now in the body, and it is also why I did not add a speculative EFIVAR_FS — noted separately above as the first thing to look at if the gaming flavors reach phase 3 and then fail at the bootloader step.

Ran 203 tests ... OK at 596ae22.


Generated by Claude Code

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed (correctness) — no blockers. One edge case worth noting:

The PR body says "every install already passes --or-update, so a retry is a no-op for refs that completed", but the first wrapped call does not: retry_flatpak install --system --noninteractive installer-local "${INSTALLER_APP_ID}" (iso/live/src/install-flatpaks.sh:100) has no --or-update. If attempt 1 installs the app but still exits nonzero, attempts 2–3 would fail with "already installed" and the retry turns a flaky success into a hard failure. Low risk in practice — this remote is a local file:// repo, not the network path the retry was added for — but adding --or-update there would make the helper's contract uniform.

Verified the rest: verify_config matches =(y|m) (scripts/install-ogc-kernel.sh:59) so --module BTRFS_FS passes the new gate at line 177, and the cache-tag rotation claim holds since scripts/kernel-cache-tag.sh hashes install-ogc-kernel.sh whole.

— hive: agent=reviewer backend=copilot model=claude-fable-5 copilot=1.0.78

The retry wrapper was added so a single timed-out Flathub object could not
fail a whole flavor's end-to-end run. Two of the three installs it wraps
already passed --or-update, which is what makes a retry safe: re-running is a
no-op for a ref that already completed. The installer-local install did not.

That asymmetry is the one case where the retry could make things worse. An
attempt that installs the app but still exits nonzero leaves attempts 2 and 3
to fail with "already installed", so a flaky success becomes a hard failure --
the opposite of the reason the wrapper exists. It is unlikely on a file://
remote rather than the network path the retry was written for, but the helper's
contract should not depend on which call site it is protecting.

The new test asserts every retried install carries --or-update, and it fails
against the previous line, so the uniformity is now a gate rather than a
convention.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W6BYLUD6KBzF8rBFXKjP7M
@hanthor hanthor changed the title fix(iso): give the OGC kernel btrfs, and retry Flathub pulls Fix the two remaining ISO flavor failures: btrfs in the OGC kernel, retried Flathub pulls Sep 19, 2026
@hanthor

hanthor commented Sep 19, 2026

Copy link
Copy Markdown
Member Author

Both review findings addressed.

--or-update on the installer-local install — you were right, and the reasoning holds exactly as you stated it: an attempt that installs the app but exits nonzero would leave attempts 2 and 3 failing with "already installed", so the retry would convert a flaky success into a hard failure. Low risk on a file:// remote, but the helper's contract should not depend on which call site it happens to be protecting. Fixed in 738d64e, and now gated rather than left as a convention — the new test_every_retried_install_is_idempotent fails against the previous line:

AssertionError: '--or-update' not found in 'retry_flatpak install --system
--noninteractive installer-local \\\n    "${INSTALLER_APP_ID}"'

Full suite after the fix: 204 tests, OK.

Title and body describing the dropped OCR half — corrected. The title and description now describe the two flavor failures this actually fixes, the flatpak retry has a section of its own, and there is an explicit scope note recording that the OCR change was superseded by #137 and dropped in ffbafbd. Thanks for catching it; the merge record would have been wrong.

One correction on the coverage point, since it was accurate when you wrote it and is no longer: at 84685d6 there genuinely was no BTRFS reference under tests/, as you said. OgcKernelConfigGateTests landed afterwards in 596ae22, and it drives the real verify_config against generated configs with a subtest that drops each required_config symbol in turn. So the gate is now unit-tested per symbol, not only by an actual kernel build.


Generated by Claude Code

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed — no findings from this perspective.

— hive: agent=reviewer backend=copilot model=claude-fable-5 copilot=1.0.78

@hanthor

hanthor commented Sep 19, 2026

Copy link
Copy Markdown
Member Author

CI is green on 738d64e — all 13 jobs, including all four images. But I want to be exact about what that does and does not prove, because it is less than it looks and the difference matters for whoever approves this.

contract                                                     success
kernel_cache                                                 success
build_main / image (main, utah, testing, x86_64)             success
build_kernel / image (gaming, utah-kernel, testing, x86_64)  success
build_kernel / image (nvidia, ...)                           success
build_kernel / image (nvidia-gaming, ...)                    success
dispatch-iso                                                 skipped

dispatch-iso is skipped, and not incidentally:

if: >-
  !cancelled() && github.ref == 'refs/heads/testing' &&
  github.event_name != 'pull_request' && ...

No ISO is built or booted on a pull request at all. So nothing in this run exercises the two things this PR fixes. The btrfs module is not loaded, mkfs.btrfs is not run, no LUKS install happens, and no Flathub pull is attempted. A green check here means the images build and the host-side suite passes — it cannot mean the ISO works.

What the fixes actually rest on, stated honestly:

  • The unit tests, which drive the real code rather than reading it. OgcKernelConfigGateTests executes the real verify_config against generated configs, dropping each required_config symbol in turn; FlatpakRetryTests extracts the real retry_flatpak and runs it against a stub flatpak, asserting a transient failure is retried to success and a persistent one still fails. That is real evidence about the logic, and it is why the --or-update gap was catchable at all.
  • The diagnosis from run 35432516418, where both failures are in the logs verbatim — unknown filesystem type 'btrfs' for the OGC flavors, and the Firefox [28] Timeout was reached for utah-gaming.

What remains unverified until this merges and testing builds: that a kernel with CONFIG_BTRFS_FS=m actually mounts the installed root, and that the retry actually recovers a real Flathub timeout. Both are only observable on the testing branch, by construction.

I am not proposing to change that gating here — building an ISO per PR is expensive and the current arrangement is a deliberate trade. But it does mean this PR is structurally unverifiable before merge, which is worth knowing when weighing it: the failure it fixes has occurred in 14 consecutive runs, the fix is small and unit-tested, and the cost of being wrong is one more red post-testing-e2e, not a broken :testing tag (promotion is gated behind that job passing).

Related: kylerankin's #158 fixes the same btrfs problem and predates this PR by two hours. I have said so there and offered to strip the btrfs half out of this one so theirs lands instead, leaving this as the Flathub retry alone. That is still my preference; I did not want to reorganise their PR unasked.


Generated by Claude Code

Danathar
Danathar previously approved these changes Sep 19, 2026

@Danathar Danathar left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving, with a concrete suggestion on how to sequence this against #158 so the fix lands today instead of waiting on a chain.

What I checked. All 13 jobs including the four image builds are green on 738d64e6. The production diff is small: BTRFS_FS added to required_config and built as a module, plus a require_config gate so a config that drops it fails the build; and a retry_flatpak wrapper (3 attempts, 15/30s backoff) around the three flatpak install calls in iso/live/src/install-flatpaks.sh, with --or-update on the installer-local install so a partial first attempt doesn't make attempts 2 and 3 fail on "already installed". The tests drive the real verify_config and the real retry_flatpak rather than grepping the scripts. hanthor's note above about what CI does and does not prove is accurate — nothing on a PR boots an ISO — and the fallback if this is wrong is one more red post-testing-e2e, not a bad :testing tag.

Sequencing. #158 (kylerankin) made the same btrfs change two hours earlier and has priority on authorship, but it cannot go green until #184 (the fork-push fix) merges, and the failure this fixes has now happened on 14 consecutive runs. I'd rather not hold a working fix on ordering. Suggest:

  1. Merge this PR as-is now — it is the only one of the two that CI can currently pass.
  2. On #158, kylerankin keeps the parts this PR doesn't have (docs/skills/kernel-cache.md, the luks-e2e.sh change) and rebases; the btrfs lines will already be on main. Credit for the btrfs diagnosis is theirs either way and should say so in the changelog.

If kylerankin would rather land #158 first, that's fine too — but then #184 needs to merge first so #158 can get a green kernel_cache, and the ISO stays broken for however long that takes.

One conflict, and it is purely additive on both sides: #184 landed
ForkPullRequestKernelCacheTests at the same point in the file where this
branch had added FlatpakRetryTests and OgcKernelConfigGateTests. Nothing
overlaps, so both are kept.

250 tests pass.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W6BYLUD6KBzF8rBFXKjP7M
@hanthor

hanthor commented Sep 19, 2026

Copy link
Copy Markdown
Member Author

@Danathar thank you — and I need to ask you to re-approve, because I cost you the first one through a sequencing mistake.

Both this PR and #184 were approved and clean at the same moment. I merged #184 first, on the reasoning that it unblocks two contributors' fork PRs. That made this branch conflict — #184 added ForkPullRequestKernelCacheTests to tests/test_iso_ci.py at the same point where this branch adds FlatpakRetryTests and OgcKernelConfigGateTests — so resolving it required a push, and require_last_push_approval dismissed your approval on 738d64e6.

Had I merged this one first it would have gone in untouched and #184 would have taken the conflict instead. Since this is the release blocker and #184 is not, that was the wrong order. My error, not a surprise from the branch protection.

The merge is mechanical and purely additive. One conflict, both sides appending different test classes to the same region of one file. Nothing overlaps; both are kept, and all six classes are present:

InputsTests · EvidenceTests · FastfetchOcrGateTests
FlatpakRetryTests · OgcKernelConfigGateTests · ForkPullRequestKernelCacheTests

250 tests pass on c246f8c. No production file changed in the merge — install-ogc-kernel.sh and install-flatpaks.sh are byte-identical to what you reviewed at 738d64e6.

On your sequencing note: agreed, and I have said the same on #158 — the btrfs diagnosis is kylerankin's, it predates this branch by two hours, and the merge commit message says so explicitly. With #184 now merged, kernel_cache should go green on #158, so the parts this PR does not have (docs/skills/kernel-cache.md and the luks-e2e.sh modprobe btrfs pre-check) can land there on a green run rather than a blocked one.

I will not approve this myself, so it waits on you or another reviewer. Once it merges, testing gets its first build with both fixes and post-testing-e2e finally has something to say about the two flavors that have been failing for fourteen runs.


Generated by Claude Code

@hanthor

hanthor commented Sep 19, 2026

Copy link
Copy Markdown
Member Author

Fresh reproduction of the exact failure this PR fixes, from twenty minutes ago — and it corrects something I said earlier in this thread.

Merging #184 and #190 produced a testing build (run 35447134853, all 13 jobs green including dispatch-iso), which triggered Post-Testing E2E run 35447837973:

resolve                                success
luks (utah, sha256:9fd8f0ee…)          success
luks (utah-nvidia, sha256:80970c53…)   success
luks (utah-gaming, sha256:8b3730d1…)   FAILURE
luks (utah-nvidia-gaming, sha256:7a85) FAILURE
documentation                          skipped
promote-to-testing                     skipped

Both failures, verbatim:

+ mount -t btrfs /dev/mapper/fisherman-root /mnt/fisherman-target
mount: /var/mnt/fisherman-target: unknown filesystem type 'btrfs'.
fisherman: fatal: mounting root: ... exit status 32
  (blkid: /dev/mapper/fisherman-root: LABEL="root" UUID="…" TYPE="btrfs")

btrfs-progs v7.1 formats it, blkid reads it back as btrfs, and the kernel cannot mount it. That is the missing CONFIG_BTRFS_FS precisely.

The correction. I have been describing this as two different failures — btrfs on utah-nvidia-gaming, a Flathub timeout on utah-gaming — based on run 35432516418. That was accurate for that run, but it led me to imply the two flavors had independent problems. They do not: both gaming flavors fail on btrfs, because both ship the OGC kernel. In run 35432516418 the Flathub timeout hit utah-gaming first and masked the btrfs failure underneath it.

That makes the btrfs fix the load-bearing one of the two, and the Flathub retry a fix for a flake that was hiding it rather than a second flavor-specific defect. Both are still worth having — a timed-out object should not fail a compose — but the priority ordering I gave was wrong.

It also means kylerankin's #158, which is btrfs-only, addresses the actual cause of both failures on its own. Another argument for landing that work rather than treating it as the lesser half.

And it confirms the release chain. promote-to-testing skipped, so no :testing tag was written — which is why all four images are still HTTP 404 from GHCR and why promotion PR #181 fails digest resolution. That is the chain end to end, observed rather than inferred:

btrfs missing from OGC kernel
  -> 2 of 4 luks flavors fail
  -> promote-to-testing skipped
  -> no :testing tags
  -> #181 digest resolution fails

Still needs re-approval on c246f8c after the conflict resolution I caused; I will not self-approve.


Generated by Claude Code

#158 merged, so BTRFS_FS is on main. One conflict, in the scripts/config line
of install-ogc-kernel.sh, where both sides set exactly the same flags and only
the line wrapping differed. Took main -- kylerankin's PR is where this landed
and its formatting is the one to keep.

What is left over main from this branch is the Flathub retry in
install-flatpaks.sh and the test classes in test_iso_ci.py. The btrfs
production change is now identical on both sides rather than duplicated.

250 tests pass.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W6BYLUD6KBzF8rBFXKjP7M
With #158 merged, BTRFS_FS is in required_config, and verify_config .config runs
on the very next line iterating that array with the identical pattern
^CONFIG_BTRFS_FS=(y|m)$. So my extra require_config line asserted something
already asserted two lines later.

Worse than merely redundant: require_config prints a fixed diagnostic listing
SCHED_CLASS_EXT, NTSYNC, ANDROID_BINDER, DEBUG_INFO and BPF settings. For a
btrfs failure that is noise pointing at the wrong subsystem, where
verify_config says plainly which symbol is missing.

What is left of this branch over main is the Flathub retry, the test classes,
and the comment recording the btrfs failure verbatim -- the mount error, why the
list had never needed a filesystem before (the missing KMS device stopped the
test at phase 2), and that BTRFS_FS selects its own crc/zlib/lzo/zstd/raid6
helpers so naming it is enough for olddefconfig. OgcKernelConfigGateTests still
drives the real verify_config against generated configs, dropping each
required_config symbol in turn, so it now gates main's code rather than this
branch's.

250 tests pass.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W6BYLUD6KBzF8rBFXKjP7M
@hanthor hanthor changed the title Fix the two remaining ISO flavor failures: btrfs in the OGC kernel, retried Flathub pulls Retry Flathub pulls, so one timed-out object cannot fail a flavor's ISO Sep 19, 2026

@Danathar Danathar left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-approving at 5a261af. What changed since my earlier approval, and what I checked: kylerankin's #158 merged, so the btrfs half of this PR now comes from main; hanthor merged main in and dropped the one require_config line that #158's required_config entry already covers (the scripts/install-ogc-kernel.sh change is now comments only). What remains is the Flathub retry — retry_flatpak (3 attempts, 15/30s backoff) around the three flatpak install calls, with --or-update on the installer-local install so a partial first attempt can't make the retries fail on "already installed" — plus its tests. tests/test_iso_ci.py passes at head (31 tests, 53 subtests). The btrfs failure hanthor reproduced on today's testing E2E run is now fixed by #158 on main; the next testing build after this lands is the one that proves the Flathub half.

@hanthor

hanthor commented Sep 19, 2026

Copy link
Copy Markdown
Member Author

@Danathar this PR now has direct evidence rather than an argument, and I need to ask you to re-approve it — your approval on 738d64e6 was dismissed by my own push resolving the conflict that merging #184 first created.

The btrfs fix works, and it exposed this PR's failure underneath it.

#158 merged, so dbd1425a ships a kernel with CONFIG_BTRFS_FS=m. Post-Testing E2E run 35469913325 ran the luks matrix against it, and in luks (utah-gaming):

occurrences of "unknown filesystem type 'btrfs'":  0

That error failed this flavor in every prior run. It is gone. What it failed on instead:

Error: Failed to install org.mozilla.firefox: While pulling
app/org.mozilla.firefox/x86_64/stable from remote flathub: While fetching
https://dl.flathub.org/repo/objects/4136af8c0282301351563f91c8b5e5...
Error: building at STEP "RUN --mount=type=cache,target=/var/cache/flatpak-dl ..."

A Flathub object failing mid-pull while composing the ISO — precisely what retry_flatpak in this PR exists to survive, and the only remaining thing between utah-gaming and a passing install.

This also settles a correction I made earlier in this thread. I had said the Flathub timeout was masking the btrfs failure on utah-gaming; with btrfs fixed, the Flathub failure surfaced again on the very next run. Two independent defects, layered — which is why both fixes were needed and why neither alone was sufficient.

luks (utah-nvidia) has since passed. utah and utah-nvidia-gaming are still running; utah-nvidia-gaming is the other flavor that carries the OGC kernel, so it is the second half of #158's verification.

What is in this PR now, after the btrfs half landed in #158 and I stripped the duplication: the retry_flatpak wrapper (three attempts, 15s then 30s backoff) around all three installs with --or-update on each, FlatpakRetryTests and OgcKernelConfigGateTests driving the real code, and a comment recording the btrfs failure. 250 tests. dispatch-iso is gated to refs/heads/testing, so CI here still cannot exercise the retry — the evidence that it is needed is the log above, not this PR's checks.

I will not self-approve. If you would rather someone else look at it given how much of this thread is mine, that is entirely fair.


Generated by Claude Code

@Danathar

Copy link
Copy Markdown
Contributor

@hanthor my approval is current: the one you see dismissed was on 738d64e6; I re-approved on 5a261af at 17:35 after your merge from main, and reviewDecision on this PR is APPROVED. Nothing has been pushed since, so no further re-approval is needed.

What was actually red on this PR is different: the three failing checks (build_main / Collect image digests, build_kernel / Collect image digests, dispatch-iso) were cancelled at 19:18–19:19, zero seconds in, not failed. The reusable build workflow keys its concurrency group on brand and stream with cancel-in-progress, so my push to #198 at 19:14 cancelled the tail of this run. That is the same cancellation problem #185 is about. I have re-run the cancelled jobs now that nothing else is building (run_attempt 2 on run 35455452847); once they finish the PR should be green and mergeable on your side.

Agreed on the reading of run 35469913325: btrfs is gone from utah-gaming, and what remains is the Flathub pull failure this PR's retry_flatpak exists for.

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed — no findings from correctness, security, intent-alignment, style, docs-currency.

— hive: agent=reviewer backend=copilot model=claude-fable-5 copilot=1.0.78

@castrojo castrojo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed diff and CI results. Adds retry_flatpak helper in install-flatpaks.sh with 3 attempts and exponential backoff to tolerate transient Flathub network timeouts during offline ISO packaging. All required checks pass.

@castrojo
castrojo merged commit 6ebbfab into main Sep 21, 2026
24 of 27 checks passed
@castrojo
castrojo deleted the claude/iso-btrfs-and-ocr branch September 21, 2026 02:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants