Repository navigation
Open
Labels
approved-directionkind/bugSomething is broken.Something is broken.needs-humantriage/acceptedScope accepted; ready for work. Use `/triage accepted`.Scope accepted; ready for work. Use `/triage accepted`.
Description
Summary: The server image's full build no longer finishes. Since November 8 every full build, including the nightly build of unchanged main, runs for about five hours and is cancelled or fails while building the image set. Because the merge queue runs that same build, no approved server pull request has merged since November 7.
What I need from you: Find why "Build and export the image set" now runs past five hours, and fix it so the merge queue can land approved PRs again.
Evidence
build.yml → job build → step 9 "Build and export the image set" (BuildStream). Non-PR runs (PR runs skip the full build):
| Run start (UTC) | Event | Result | Duration |
|---|---|---|---|
| 2026-11-08 20:42 | merge_group (pr-591) | success | last success |
| 2026-11-08 21:23 | merge_group (pr-591, pr-640) | cancelled | ~300 min |
| 2026-11-08 22:50 | merge_group (pr-595) | cancelled | ~300 min |
| 2026-11-09 03:26–03:27 | merge_group (pr-591, pr-595) | cancelled ×3 | ~300 min |
| 2026-11-09 05:32 | schedule (main) | failure | 43 min |
| 2026-11-09 06:24 | merge_group (pr-640) | cancelled | ~300 min |
| 2026-11-09 12:18 | merge_group (pr-640) | failure | 246 min |
| 2026-11-09 21:14 | merge_group (pr-640) | cancelled | ~300 min |
| 2026-11-10 03:15 | merge_group (pr-640) | cancelled | ~301 min |
| 2026-11-10 05:32 | schedule (main) | cancelled | 300 min (step 9 ran 05:33–10:32) |
| 2026-11-10 09:14 | merge_group (pr-640) | still running at 12:45 | — |
mainhas not changed since21b81ce(2026-11-07 23:09, docs: note the SBOM lists a package once even when it ships from two sources #635), yet the nightly build of main is also cancelled, so this is not caused by a queued PR.- The merge queue ruleset has
check_response_timeout_minutes: 360; the long build jobs havetimeout-minutes: 240and run onubuntu-26.04. - Consequence: fix(homelab): skip unprovable join candidates and stop trusting a bare k0s token #591, fix(update): fail on A/B slot headroom and keep two k0s images #595 and chore(deps): update github-actions (non-major) #640 have cycled through the queue for two days; nothing has merged since 2026-11-07.
Likely areas to check (not verified)
- BuildStream artifact cache: a cache miss or an unreachable remote cache would rebuild everything from source and run past the limit. Compare cache hit rates between the 11-08 20:42 success and the first cancelled run.
- Runner image and disk: compare "Set up runner disk space" and the BuildStream log tail of a cancelled run with the 20:42 success.
- The 2026-11-09 05:32 scheduled run failed after 43 minutes instead of timing out; its error may be the clearest clue.
Acceptance criteria
- A full build of current main completes within the job timeout (scheduled run or a manual
workflow_dispatch). - The merge queue lands at least one queued PR (e.g. chore(deps): update github-actions (non-major) #640) with the build passing.
- The fixing PR states the root cause (cache, runner, element change, or other) with evidence.
Out of scope
- Raising timeouts to hide the problem, unless the cause is shown to be a legitimate, permanent build-time increase.
- Changing which checks the merge queue requires.
Metadata
Metadata
Assignees
Labels
approved-directionkind/bugSomething is broken.Something is broken.needs-humantriage/acceptedScope accepted; ready for work. Use `/triage accepted`.Scope accepted; ready for work. Use `/triage accepted`.
Activity
jeefy commentedon Nov 10, 2026
Issue triage
Next action: a triager or maintainer reviews this issue.
Reporter or contributor
kind/*label is set, categorize the issue with one of:/kind bug,/kind regression,/kind security,/kind feature,/kind documentationor/kind cleanup./area <area>:desktop,flatpak,gaming,hardware,installerordx.Triager or maintainer
/triage accepted. Only triagers and maintainers can do this; acceptance does not assign an implementer.needs-kindclears when a kind is set.needs-triageand this post clear when triage is accepted.See Triage & Prow Commands for all available commands.
Danathar commentedon Nov 10, 2026
/triage accepted
/kind bug
hivecommons-hive commentedon Nov 10, 2026
Good first issue— withdrawn by hosted-projectbluefin-knuckle-gjvq: this issue is not open to new contributors — it carries theblockedlabel, so it cannot be worked right now. Thegood first issueandhelp wantedlabels have been removed (2026-11-10T15:10:16Z). Sorry for the noise! Issues open to new contributors are listed on the relay page: https://hosted-projectbluefin-common-nmq5.hive.hivecommons.dev/contributeDanathar commentedon Nov 12, 2026
Moved from a silent
blockedtoneeds-human. A contributor relay marked thisblockedwithout naming what it waits on, so nothing could lift it, and the hive's own fix-up was held by its write-back gate (the issue predates the hive's write access). Removingblockedand addingneeds-humankeeps agents off it and puts it in front of a maintainer, which is what the hive would have done itself.Still happening: nothing has merged in this repo since 2026-11-07, and today's merge-queue run of "Build image artifacts" failed again (2026-11-12 19:10 UTC). The fix needs runs of the five-hour full build, which a relay can't start or wait on.
Maintainer question: who digs into "Build and export the image set"? This blocks every merge in the repo.
Danathar commentedon Nov 18, 2026
Status check (1:30 PM EST, 11-18): the symptom in the title no longer reproduces. I'm leaving the labels as they are, so a server maintainer can confirm and close this.
Evidence:
If nobody has seen a hang since then, this can close.