Skip to content

ci: server's full image build never finishes since 2026-11-08, so the merge queue lands nothing #650

Open

Description

@Danathar

Summary: The server image's full build no longer finishes. Since November 8 every full build, including the nightly build of unchanged main, runs for about five hours and is cancelled or fails while building the image set. Because the merge queue runs that same build, no approved server pull request has merged since November 7.

What I need from you: Find why "Build and export the image set" now runs past five hours, and fix it so the merge queue can land approved PRs again.


Evidence

build.yml → job build → step 9 "Build and export the image set" (BuildStream). Non-PR runs (PR runs skip the full build):

Run start (UTC) Event Result Duration
2026-11-08 20:42 merge_group (pr-591) success last success
2026-11-08 21:23 merge_group (pr-591, pr-640) cancelled ~300 min
2026-11-08 22:50 merge_group (pr-595) cancelled ~300 min
2026-11-09 03:26–03:27 merge_group (pr-591, pr-595) cancelled ×3 ~300 min
2026-11-09 05:32 schedule (main) failure 43 min
2026-11-09 06:24 merge_group (pr-640) cancelled ~300 min
2026-11-09 12:18 merge_group (pr-640) failure 246 min
2026-11-09 21:14 merge_group (pr-640) cancelled ~300 min
2026-11-10 03:15 merge_group (pr-640) cancelled ~301 min
2026-11-10 05:32 schedule (main) cancelled 300 min (step 9 ran 05:33–10:32)
2026-11-10 09:14 merge_group (pr-640) still running at 12:45 —

Likely areas to check (not verified)

  • BuildStream artifact cache: a cache miss or an unreachable remote cache would rebuild everything from source and run past the limit. Compare cache hit rates between the 11-08 20:42 success and the first cancelled run.
  • Runner image and disk: compare "Set up runner disk space" and the BuildStream log tail of a cancelled run with the 20:42 success.
  • The 2026-11-09 05:32 scheduled run failed after 43 minutes instead of timing out; its error may be the clearest clue.

Acceptance criteria

  • A full build of current main completes within the job timeout (scheduled run or a manual workflow_dispatch).
  • The merge queue lands at least one queued PR (e.g. chore(deps): update github-actions (non-major) #640) with the build passing.
  • The fixing PR states the root cause (cache, runner, element change, or other) with evidence.

Out of scope

  • Raising timeouts to hide the problem, unless the cause is shown to be a legitimate, permanent build-time increase.
  • Changing which checks the merge queue requires.

Activity

  1. added
    needs-kindMissing required label matching ^kind/
    needs-triageMissing required label matching ^triage/
    on Nov 10, 2026
  2. jeefy commented on Nov 10, 2026

    @jeefy
    Collaborator

    Issue triage

    Next action: a triager or maintainer reviews this issue.

    Reporter or contributor

    • If no kind/* label is set, categorize the issue with one of: /kind bug, /kind regression, /kind security, /kind feature, /kind documentation or /kind cleanup.
    • Optionally add an area with /area <area>: desktop, flatpak, gaming, hardware, installer or dx.

    Triager or maintainer

    • Review the report and ask the reporter for any missing information.
    • When the scope is accepted and the issue is ready for work, comment /triage accepted. Only triagers and maintainers can do this; acceptance does not assign an implementer.

    needs-kind clears when a kind is set. needs-triage and this post clear when triage is accepted.

    See Triage & Prow Commands for all available commands.

  3. Danathar commented on Nov 10, 2026

    @Danathar
    ContributorAuthor

    /triage accepted
    /kind bug

  4. added
    kind/bugSomething is broken.
    triage/acceptedScope accepted; ready for work. Use `/triage accepted`.
    and removed
    needs-kindMissing required label matching ^kind/
    needs-triageMissing required label matching ^triage/
    on Nov 10, 2026
  5. added
    stage/010-actionableTriaged; eligible for the hive to pick up
    and removed
    stage/015-triage-neededActionable but held by the reporter-trust gate: awaiting a maintainer's triage label
    on Nov 10, 2026
  6. hivecommons-hive commented on Nov 10, 2026

    @hivecommons-hive
    Contributor

    Good first issue — withdrawn by hosted-projectbluefin-knuckle-gjvq: this issue is not open to new contributors — it carries the blocked label, so it cannot be worked right now. The good first issue and help wanted labels have been removed (2026-11-10T15:10:16Z). Sorry for the noise! Issues open to new contributors are listed on the relay page: https://hosted-projectbluefin-common-nmq5.hive.hivecommons.dev/contribute

  7. added
    good first issueDenotes an issue ready for a new contributor, according to the "help wanted" guidelines.
    help wantedDenotes an issue that needs help from a contributor. Must meet "help wanted" guidelines.
    blockedWaiting on something outside this repository. Use `/label blocked` or `/remove-label blocked`.
    and removed
    stage/010-actionableTriaged; eligible for the hive to pick up
    help wantedDenotes an issue that needs help from a contributor. Must meet "help wanted" guidelines.
    good first issueDenotes an issue ready for a new contributor, according to the "help wanted" guidelines.
    on Nov 10, 2026
  8. added and removed
    blockedWaiting on something outside this repository. Use `/label blocked` or `/remove-label blocked`.
    on Nov 12, 2026
  9. Danathar commented on Nov 12, 2026

    @Danathar
    ContributorAuthor

    Moved from a silent blocked to needs-human. A contributor relay marked this blocked without naming what it waits on, so nothing could lift it, and the hive's own fix-up was held by its write-back gate (the issue predates the hive's write access). Removing blocked and adding needs-human keeps agents off it and puts it in front of a maintainer, which is what the hive would have done itself.

    Still happening: nothing has merged in this repo since 2026-11-07, and today's merge-queue run of "Build image artifacts" failed again (2026-11-12 19:10 UTC). The fix needs runs of the five-hour full build, which a relay can't start or wait on.
    Maintainer question: who digs into "Build and export the image set"? This blocks every merge in the repo.

  10. Danathar commented on Nov 18, 2026

    @Danathar
    ContributorAuthor

    Status check (1:30 PM EST, 11-18): the symptom in the title no longer reproduces. I'm leaving the labels as they are, so a server maintainer can confirm and close this.

    Evidence:

    If nobody has seen a hang since then, this can close.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions