Skip to content

Opt-in feature e2e failing #620

Closed
Closed

Description

@github-actions

The weekly opt-in feature e2e run failed: https://github.com/projectbluefin/server/actions/runs/41805416025

This issue closes when a later weekly run passes.

Activity

  1. hivecommons-hive commented on Nov 5, 2026

    @hivecommons-hive
    Contributor

    Summary: Triage of this week's failed weekly home-lab test. Two of the six test scenarios failed, for two unrelated reasons, and neither looks like the same problem as last week's failure. One is a test machine whose emulated security chip stopped responding during installation; the other is a home-lab setup step that failed on the main test machine, whose cause was not saved — a separate report (#623) asks for that log to be kept.


    Run 41805416025 (2026-11-04, head main): 4/6 scenarios green (mdns, kubeadm, nvidia, homelab-templates).

    e2e (homelab-cluster) — all check assertions matched and all three guests printed PROBE failed=0; rc=1 came from wait "${cp_pid}", because the control-plane guest's boot check saw a failed unit:
    [FAILED] Failed to start Apply the Bluefin … components to this node's cluster. (bluefin-homelab-apply.service, which has Restart=on-failure, so a first-attempt failure is enough to fail the harness). The unit's own output is in cp-set/dogfood-serial.log, which the upload step does not collect → #623.

    e2e (homelab-installer) — FAIL: cp: install timed out after 1207 s. cp-2-install.ttyS0.log stops right after:
    tpm tpm0: Operation Timed out / tpm_crb_acpi MSFT0101:00: probe with driver tpm_crb_acpi failed with error -62
    and cp-2-install.swtpm.log is empty — the emulated TPM (swtpm) stopped answering, so the install never progressed. Looks like an emulator/runner fault rather than an image change; worth a re-check on the next weekly run before treating it as a regression.

    Previous failure (run 40662525415, 2026-10-28) was different: homelab-cluster node never reported (FAIL: boot probe did not report within 3300s), while the CP and the wrong-passphrase guest passed.

    Three different symptoms in two runs → no single code cause yet. Next step: land #623 so the next failure carries the guest serial logs.


    ci-maintainer triage

    — hive: agent=ci-maintainer backend=copilot model=claude-opus-5.5 copilot=1.0.88

  2. Danathar commented on Nov 9, 2026

    @Danathar
    Contributor

    /triage accepted
    /kind bug

  3. added
    kind/bugSomething is broken.
    triage/acceptedScope accepted; ready for work. Use `/triage accepted`.
    on Nov 9, 2026
  4. hivecommons-hive commented on Nov 9, 2026

    @hivecommons-hive
    Contributor

    Good first issue — withdrawn by hosted-projectbluefin-knuckle-gjvq: this issue is not open to new contributors — it carries the blocked label, so it cannot be worked right now. The good first issue and help wanted labels have been removed (2026-11-10T00:28:17Z). Sorry for the noise! Issues open to new contributors are listed on the relay page: https://hosted-projectbluefin-common-nmq5.hive.hivecommons.dev/contribute

  5. added
    good first issueDenotes an issue ready for a new contributor, according to the "help wanted" guidelines.
    help wantedDenotes an issue that needs help from a contributor. Must meet "help wanted" guidelines.
    on Nov 9, 2026
  6. Karibhou commented on Nov 10, 2026

    @Karibhou
    Contributor

    Summary: This weekly home-lab test failure can't be fixed yet because the log that explains it was never saved. A pending change that keeps those logs (#627, for the report about lost boot logs, #623) has to land first. After it merges, the next weekly run will show the real cause.


    • No weekly run since the triaged one: the latest e2e-opt-in.yml run is still 41805416025 (2026-11-04).
    • homelab-cluster: the output from bluefin-homelab-apply.service is only in cp-set/dogfood-serial.log, and that file is not uploaded. PR ci: keep boot logs from weekly home-lab tests #627 (still open) adds it to the upload.
    • homelab-installer: the swtpm emulated chip timed out, and its log is empty. Nothing in the repo points to a cause yet, so this needs another run to confirm.

    Making a code change now would only be a guess. Waiting on #627 and the next weekly run.

    — hive: agent=scanner backend=omp model=anthropic/claude-opus-5-5


    🐝 Hive Agent: contributor | SHA: unknown

  7. added
    blockedWaiting on something outside this repository. Use `/label blocked` or `/remove-label blocked`.
    and removed
    stage/010-actionableTriaged; eligible for the hive to pick up
    help wantedDenotes an issue that needs help from a contributor. Must meet "help wanted" guidelines.
    good first issueDenotes an issue ready for a new contributor, according to the "help wanted" guidelines.
    on Nov 10, 2026
  8. galiliya commented on Nov 10, 2026

    @galiliya

    On e2e (homelab-cluster): if the boot check is reading systemd unit state, it's going to trip on any [FAILED] line from bluefin-homelab-apply.service even when Restart=on-failure recovers it on the next attempt — a first attempt failing isn't really a test failure. Having the check wait out the restart window and then assert on the unit's settled state (systemctl is-failed / is-active) would make that signal meaningful instead of flaky.

    That said, we still don't know why the apply failed the first time, since cp-set/dogfood-serial.log never gets uploaded — that's #623, and I'd fix that before tuning the assertion. Is the harness assertion change something you want as its own PR, or should it ride along with #623?

  9. github-actions commented on Nov 11, 2026

    @github-actions
    ContributorAuthor
  10. removed
    blockedWaiting on something outside this repository. Use `/label blocked` or `/remove-label blocked`.
    on Nov 17, 2026
  11. hivecommons-hive commented on Nov 17, 2026

    @hivecommons-hive
    Contributor

    Hive status — updated in place; edits do not notify. Full history is in the Activity Monitor.

    Unblocked: #627 is no longer open; re-triaging.

  12. Karibhou commented on Nov 17, 2026

    @Karibhou
    Contributor

    Summary: Last week's weekly home-lab test failed for a reason that has since been fixed. The test machines' internal clocks jumped back to 2011, so they rejected the release's valid digital signature as "not yet made" and wouldn't download the system. A fix from a few days ago that keeps the clock from jumping back (#664) is in every release since then. The next weekly run tests the newest release, and when it passes this issue closes on its own.


    Run 42993076271 (2026-11-11, release v26.11.1714) failed 5/6 scenarios. Every netbooted guest stopped at the same point:

    systemd-importd: (transfer1) DOWNLOAD INVALID: Signature verification failed.
    Failed to start Download of http://10.0.2.2:8765/bluefin-server_26.11.1714.raw.
    
    • The signature is valid. The host step verified it (Good signature … 7A87E575…9C34), and I reproduced Good signature offline. I ran the release's own initrd gpg inside a chroot with the exact arguments systemd-pull uses, against the UKI's /usr/lib/systemd/import-pubring.pgp.
    • The guest clock is the cause. The installer guest's serial log shows time was further ahead than 15y after built-in epoch, clock reset to Thu 2011-11-10 15:00:00. systemd's built-in epoch is FSDK's 2011-11-10, so from 2026-11-10 on, PID 1 resets a correct RTC to 2011. gpg then rejects the 2026 signature and key because they look like they come from the future. The 2026-11-04 run fell one week before the 15-year window ran out, which explains why mdns, kubeadm and nvidia passed then and failed now.
    • Fixed by fix(build): repair shared caching and boot clock epoch #664 (merged 2026-11-14, e0a63e8, first in v26.11.2108), which adds a /usr/lib/clock-epoch clock floor. The latest release v26.11.2264 (the one the next scheduled run picks up) has it in the netboot initrd: usr/lib/clock-epoch has an Aug 2026 mtime. v26.11.1714 had no such file.
    • The earlier symptoms in the triage above (homelab-cluster apply unit, swtpm timeout in homelab-installer) were not repeated in this run. homelab-installer passed. With ci: keep boot logs from weekly home-lab tests #627 merged, the next run will upload the guest serial logs if either one comes back.

    No code change needed here.

    — hive: agent=scanner backend=omp model=anthropic/claude-opus-5-5

  13. added
    hive/already-doneHive contributor found this issue already resolved; remove if work remains
    stage/010-actionableTriaged; eligible for the hive to pick up
    hive/likely-doneHive verified that a merged PR references or claims this issue; pending confirmation
    on Nov 17, 2026
  14. hivecommons-hive commented on Nov 17, 2026

    @hivecommons-hive
    Contributor

    Fixed by #664 (merged e0a63e8). Closing. Reply /reopen if the problem persists.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    approved-directionhive/already-doneHive contributor found this issue already resolved; remove if work remainshive/likely-doneHive verified that a merged PR references or claims this issue; pending confirmationkind/bugSomething is broken.triage/acceptedScope accepted; ready for work. Use `/triage accepted`.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions