You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Summary: Triage of this week's failed weekly home-lab test. Two of the six test scenarios failed, for two unrelated reasons, and neither looks like the same problem as last week's failure. One is a test machine whose emulated security chip stopped responding during installation; the other is a home-lab setup step that failed on the main test machine, whose cause was not saved — a separate report (#623) asks for that log to be kept.
Run 41805416025 (2026-11-04, head main): 4/6 scenarios green (mdns, kubeadm, nvidia, homelab-templates).
e2e (homelab-cluster) — all check assertions matched and all three guests printed PROBE failed=0; rc=1 came from wait "${cp_pid}", because the control-plane guest's boot check saw a failed unit: [FAILED] Failed to start Apply the Bluefin … components to this node's cluster. (bluefin-homelab-apply.service, which has Restart=on-failure, so a first-attempt failure is enough to fail the harness). The unit's own output is in cp-set/dogfood-serial.log, which the upload step does not collect → #623.
e2e (homelab-installer) — FAIL: cp: install timed out after 1207 s. cp-2-install.ttyS0.log stops right after: tpm tpm0: Operation Timed out / tpm_crb_acpi MSFT0101:00: probe with driver tpm_crb_acpi failed with error -62
and cp-2-install.swtpm.log is empty — the emulated TPM (swtpm) stopped answering, so the install never progressed. Looks like an emulator/runner fault rather than an image change; worth a re-check on the next weekly run before treating it as a regression.
Previous failure (run 40662525415, 2026-10-28) was different: homelab-cluster node never reported (FAIL: boot probe did not report within 3300s), while the CP and the wrong-passphrase guest passed.
Three different symptoms in two runs → no single code cause yet. Next step: land #623 so the next failure carries the guest serial logs.
Good first issue — withdrawn by hosted-projectbluefin-knuckle-gjvq: this issue is not open to new contributors — it carries the blocked label, so it cannot be worked right now. The good first issue and help wanted labels have been removed (2026-11-10T00:28:17Z). Sorry for the noise! Issues open to new contributors are listed on the relay page: https://hosted-projectbluefin-common-nmq5.hive.hivecommons.dev/contribute
Summary: This weekly home-lab test failure can't be fixed yet because the log that explains it was never saved. A pending change that keeps those logs (#627, for the report about lost boot logs, #623) has to land first. After it merges, the next weekly run will show the real cause.
No weekly run since the triaged one: the latest e2e-opt-in.yml run is still 41805416025 (2026-11-04).
homelab-cluster: the output from bluefin-homelab-apply.service is only in cp-set/dogfood-serial.log, and that file is not uploaded. PR ci: keep boot logs from weekly home-lab tests #627 (still open) adds it to the upload.
homelab-installer: the swtpm emulated chip timed out, and its log is empty. Nothing in the repo points to a cause yet, so this needs another run to confirm.
Making a code change now would only be a guess. Waiting on #627 and the next weekly run.
On e2e (homelab-cluster): if the boot check is reading systemd unit state, it's going to trip on any [FAILED] line from bluefin-homelab-apply.service even when Restart=on-failure recovers it on the next attempt — a first attempt failing isn't really a test failure. Having the check wait out the restart window and then assert on the unit's settled state (systemctl is-failed / is-active) would make that signal meaningful instead of flaky.
That said, we still don't know why the apply failed the first time, since cp-set/dogfood-serial.log never gets uploaded — that's #623, and I'd fix that before tuning the assertion. Is the harness assertion change something you want as its own PR, or should it ride along with #623?
Summary: Last week's weekly home-lab test failed for a reason that has since been fixed. The test machines' internal clocks jumped back to 2011, so they rejected the release's valid digital signature as "not yet made" and wouldn't download the system. A fix from a few days ago that keeps the clock from jumping back (#664) is in every release since then. The next weekly run tests the newest release, and when it passes this issue closes on its own.
Run 42993076271 (2026-11-11, release v26.11.1714) failed 5/6 scenarios. Every netbooted guest stopped at the same point:
systemd-importd: (transfer1) DOWNLOAD INVALID: Signature verification failed.
Failed to start Download of http://10.0.2.2:8765/bluefin-server_26.11.1714.raw.
The signature is valid. The host step verified it (Good signature … 7A87E575…9C34), and I reproduced Good signature offline. I ran the release's own initrd gpg inside a chroot with the exact arguments systemd-pull uses, against the UKI's /usr/lib/systemd/import-pubring.pgp.
The guest clock is the cause. The installer guest's serial log shows time was further ahead than 15y after built-in epoch, clock reset to Thu 2011-11-10 15:00:00. systemd's built-in epoch is FSDK's 2011-11-10, so from 2026-11-10 on, PID 1 resets a correct RTC to 2011. gpg then rejects the 2026 signature and key because they look like they come from the future. The 2026-11-04 run fell one week before the 15-year window ran out, which explains why mdns, kubeadm and nvidia passed then and failed now.
Fixed by fix(build): repair shared caching and boot clock epoch #664 (merged 2026-11-14, e0a63e8, first in v26.11.2108), which adds a /usr/lib/clock-epoch clock floor. The latest release v26.11.2264 (the one the next scheduled run picks up) has it in the netboot initrd: usr/lib/clock-epoch has an Aug 2026 mtime. v26.11.1714 had no such file.
The earlier symptoms in the triage above (homelab-cluster apply unit, swtpm timeout in homelab-installer) were not repeated in this run. homelab-installer passed. With ci: keep boot logs from weekly home-lab tests #627 merged, the next run will upload the guest serial logs if either one comes back.
Activity
hivecommons-hive commentedon Nov 5, 2026
Summary: Triage of this week's failed weekly home-lab test. Two of the six test scenarios failed, for two unrelated reasons, and neither looks like the same problem as last week's failure. One is a test machine whose emulated security chip stopped responding during installation; the other is a home-lab setup step that failed on the main test machine, whose cause was not saved — a separate report (#623) asks for that log to be kept.
Run 41805416025 (2026-11-04, head
main): 4/6 scenarios green (mdns,kubeadm,nvidia,homelab-templates).e2e (homelab-cluster)— allcheckassertions matched and all three guests printedPROBE failed=0;rc=1came fromwait "${cp_pid}", because the control-plane guest's boot check saw a failed unit:[FAILED] Failed to start Apply the Bluefin … components to this node's cluster.(bluefin-homelab-apply.service, which hasRestart=on-failure, so a first-attempt failure is enough to fail the harness). The unit's own output is incp-set/dogfood-serial.log, which the upload step does not collect → #623.e2e (homelab-installer)—FAIL: cp: install timed outafter 1207 s.cp-2-install.ttyS0.logstops right after:tpm tpm0: Operation Timed out/tpm_crb_acpi MSFT0101:00: probe with driver tpm_crb_acpi failed with error -62and
cp-2-install.swtpm.logis empty — the emulated TPM (swtpm) stopped answering, so the install never progressed. Looks like an emulator/runner fault rather than an image change; worth a re-check on the next weekly run before treating it as a regression.Previous failure (run 40662525415, 2026-10-28) was different:
homelab-clusternode never reported (FAIL: boot probe did not report within 3300s), while the CP and the wrong-passphrase guest passed.Three different symptoms in two runs → no single code cause yet. Next step: land #623 so the next failure carries the guest serial logs.
ci-maintainer triage
— hive: agent=ci-maintainer backend=copilot model=claude-opus-5.5 copilot=1.0.88
Danathar commentedon Nov 9, 2026
/triage accepted
/kind bug
hivecommons-hive commentedon Nov 9, 2026
Good first issue— withdrawn by hosted-projectbluefin-knuckle-gjvq: this issue is not open to new contributors — it carries theblockedlabel, so it cannot be worked right now. Thegood first issueandhelp wantedlabels have been removed (2026-11-10T00:28:17Z). Sorry for the noise! Issues open to new contributors are listed on the relay page: https://hosted-projectbluefin-common-nmq5.hive.hivecommons.dev/contributeKaribhou commentedon Nov 10, 2026
Summary: This weekly home-lab test failure can't be fixed yet because the log that explains it was never saved. A pending change that keeps those logs (#627, for the report about lost boot logs, #623) has to land first. After it merges, the next weekly run will show the real cause.
e2e-opt-in.ymlrun is still 41805416025 (2026-11-04).homelab-cluster: the output frombluefin-homelab-apply.serviceis only incp-set/dogfood-serial.log, and that file is not uploaded. PR ci: keep boot logs from weekly home-lab tests #627 (still open) adds it to the upload.homelab-installer: the swtpm emulated chip timed out, and its log is empty. Nothing in the repo points to a cause yet, so this needs another run to confirm.Making a code change now would only be a guess. Waiting on #627 and the next weekly run.
— hive: agent=scanner backend=omp model=anthropic/claude-opus-5-5
🐝 Hive Agent:
contributor| SHA:unknowngaliliya commentedon Nov 10, 2026
On
e2e (homelab-cluster): if the boot check is reading systemd unit state, it's going to trip on any[FAILED]line frombluefin-homelab-apply.serviceeven whenRestart=on-failurerecovers it on the next attempt — a first attempt failing isn't really a test failure. Having the check wait out the restart window and then assert on the unit's settled state (systemctl is-failed/is-active) would make that signal meaningful instead of flaky.That said, we still don't know why the apply failed the first time, since
cp-set/dogfood-serial.lognever gets uploaded — that's #623, and I'd fix that before tuning the assertion. Is the harness assertion change something you want as its own PR, or should it ride along with #623?github-actions commentedon Nov 11, 2026
Failed again: https://github.com/projectbluefin/server/actions/runs/42993076271
hivecommons-hive commentedon Nov 17, 2026
Hive status — updated in place; edits do not notify. Full history is in the Activity Monitor.
Unblocked: #627 is no longer open; re-triaging.
Karibhou commentedon Nov 17, 2026
Summary: Last week's weekly home-lab test failed for a reason that has since been fixed. The test machines' internal clocks jumped back to 2011, so they rejected the release's valid digital signature as "not yet made" and wouldn't download the system. A fix from a few days ago that keeps the clock from jumping back (#664) is in every release since then. The next weekly run tests the newest release, and when it passes this issue closes on its own.
Run 42993076271 (2026-11-11, release
v26.11.1714) failed 5/6 scenarios. Every netbooted guest stopped at the same point:Good signature … 7A87E575…9C34), and I reproducedGood signatureoffline. I ran the release's own initrdgpginside a chroot with the exact argumentssystemd-pulluses, against the UKI's/usr/lib/systemd/import-pubring.pgp.time was further ahead than 15y after built-in epoch, clock reset to Thu 2011-11-10 15:00:00. systemd's built-in epoch is FSDK's 2011-11-10, so from 2026-11-10 on, PID 1 resets a correct RTC to 2011. gpg then rejects the 2026 signature and key because they look like they come from the future. The 2026-11-04 run fell one week before the 15-year window ran out, which explains why mdns, kubeadm and nvidia passed then and failed now.e0a63e8, first inv26.11.2108), which adds a/usr/lib/clock-epochclock floor. The latest releasev26.11.2264(the one the next scheduled run picks up) has it in the netboot initrd:usr/lib/clock-epochhas an Aug 2026 mtime.v26.11.1714had no such file.No code change needed here.
— hive: agent=scanner backend=omp model=anthropic/claude-opus-5-5
hivecommons-hive commentedon Nov 17, 2026
Fixed by #664 (merged e0a63e8). Closing. Reply /reopen if the problem persists.