Skip to content

πŸ› hub: pull-only clusters use heartbeat health; neutral awaiting-heartbeat state - #10580

Closed
hivecommons-hive[bot] wants to merge 2 commits into
v5from
scanner/fix-10559
Closed

hivecommons-hive[bot] wants to merge 2 commits into
v5from
scanner/fix-10559

Conversation

@hivecommons-hive

Copy link
Copy Markdown
Contributor

Root cause (code-level)

For a pull-only cluster, buildSingleClusterHealth returns errClusterPullOnly by design, and the hub falls back to heartbeatHealth[clusterID]. That entry is only written when a heartbeat carries cluster_health.

Spoke-side gap: cmd/hive attached ClusterHealth only when the HIVE_CLUSTER_ID env var was set. The payload's cluster_id, which the hub uses as the heartbeatHealth key, comes from the resolved id: hub.cluster_id from hive.yaml, overridden by the env var. A firewalled spoke configured through hive.yaml (hub.cluster_id: vllm-d) therefore heartbeated with cluster_id=vllm-d but never sent node health. The hub had nothing to fall back to and showed 0 nodes plus the red error.

I checked the hub-side key path too. On write, entry.ClusterID comes from the SaaS record's ClusterID, then the payload cluster_id, then defaultClusterID. On read, the registry id is used. The two match in code. heartbeatHealthStaleness only marks entries stale; nothing is evicted. Nothing skips the report when the hive count is 0.

Fix

  • Spoke: new heartbeatClusterHealth(clusterID, collect) in boot_seams.go gates collection on the resolved b.cfg.Hub.ClusterID instead of the env var alone. With no cluster id it still sends nothing and doesn't run the collector.
  • Hub: when a pull-only cluster has no heartbeat health, it now returns PerClusterHealth{PullOnly, AwaitingHeartbeat, LastSeen} with no Error. LastSeen is the newest LastHeartbeat of any registry hive on that cluster, or never. Heartbeat-backed pull-only entries also carry pull_only.
  • Dashboard: awaiting state renders as a neutral pull-only Β· awaiting spoke heartbeat (last seen: never/<age>) line. Red is kept for real errors and for heartbeat data that went stale (data_stale), which the dashboard didn't show before.
  • Regression tests: src/pkg/hub/cluster_health_pull_only_test.go, src/cmd/hive/boot_heartbeat_cluster_health_test.go.
  • Changelog: changelog.d/fixed-10559.md.

Remains (needs-human)

The card's 0 hives means no registry hive is currently keyed to vllm-d at all. The spoke-side gap above is real, but it doesn't explain that on its own. Live hub logs are needed to confirm:

  • whether a spoke on vllm-d is heartbeating at all;
  • if one is, which cluster_id it is filed under. A SaaS record whose cluster_id was defaulted to hive-oke by ensureClusterIDForClaim would override the spoke's own report;
  • whether metrics-server is reachable from that spoke. CollectClusterHealth returns nil silently, at debug level only, when it isn't.

With this change, the new last seen value on the card answers the first question directly.

Refs #10559 (needs-human: live hub logs needed to confirm whether a vllm-d spoke is heartbeating)

β€” hive: agent=scanner backend=copilot model=claude-fable-5.1 copilot=1.0.88 requested_by=@clubanderson

… awaiting state

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: scanner <scanner@hive.kubestellar.io>
Co-authored-by: clubanderson <407614+clubanderson@users.noreply.github.com>
@kubestellar-prow kubestellar-prow Bot added the dco-signoff: yes Indicates the PR's author has signed the DCO. label Oct 5, 2026
@kubestellar-prow

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign clubanderson for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@kubestellar-prow kubestellar-prow Bot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Oct 5, 2026
@hivecommons-hive

Copy link
Copy Markdown
Contributor Author

hive/awaiting-ci: CI pending β€” sweep will merge when green.


🐝 Hive Agent: scanner | Instance: hosted-available-oke-11-placeholder-r05x | SHA: unknown

β€” hive: agent=scanner backend=copilot model=claude-fable-5.1 copilot=1.0.88

@github-actions

github-actions Bot commented Oct 5, 2026 •

Copy link
Copy Markdown
Contributor

Open PR overlap check

Base branch: v5

This PR conflicts with v5 itself. Rebase it first: until it merges cleanly into the base, conflicts with other open PRs cannot be told apart from base drift, so only same-file overlaps are listed.

Conflicts are checked with every PR merged onto the current tip of v5, so a PR that only needs a rebase is not reported as conflicting with its siblings.

PR Title Relation Files
#10533 πŸ› fix: refuse automated claims on needs-human issues same files src/cmd/hive/main.go

@github-actions github-actions Bot added conflicts-with-open-pr Open PR has a merge conflict with another open PR copilot labels Oct 5, 2026
@github-actions github-actions Bot removed the conflicts-with-open-pr Open PR has a merge conflict with another open PR label Oct 5, 2026
@hivecommons-hive

Copy link
Copy Markdown
Contributor Author

Supersession sweep found that this PR's claimed issue is already closed by another merged PR.

  • Claimed issue: hive#10559
  • Closing PR: #10567

this PR touches files the merged PR did not. A human should decide whether to close this PR.

This comment is edited in place by the supersession sweep; it is not duplicated.

…atchet

The awaiting-heartbeat and stale notes added inline style attributes with raw
rgba() colors and px padding, pushing the hub static pages over the
TestStyleRatchet baseline (961/456/167). Move them to a token-only
.cluster-note class so the ratchet counts stay exact.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: adjudicator <adjudicator@hive.kubestellar.io>
@kubestellar-prow kubestellar-prow Bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Oct 5, 2026
@kubestellar-prow

Copy link
Copy Markdown
Contributor

PR needs rebase.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[reviewer] recommend close: superseded β€” issue #10559 was closed 2026-10-05T14:19Z by human-authored PR #10567 (fix/pull-only-heartbeat-health-10559, commit 4abc571) merged to v5 before this PR could land. #10567 implements the same fix: pull-only clusters get heartbeat-reported health, a PerClusterHealth status/note (awaiting_heartbeat / stale_heartbeat), a neutral dashboard message class, and the boot_seams wiring. This branch edits the same regions of saas_cluster_health.go, boot_seams.go, main.go and dashboard.html with a parallel design (awaiting_heartbeat/pull_only bool fields), so merging it would duplicate fields and rendering paths. evidence: git log origin/v5 -S awaiting_heartbeat β†’ 4abc571; merge db4d5c2 of #10567; gh issue view 10559 β†’ CLOSED/COMPLETED at 14:19:09Z. tests: CI failure (TestStyleRatchet: +2 inline style=, +2 raw rgba colors, +2 raw padding over the 961/456/167 baseline) was confirmed with a scratch build of the ratchet counters and is fixed on this branch by commit 1f0e2b7, but that is moot given supersession. remaining risk: none β€” the shipped fix in v5 covers the issue; the one difference (#10567 does not expose a pull_only bool) is not needed by the dashboard.

β€” hive: agent=adjudicator backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive

Copy link
Copy Markdown
Contributor Author

Closing as superseded by #10567 (merged to v5, closed #10559). See the reviewer adjudication comment above.

@hivecommons-hive

Copy link
Copy Markdown
Contributor Author

βœ… Escalation cleared β€” back in the automated lane

CI is green on the current head β€” returning this PR to the automated lane.

The needs-human label has been removed; the fix loop owns this PR again.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

copilot dco-signoff: yes Indicates the PR's author has signed the DCO. needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. reviewer-recommend-close size/L Denotes a PR that changes 100-499 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

0 participants