Skip to content

πŸ› hub: restore node stats for pull-only clusters via heartbeat health; neutral awaiting-heartbeat state - #10567

Merged
clubanderson merged 4 commits into
v5from
fix/pull-only-heartbeat-health-10559
Oct 5, 2026
Merged

clubanderson merged 4 commits into
v5from
fix/pull-only-heartbeat-health-10559

Conversation

@clubanderson

Copy link
Copy Markdown
Member

Closes #10559

Root cause

Pull-only clusters intentionally make buildSingleClusterHealth return errClusterPullOnly, after which the hub falls back to s.heartbeatHealth[clusterID]. The hub stores heartbeat health under the resolved registry/SaaS entry.ClusterID, but current vllm-d spokes were heartbeating without a cluster health report: the spoke only populated ClusterHealth when the HIVE_CLUSTER_ID environment variable was set, even though boot had already resolved the effective cluster ID from config or env into b.cfg.Hub.ClusterID.

Live evidence from the hub showed vllm-d spokes heartbeating every ~2 minutes, repeated pull-only cluster-health queries for vllm-d, and no cluster health: using heartbeat fallback / stored health for vllm-d in the sampled logs. The vllm-d kube context was not reachable from this workstation, matching the pull-only design.

Fix

  • Collect spoke heartbeat cluster health whenever the resolved hub cluster ID is non-empty, not only when HIVE_CLUSTER_ID is present.
  • Add structured status/note fields for per-cluster health states.
  • Render pull-only clusters with no heartbeat health as a neutral informational state: pull-only Β· awaiting spoke heartbeat, including last registry heartbeat age when known.
  • Mark stale heartbeat health with a warning status/note instead of treating the configured pull-only condition as a red hub query error.
  • Added regression coverage for the spoke-side gate, pull-only awaiting-heartbeat state, and stale heartbeat conversion.

Validation

  • gofmt -l on changed Go files
  • git diff --check
  • Per AGENTS.md, did not run local go test/go build; CI is the required verifier.

@kubestellar-prow

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign nicholasjackson for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@kubestellar-prow kubestellar-prow Bot added the dco-signoff: yes Indicates the PR's author has signed the DCO. label Oct 5, 2026
@github-actions

github-actions Bot commented Oct 5, 2026

Copy link
Copy Markdown
Contributor

Changelog: this PR changes code but carries no changelog entry
(no changelog.d/ fragment and no CHANGELOG.md edit).

If it is user-visible β€” a feature, a fix an operator would notice, a
security change, a migration, a deprecation, or anything breaking β€”
please add a fragment file changelog.d/<category>-<pr-or-slug>.md
containing exactly your entry (see changelog.d/README.md; the
category is one of added/changed/deprecated/fixed/security). Please
do not append to CHANGELOG.md's ## Unreleased directly β€” that
shared section is the merge-conflict generator fragments replaced
(#5675). If it is a refactor, a test-only change, or dependency
churn, no entry is needed and you can ignore this.

This is a reminder, not a gate; it never blocks a merge.

@kubestellar-prow kubestellar-prow Bot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Oct 5, 2026
@github-actions

github-actions Bot commented Oct 5, 2026 •

Copy link
Copy Markdown
Contributor

Open PR overlap check

Base branch: v5

Conflicts are checked with every PR merged onto the current tip of v5, so a PR that only needs a rebase is not reported as conflicting with its siblings.

PR Title Relation Files
#10580 πŸ› hub: pull-only clusters use heartbeat health; neutral awaiting-heartbeat state CONFLICT src/cmd/hive/main.go
src/pkg/hub/assets/dashboard.html
src/pkg/hub/saas_cluster_health.go
#10533 πŸ› fix: refuse automated claims on needs-human issues same files src/cmd/hive/main.go
#10562 ✨ hub dashboard: NPS feedback gets its own collapsible section above Users same files src/pkg/hub/assets/dashboard.html

Generated by changelog-fragment-guard from the PR title (#5675).
Edit the wording freely, or replace this file with the no-changelog
label if the change is not user-visible.

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Signed-off-by: clubanderson <407614+clubanderson@users.noreply.github.com>
Signed-off-by: Andy Anderson <andy@clubanderson.com>
@github-actions github-actions Bot added the conflicts-with-open-pr Open PR has a merge conflict with another open PR label Oct 5, 2026
@clubanderson
clubanderson merged commit db4d5c2 into v5 Oct 5, 2026
58 of 59 checks passed
@kubestellar-prow
kubestellar-prow Bot deleted the fix/pull-only-heartbeat-health-10559 branch October 5, 2026 14:19
@github-actions

github-actions Bot commented Oct 5, 2026

Copy link
Copy Markdown
Contributor

Thank you for your contribution! Your PR has been merged.

We'd love to hear how your experience was: share feedback

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

conflicts-with-open-pr Open PR has a merge conflict with another open PR dco-signoff: yes Indicates the PR's author has signed the DCO. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

πŸ› hub: vllm-d (pull-only) cluster no longer reports node stats β€” heartbeat health fallback is empty

1 participant