🐛 Push-reported clusters (vllm-d) show 0 nodes: spoke heartbeats carry no node health #11428

Symptom

The hub fleet page still shows vLLM-d as push-reported but with no node stats:

vLLM-d (GPU) — 0 nodes · 0 vCPU · 0% cpu · 0% mem · — disk · 36 hives

push-reported · heartbeat carries no node health — check spoke node-metrics RBAC and metrics-server (last seen just now)

Maintainer screenshot: /Users/andan02/Downloads/Screenshot 2026-10-10 at 9.15.17 AM.png.

Live read-only evidence

Hub pod is running ghcr.io/hivecommons/hive-hub:8a0539c on hive-oke.

Hub logs today show the hub correctly recognizes vLLM-d as push-reported and receives fresh spoke heartbeats, but has no heartbeat cluster-health fallback for vLLM-d:

cluster health: push-reported cluster, using the health its spokes report over the heartbeat cluster=vllm-d
cluster health: push-reported cluster, using the health its spokes report over the heartbeat cluster=a-ks-wec2
cluster health: using heartbeat fallback cluster=a-ks-wec2

The hub registry has 36 vLLM-d hives with fresh lastHeartbeat values and spoke versions newer than #10567/#10708, for example:

{"id":"hosted-available-vllmd-260731-nq4a","clusterId":"vllm-d","lastHeartbeat":"2026-10-10T13:20:09Z","version":"5.143.1-1-g4d2942a9b","gitHash":"4d2942a","imageRef":"ghcr.io/hivecommons/hive:stable"}
{"id":"hosted-available-vllmd-260731-jnip","clusterId":"vllm-d","lastHeartbeat":"2026-10-10T13:19:54Z","version":"5.143.2-3-gcc6992669","gitHash":"cc69926","imageRef":"ghcr.io/hivecommons/hive:candidate"}

The same log window shows a-ks-wec2 does store/use heartbeat fallback, so hub-side aggregation can work when a spoke sends cluster_health.

The vLLM-d API was not reachable from this workstation with read-only kubectl (context deadline exceeded to api.fmaas-vllm-d...:6443), matching the push-reported design.

Root cause

The hosted-spoke provisioning template records cluster_id in the hub-side SaaS meta.json, but did not render that identity into the spoke runtime config (hub.cluster_id) or Deployment env (HIVE_CLUSTER_ID). The spoke-side heartbeat intentionally gates cluster-health collection on its own resolved hub cluster ID:

if !shouldCollectHeartbeatClusterHealth(b.cfg.Hub.ClusterID) { return nil }

So vLLM-d heartbeats prove liveness and let the hub overlay clusterId from meta.json, but the raw heartbeat omits cluster_health entirely. RBAC-only fixes do not help a spoke that never starts the collector.

Prior attempts:

• 🐛 hub: restore node stats for pull-only clusters via heartbeat health; neutral awaiting-heartbeat state #10567 fixed the gate to use the resolved hub.cluster_id, but existing hosted manifests still did not set one.
• 🐛 hub: call push-reporting clusters push-reported, and say why vllm-d has no node health #10607 improved push-reported wording and old-spoke diagnostics.
• 🐛 hub: push-reported spoke shows no node stats — metrics-first collector dropped capacity #10708 fixed the metrics-first collector and added node-health RBAC/error reporting, but existing vLLM-d spokes still lack the cluster identity that enables the collector.

Fix needed

• Render hub.cluster_id and HIVE_CLUSTER_ID for hosted spokes.
• Keep recent non-empty heartbeat node data when another spoke sends an empty/error health report, so one broken spoke cannot blank a push-reported cluster row.
• Update the no-health banner to call out missing spoke cluster identity before RBAC/metrics-server.
• Document the exact one-time vLLM-d operator command: apply read-only Node/NodeMetrics RBAC and set HIVE_CLUSTER_ID=vllm-d on existing spoke Deployments. Do not apply this from the PR.