diff --git a/changelog.d/fixed-10607.md b/changelog.d/fixed-10607.md new file mode 100644 index 0000000000..ce8d0efe88 --- /dev/null +++ b/changelog.d/fixed-10607.md @@ -0,0 +1 @@ +- hub: call push-reporting clusters push-reported, and say why vllm-d has no node health (#10607) diff --git a/src/changelog.d/changed-push-reported-clusters-10601.md b/src/changelog.d/changed-push-reported-clusters-10601.md new file mode 100644 index 0000000000..92083b97cb --- /dev/null +++ b/src/changelog.d/changed-push-reported-clusters-10601.md @@ -0,0 +1 @@ +- Changed hub cluster-health wording to call heartbeat-only clusters push-reported and explain missing node health from old spokes (#10601) diff --git a/src/cmd/hive/main.go b/src/cmd/hive/main.go index caccc97060..5f269b3f4c 100644 --- a/src/cmd/hive/main.go +++ b/src/cmd/hive/main.go @@ -1393,7 +1393,7 @@ func (b *boot) wireBootClosures() { } // Prefer the host our OWN Route/Ingress actually serves. The synthesised // "." below is only correct when this spoke is fronted by - // the hub's wildcard domain; pull-only clusters must report their live host. + // the hub's wildcard domain; push-reported clusters must report their live host. if host := spoke.SpokeServedHost(b.ctx); host != "" { return "https://" + host } @@ -4444,7 +4444,7 @@ func (b *boot) bootHeartbeatWith(deps bootHeartbeatDeps) { // Hash only, never the raw token: lets the hub verify this // spoke's upgrade-proof credential without reading the // hive-secrets secret from a cluster it may not reach - // (pull-only). Empty when no token is configured. + // (push-reported). Empty when no token is configured. DashboardTokenHash: func() string { if b.cfg.Dashboard.AuthToken == "" { return "" @@ -4636,7 +4636,7 @@ func (b *boot) bootHeartbeatWith(deps bootHeartbeatDeps) { // Component reach counters (#3993, phase 2a of #3973): per // (component, running commit) span counts aggregated in-process, // exporter or not (D2) — the heartbeat is the only channel that - // reaches every spoke, pull-only ones included (D1). nil until + // reaches every spoke, push-reported ones included (D1). nil until // the first span, which the hub reads as "no data", never as // zero reach. Capped at tracing.MaxReachComponents entries. ComponentReach: tracing.ReachSnapshot(), diff --git a/src/docs/design/README.md b/src/docs/design/README.md index 091ca998aa..2b3b80f071 100644 --- a/src/docs/design/README.md +++ b/src/docs/design/README.md @@ -72,9 +72,9 @@ that status is the thing to check before treating a page as current behaviour: generations code and the session-cookie domain have landed (`src/pkg/hub/hub_generations*.go`, `hub_cookie_generations.go`); the remaining per-domain adoptions listed under "Follow-on PRs" are still pending. -- [Wrapped master delivery to pull-only spokes](master-delivery-wrapped.md) — +- [Wrapped master delivery to push-reported spokes](master-delivery-wrapped.md) — **design only**. Read it after master-key-rotation.md: rotation is complete on - the hub and delivers nothing to the two thirds of the fleet on pull-only + the hub and delivers nothing to the two thirds of the fleet on push-reported clusters, which the hub cannot write to by design. Records Option D — the spoke generates a keypair, publishes the public half over its own outbound heartbeat, and the hub seals each new master to it — and why Options A and B were @@ -165,7 +165,7 @@ that status is the thing to check before treating a page as current behaviour: The first inbound GitHub trigger: a human summons an agent by mentioning the App on an issue or PR, mirroring the Linear agent-session path (`pkg/linearagent`) that already exists. Poll-first transport so it works on - pull-only spokes, a mention grammar with `ask ` routing under the + push-reported spokes, a mention grammar with `ask ` routing under the `linear.session_agent` resolution rule, a kick built like the Linear responder's, replies through `Converse` on the existing watcher write path with no new outbound surface, and seven guards each mapped to a mechanism diff --git a/src/docs/design/escalation-surfaces.md b/src/docs/design/escalation-surfaces.md index 7833622f52..17afb80617 100644 --- a/src/docs/design/escalation-surfaces.md +++ b/src/docs/design/escalation-surfaces.md @@ -75,7 +75,7 @@ Severity is the routing key, set by the producer: ### Outbound (phase 1) Plain SMTP submission — `net/smtp` with STARTTLS/implicit-TLS, no third-party -mail dependency. Works from pull-only clusters (outbound 465/587 only). +mail dependency. Works from push-reported clusters (outbound 465/587 only). ```yaml email: @@ -106,7 +106,7 @@ email: ### Inbound reply-to-act (phase 2, explicitly later) Mirrors the Linear inbound model (`pkg/linearagent`: guards → ack → kick, -decoupled): an IMAP poller (outbound-only, fits pull-only clusters) watches a +decoupled): an IMAP poller (outbound-only, fits push-reported clusters) watches a dedicated mailbox; a reply to an escalation mail can carry exactly the verbs the mail offered (`approve`, `deny`, `kick : `). @@ -185,7 +185,7 @@ sinks are tested against local fakes (`net/smtp` test server, httptest). ## What this deliberately does not do - No webhook *receiver* for provider callbacks (PagerDuty ack-back etc.) — - outbound-only keeps the pull-only story intact. + outbound-only keeps the push-reported story intact. - No templating language for mail bodies; producers compose, full stop. - No SMS provider (Twilio) — Pushover/ntfy cover the phone story without per-message billing; revisit only on demand. diff --git a/src/docs/design/github-mention-triggers.md b/src/docs/design/github-mention-triggers.md index d3ed5bad17..72ebe9466e 100644 --- a/src/docs/design/github-mention-triggers.md +++ b/src/docs/design/github-mention-triggers.md @@ -99,7 +99,7 @@ the same PR — never as a config key first. The Linear path is webhook-only because Linear pushes and demands a 5-second answer. GitHub is different in a way that matters for this fleet: most spokes -run on pull-only clusters the hub cannot write to (see +run on push-reported clusters the hub cannot write to (see [wrapped master delivery](master-delivery-wrapped.md)), and such a spoke has no public URL for GitHub to deliver to either. A webhook-only design would work on the hosted hub and nowhere else. @@ -300,7 +300,7 @@ agents: mention-kicked agent without `Converse` cannot answer. 3. **Webhook accelerator.** The hub's verifier gains the three event types and a spoke-side mount; deliveries advance the poller's watermark. Hub-to-spoke - relay for pull-only clusters is a later question that the master-delivery + relay for push-reported clusters is a later question that the master-delivery design already frames. ## Non-goals diff --git a/src/docs/design/master-delivery-wrapped.md b/src/docs/design/master-delivery-wrapped.md index e0f7288a34..8e1938a6b1 100644 --- a/src/docs/design/master-delivery-wrapped.md +++ b/src/docs/design/master-delivery-wrapped.md @@ -1,4 +1,4 @@ -# Wrapped master delivery to pull-only spokes +# Wrapped master delivery to push-reported spokes Status: DESIGN ONLY. No implementation. This document extends `master-key-rotation.md` and should be read after it. @@ -31,7 +31,7 @@ regenerates the hub's *private* seed via `SSOSigningSeedFromMaster` The reconcile lane is therefore an optimisation, not a dependency. It pre-computes what the spoke would compute anyway. That is why the fleet ran -normally while the lane was dead, and why the 44 pull-only spokes work today. +normally while the lane was dead, and why the 44 push-reported spokes work today. **The master is the one value a spoke cannot self-derive.** Delivering a *new* master after rotation is the lane's only irreplaceable job, and on 44 of 66 @@ -41,7 +41,7 @@ spokes there is no delivery path. returns false unconditionally. Per the operator correction of 2026-08-14, `pull_only` is an **intentional architectural boundary on both clusters** — the hub is not meant to hold write credentials into the heartbeat-only clusters. The -field's own doc comment already says this plainly: a pull-only cluster's spokes +field's own doc comment already says this plainly: a push-reported cluster's spokes "connect outbound over the heartbeat and nothing here can kubectl into them" (`src/pkg/hub/saas_provision.go:255-256`). @@ -308,7 +308,7 @@ heartbeat. For any hive that is not, there is no race at all — **the attacker pins by default, unopposed, with no timing requirement whatsoever.** The categories are not hypothetical: -- **Not yet rolled to the new image.** Migration step 1 reaches pull-only +- **Not yet rolled to the new image.** Migration step 1 reaches push-reported spokes via self-upgrade, which is not instantaneous or simultaneous. Between the hub gaining the pin store and the last spoke rolling, every unrolled hive is unopposed. @@ -343,7 +343,7 @@ the estimate of how much of the fleet the rule protects on day one. **This is unavoidable given the constraints, and it should be stated as a residual rather than engineered around.** The hub cannot reach in to seed a key. There is no out-of-band channel from the hub. The only pre-existing shared -secret with a pull-only spoke *is* the master. Bootstrapping trust from +secret with a push-reported spoke *is* the master. Bootstrapping trust from something that is not the master requires something the spoke or the provisioning process supplies — which is §5. @@ -358,7 +358,7 @@ This is the only candidate for a non-master anchor, since the hub cannot reach in and there is no out-of-band hub channel. Provisioning is the moment a spoke is created, and the hub controls the -manifest even for pull-only clusters — the manifest is applied by an operator +manifest even for push-reported clusters — the manifest is applied by an operator context, not by the hub. The template already injects per-hive secret material (`src/pkg/hub/saas_provision.go:2567-2589`: `HeartbeatKey`, `SessionPublicKey`, `SSOPublicKey`, `TerminalKey`, `InviteKey`), and the `/secrets` read-only projected mount @@ -418,13 +418,13 @@ initiate a connection to a spoke.** | 6 | Enrolment value added to provisioning template for new hives (§5) | operator applies manifest | new hives only | **no** | Step 1 is the only one needing a spoke image roll, and it needs **no hub -action at all** — which is precisely why it works for pull-only clusters. The +action at all** — which is precisely why it works for push-reported clusters. The spoke's image is updated by its existing self-upgrade path (`HeartbeatResponse.SwitchToTag`, `src/pkg/hub/spoke/heartbeat.go:2133-2137` (field at `:1698`)), whose doc comment records that this exists for exactly this reason: "Used for branch switches on clusters the hub can't reach over kubectl — the spoke has in-cluster RBAC (hive-self-upgrade role) to patch its own deployment." The delivery mechanism -for the fix is one the pull-only boundary already accommodates. +for the fix is one the push-reported boundary already accommodates. Step 4's carrier has direct precedent. `HeartbeatResponse.PendingGateway` (`src/pkg/hub/spoke/heartbeat.go:2167-2172`) is a **secret** delivered on the heartbeat response, @@ -459,7 +459,7 @@ SafeToRetirePrevious = hasPrevious ``` One critical adjustment. `FleetFullyObserved` is defined in terms of -*Deployment-read* observation, and pull-only spokes are unreachable by +*Deployment-read* observation, and push-reported spokes are unreachable by definition — so under Option D the 44 must become observable by a different route: **their heartbeat publication is the observation.** A hive that is publishing a pinned wrapping key and receiving wrapped masters is converged, @@ -475,7 +475,7 @@ here. The resolution is that the **denominator must stay registry-sourced** while only the **numerator** becomes heartbeat-sourced. A paused or vanished -pull-only spoke then keeps its place in the denominator, never publishes, and +push-reported spoke then keeps its place in the denominator, never publishes, and correctly blocks retirement. What must never be built is a denominator of "hives that heartbeated recently". → this is the single most likely place for a test to pass for the wrong reason; see §10. @@ -607,7 +607,7 @@ anticipates: If `HIVE_HUB_SECRET` is stripped and `HIVE_HEARTBEAT_KEY` is present, the spoke authenticates on lane 1 (`src/pkg/hub/hub_keys.go:456-458`) and never needs the master — fine. But after a rotation, `HIVE_HEARTBEAT_KEY` is stale. On a reachable spoke -the reconcile lane patches it. **On a pull-only spoke nothing patches it**, and +the reconcile lane patches it. **On a push-reported spoke nothing patches it**, and the spoke cannot self-derive a fresh one without the master. So it must authenticate with the *old* bearer, which works only while the previous generation is still acceptable (`acceptableGenerations`, @@ -615,7 +615,7 @@ generation is still acceptable (`acceptableGenerations`, This yields a workable but strictly bounded property: -- **A pull-only spoke can have `HIVE_HUB_SECRET` stripped** once it holds a +- **A push-reported spoke can have `HIVE_HUB_SECRET` stripped** once it holds a pinned wrapping key and a valid `HIVE_HEARTBEAT_KEY`. - **It must then receive and apply each wrapped master within `defaultVerifyWindow`** (7 days, `src/pkg/hub/hub_generations.go:118`), because after that @@ -624,8 +624,8 @@ This yields a workable but strictly bounded property: verify window is permanently stranded** and requires operator intervention on its own cluster. -That last bullet is the cost of closing RESIDUAL-1 on pull-only spokes, and it -is not small. Today a stranded pull-only spoke self-heals the moment it comes +That last bullet is the cost of closing RESIDUAL-1 on push-reported spokes, and it +is not small. Today a stranded push-reported spoke self-heals the moment it comes back, because it holds the master and re-derives everything. After stripping, it does not. @@ -994,7 +994,7 @@ access (§6). Steps 1–5 need nothing from them. 1. F20 (fail closed on unreadable generations file) — independent, small. 2. Spoke keypair generation + publication (§3, migration step 1). Ships in the - spoke image, needs no hub action, reaches pull-only clusters via + spoke image, needs no hub action, reaches push-reported clusters via self-upgrade. 3. Hub-side pin store + coverage counter folded into `FleetFullyObserved` (§6). 4. F19 (wire `provisionGenerationSet()` to the live set). diff --git a/src/docs/design/master-key-rotation.md b/src/docs/design/master-key-rotation.md index ef1301a079..b13c668ccb 100644 --- a/src/docs/design/master-key-rotation.md +++ b/src/docs/design/master-key-rotation.md @@ -549,7 +549,7 @@ generation, and strand them when retirement fires **unconditionally** at into a 7-day-delayed outage". So `rotateMasterSecret` now refuses unless `FleetFullyObserved` — the last sweep -saw every admitted hive with none unreachable. A pull-only cluster increments +saw every admitted hive with none unreachable. A push-reported cluster increments `UnreachableHives` for every hive on it, so this is **false on the production hub today and stays false** until every spoke is reachable. The wiring lands now; the ability to fire it unlocks only when the thing that makes it safe exists. diff --git a/src/docs/design/pr-reach-telemetry.md b/src/docs/design/pr-reach-telemetry.md index 3a4bc9c7e1..8d80b56ce9 100644 --- a/src/docs/design/pr-reach-telemetry.md +++ b/src/docs/design/pr-reach-telemetry.md @@ -71,7 +71,7 @@ diff's lines run." Line/function-level attribution is out of scope for phase 1 The four design decisions, accepted in #3973: - **D1 — Aggregation is spoke-side and rides the heartbeat.** A span-backend - query can never see the pull-only spokes, and the `otel:` exporter is opt-in + query can never see the push-reported spokes, and the `otel:` exporter is opt-in — most spokes export nowhere. A hub-side span query would report ~0 reach for most of the fleet and call it "unused": the anchoring-rule trap, one level up. The heartbeat is the outbound channel every spoke already opens. diff --git a/src/docs/documentation-map.md b/src/docs/documentation-map.md index f7557a8900..c76997cd99 100644 --- a/src/docs/documentation-map.md +++ b/src/docs/documentation-map.md @@ -164,7 +164,7 @@ Documentation for the current Hive line (branch `v5`; the code and docs live und - [Podman CI runner map](podman-ci-runner-map.md) — measured hosted-runner capabilities and which Podman lane goes where; SELinux is the only lane needing non-hosted infrastructure. - [CI runner labels](ci-runner-labels.md) — how `runs-on:` picks the self-hosted fleet, why a fork must degrade to a GitHub-hosted runner, and the one variable `hivecommons` sets. - [Gate-integrity invariants for agent lanes](design/gate-integrity-invariants.md) — proposed write-gate rules for agent lanes: no history rewrites on branches a lane did not create, no sign-off on other authors' commits, and hold-gated demotion for gate manipulation. -- [Design documents](design/README.md) — longer-form design records with the full reasoning behind a decision, indexed with a status each (shipped / partly shipped / design only / historical) so a proposal is not mistaken for current behaviour: master secret rotation, wrapped master delivery to pull-only spokes, PR reach telemetry, and the knowledge system. +- [Design documents](design/README.md) — longer-form design records with the full reasoning behind a decision, indexed with a status each (shipped / partly shipped / design only / historical) so a proposal is not mistaken for current behaviour: master secret rotation, wrapped master delivery to push-reported spokes, PR reach telemetry, and the knowledge system. - [Discord reaction-consensus issue promotion](design/discord-issue-promotion.md) — proposed design for turning Discord reaction consensus into an audited Hive label write that composes with `project.issue_filter.require_labels` (RFC #6239). - [GitHub @-mention triggers](design/github-mention-triggers.md) — v6 design, now shipped on the `v6` branch in all three phases (#7582, #7597, #7623), for the first inbound GitHub trigger: a human summons an agent by mentioning the App on an issue or PR, mirroring the Linear agent-session path, replying through `Converse` on the existing write path, with poll-first transport and guards mapped to mechanisms that already exist (RFC #7483). Not present on `v4` or `v5`. - [Podman Compose-provider selection spike](podman-compose-provider-spike.md) — why `podman compose` must name its provider explicitly, and which provider needs no Docker tooling. diff --git a/src/docs/hub-deployment.md b/src/docs/hub-deployment.md index 3c47ee8c96..eeee93d8e3 100644 --- a/src/docs/hub-deployment.md +++ b/src/docs/hub-deployment.md @@ -97,7 +97,13 @@ The hub's SaaS provisioner creates or records: 3. `HIVE_ID`, `POD_NAMESPACE`, `HIVE_LEVEL`, `HIVE_HUB_URL`, `HIVE_HUB_SECRET`, optional `HIVE_AUTHORIZED_USERS`, and inference endpoint env vars in the spoke Deployment. 4. `/data/saas/hives//meta.json` on the hub PVC. Heartbeats can be accepted without this record, but hub-driven upgrades and dashboard links need it. -For heartbeat-only clusters where the hub cannot run `kubectl`, follow the manifest-level workflow in `manual-provisioning.md`; this guide intentionally keeps the hub setup consistent with that battle-tested path. +For push-reported clusters where the hub cannot run `kubectl`, follow the manifest-level workflow in `manual-provisioning.md`; this guide intentionally keeps the hub setup consistent with that battle-tested path. + +### Push-reported cluster entries + +`/data/saas/clusters.json` keeps the historical compatibility key `pull_only`. Set `pull_only: true` when the hub cannot reach that cluster's Kubernetes API and the spokes must dial out to the hub instead. Operator-facing surfaces call these clusters **push-reported** because their node stats, dashboard token proofs, and liveness arrive in spoke heartbeats pushed from the cluster. + +A push-reported entry still needs its stable identity (for example `id`, `name`, `domain`, app/forge fields), but it does not need a usable `kubeconfig_path` for hub-side reads. If node stats are absent, upgrade the spokes to a build that sends heartbeat cluster health and then check spoke node-metrics RBAC and metrics-server in that cluster. ## API surface to know diff --git a/src/pkg/hub/assets/dashboard.html b/src/pkg/hub/assets/dashboard.html index a9efe48167..582ba3c0ca 100644 --- a/src/pkg/hub/assets/dashboard.html +++ b/src/pkg/hub/assets/dashboard.html @@ -9294,7 +9294,7 @@

' + esc(c.error) + ''; - } else if (c.status === 'awaiting_heartbeat' && c.note) { + } else if ((c.status === 'awaiting_heartbeat' || c.status === 'missing_heartbeat_health') && c.note) { statusLine = '
' + esc(c.note) + '
'; } else if ((c.status === 'stale_heartbeat' || c.data_stale) && c.note) { statusLine = '
' + esc(c.note) + '
'; diff --git a/src/pkg/hub/cluster_health_fallback_coverage_test.go b/src/pkg/hub/cluster_health_fallback_coverage_test.go index a2c42ebb0c..f41506bc66 100644 --- a/src/pkg/hub/cluster_health_fallback_coverage_test.go +++ b/src/pkg/hub/cluster_health_fallback_coverage_test.go @@ -65,7 +65,45 @@ func TestBuildClusterHealthHeartbeatFallback(t *testing.T) { clusterHealthCacheMu.Unlock() } -func TestBuildClusterHealthPullOnlyAwaitingHeartbeat(t *testing.T) { +func TestBuildClusterHealthPushReportedAwaitingHeartbeat(t *testing.T) { + cleanup := helperSetupTempDirs(t) + defer cleanup() + + clusterHealthCacheMu.Lock() + clusterHealthCache = nil + clusterHealthCacheMu.Unlock() + + s := &HubServer{ + logger: slog.Default(), + clusters: map[string]ClusterConfig{ + "vllm-d": {ID: "vllm-d", Name: "vLLM-d", PullOnly: true}, + }, + heartbeatHealth: make(map[string]*HeartbeatHealthEntry), + } + + resp, err := buildClusterHealth(s) + if err != nil { + t.Fatalf("buildClusterHealth: %v", err) + } + if len(resp.Clusters) != 1 { + t.Fatalf("clusters = %+v, want one push-reported cluster", resp.Clusters) + } + got := resp.Clusters[0] + if got.Error != "" { + t.Fatalf("push-reported awaiting heartbeat rendered as error %q", got.Error) + } + if got.Status != perClusterHealthStatusAwaitingHeartbeat { + t.Fatalf("status = %q, want %q", got.Status, perClusterHealthStatusAwaitingHeartbeat) + } + if got.HiveCount != 0 || got.Summary.HiveCount != 0 { + t.Fatalf("hive counts = cluster %d summary %d, want 0", got.HiveCount, got.Summary.HiveCount) + } + if got.Note != "push-reported · awaiting spoke heartbeat" { + t.Fatalf("note = %q", got.Note) + } +} + +func TestBuildClusterHealthPushReportedOldSpokeExplainsMissingHealth(t *testing.T) { cleanup := helperSetupTempDirs(t) defer cleanup() @@ -84,7 +122,9 @@ func TestBuildClusterHealthPullOnlyAwaitingHeartbeat(t *testing.T) { Hives: []RegistryEntry{{ ID: "hosted-vllmd", ClusterID: "vllm-d", - LastHeartbeat: now.Add(-time.Minute).Format(time.RFC3339), + Version: "5.132.4-2-g80eec9a63", + GitHash: "80eec9a", + LastHeartbeat: now.Format(time.RFC3339), }}, }, } @@ -94,20 +134,55 @@ func TestBuildClusterHealthPullOnlyAwaitingHeartbeat(t *testing.T) { t.Fatalf("buildClusterHealth: %v", err) } if len(resp.Clusters) != 1 { - t.Fatalf("clusters = %+v, want one pull-only cluster", resp.Clusters) + t.Fatalf("clusters = %+v, want one push-reported cluster", resp.Clusters) } got := resp.Clusters[0] - if got.Error != "" { - t.Fatalf("pull-only awaiting heartbeat rendered as error %q", got.Error) + if got.Status != perClusterHealthStatusMissingHealth { + t.Fatalf("status = %q, want %q", got.Status, perClusterHealthStatusMissingHealth) } - if got.Status != perClusterHealthStatusAwaitingHeartbeat { - t.Fatalf("status = %q, want %q", got.Status, perClusterHealthStatusAwaitingHeartbeat) + want := "push-reported · heartbeat carries no node health — spoke build predates #10567; upgrade spokes to ≥ v5.133.0 (last seen just now)" + if got.Note != want { + t.Fatalf("note = %q, want %q", got.Note, want) + } +} + +func TestBuildClusterHealthPushReportedCurrentSpokeExplainsMissingHealth(t *testing.T) { + cleanup := helperSetupTempDirs(t) + defer cleanup() + + clusterHealthCacheMu.Lock() + clusterHealthCache = nil + clusterHealthCacheMu.Unlock() + + now := time.Now().UTC() + s := &HubServer{ + logger: slog.Default(), + clusters: map[string]ClusterConfig{ + "vllm-d": {ID: "vllm-d", Name: "vLLM-d", PullOnly: true}, + }, + heartbeatHealth: make(map[string]*HeartbeatHealthEntry), + registry: Registry{ + Hives: []RegistryEntry{{ + ID: "hosted-vllmd", + ClusterID: "vllm-d", + Version: clusterHealthUpgradeVersion, + GitHash: clusterHealthUpgradeFixSHA, + LastHeartbeat: now.Format(time.RFC3339), + }}, + }, } - if got.HiveCount != 1 || got.Summary.HiveCount != 1 { - t.Fatalf("hive counts = cluster %d summary %d, want 1", got.HiveCount, got.Summary.HiveCount) + + resp, err := buildClusterHealth(s) + if err != nil { + t.Fatalf("buildClusterHealth: %v", err) } - if got.Note == "" { - t.Fatal("awaiting heartbeat note should explain the neutral state") + if len(resp.Clusters) != 1 { + t.Fatalf("clusters = %+v, want one push-reported cluster", resp.Clusters) + } + got := resp.Clusters[0] + want := "push-reported · heartbeat carries no node health — check spoke node-metrics RBAC and metrics-server (last seen just now)" + if got.Note != want { + t.Fatalf("note = %q, want %q", got.Note, want) } } diff --git a/src/pkg/hub/clusters_registry.go b/src/pkg/hub/clusters_registry.go index b16ecb41e3..e07495bc6e 100644 --- a/src/pkg/hub/clusters_registry.go +++ b/src/pkg/hub/clusters_registry.go @@ -17,7 +17,7 @@ import ( // because losing an ack merely un-silences an alert — the safe direction. A // corrupt REGISTRY must not be discarded and replaced with a synthesized // default, because the default names only the hub-reachable cluster and would silently strand -// every hive on the two pull-only clusters, re-routing their App and host +// every hive on the two push-reported clusters, re-routing their App and host // resolution to the wrong cluster. The bad bytes are left on disk under this // suffix so an operator can see what arrived. const clustersQuarantineSuffix = ".corrupt" @@ -91,7 +91,7 @@ const ( // THIS IS THE CASE THAT MUST NOT YIELD AN EMPTY MAP, and must not fall back // to the default either. The hub knows a registry was placed here and // cannot tell what it says. Returning empty disables writes to the hub-reachable cluster; - // returning the default silently re-routes every pull-only hive. Both are + // returning the default silently re-routes every push-reported hive. Both are // widenings of the blast radius of a corrupt file, so the hub refuses. clustersUntrusted ) @@ -200,7 +200,7 @@ func validateClusterEntries(configs []ClusterConfig, logger *slog.Logger) map[st if !c.InCluster && c.KubeconfigPath == "" && !c.PullOnly { logger.Warn("skipping remote cluster with no kubeconfig_path", "cluster", c.ID, - "remedy", "set kubeconfig_path, or pull_only: true if the hub cannot reach this cluster and its spokes connect outbound over the heartbeat") + "remedy", "set kubeconfig_path, or pull_only: true if the hub cannot reach this cluster and its spokes push heartbeats outbound") continue } if c.Domain == "" { diff --git a/src/pkg/hub/contribute_ingress_reconcile.go b/src/pkg/hub/contribute_ingress_reconcile.go index 8c450d6653..d4482022bd 100644 --- a/src/pkg/hub/contribute_ingress_reconcile.go +++ b/src/pkg/hub/contribute_ingress_reconcile.go @@ -218,7 +218,7 @@ func (s *HubServer) reconcileContributeIngress() { skippedNoIngress++ continue } - // A pull-only cluster is reached only by answering its outbound + // A push-reported cluster is reached only by answering its outbound // heartbeat; the hub has no kubectl path into it. if !cluster.KubectlReachable() { skippedNoIngress++ diff --git a/src/pkg/hub/hosted_namespace_identity.go b/src/pkg/hub/hosted_namespace_identity.go index 6d442fb2e8..fcd62eb352 100644 --- a/src/pkg/hub/hosted_namespace_identity.go +++ b/src/pkg/hub/hosted_namespace_identity.go @@ -156,7 +156,7 @@ func stampHostedNamespaceIdentity(cluster *ClusterConfig, namespace, name, org, return } if cluster.PullOnly { - // A pull-only cluster's namespaces are not ours to label. Before + // A push-reported cluster's namespaces are not reachable for hub-side labeling. Before // clusters like this could be registered, clusterForHive fell back to // the DEFAULT cluster and this ran against the hub's own cluster, // stamping (or failing to find) a namespace that lives on another diff --git a/src/pkg/hub/hub_generations_store.go b/src/pkg/hub/hub_generations_store.go index 062bd15aa0..b2d5f15623 100644 --- a/src/pkg/hub/hub_generations_store.go +++ b/src/pkg/hub/hub_generations_store.go @@ -521,7 +521,7 @@ func (s *HubServer) rotateMasterSecret(now time.Time, force bool) (*generationSe // // F19 wires the derivation path to the live set, which means a rotation now // actually reaches the fleet. F21 measured that 44 of the 70 hosted spokes - // sit on pull_only clusters (the heartbeat-only cluster, a spoke cluster) that the reconcile sweep + // sit on push-reported clusters (configured with pull_only) that the reconcile sweep // structurally cannot read or patch — by design, and that is not changing // until the Option D wrapped-master delivery path lands. Those two facts // together are dangerous in a way neither is alone: before F19 a rotation @@ -533,7 +533,7 @@ func (s *HubServer) rotateMasterSecret(now time.Time, force bool) (*generationSe // // So: refuse to rotate unless the last sweep OBSERVED THE ENTIRE FLEET. // FleetFullyObserved is (ConsideredHives > 0 && UnreachableHives == 0), and - // a pull-only cluster increments UnreachableHives for every hive on it, so + // a push-reported cluster increments UnreachableHives for every hive on it, so // this is false on the production hub today and will stay false until every // spoke is reachable by whatever delivery mechanism ships. That is the // intended effect: the wiring lands now, and the ability to fire it unlocks diff --git a/src/pkg/hub/kubectl_ambient_escape_test.go b/src/pkg/hub/kubectl_ambient_escape_test.go index 32366694ef..48f9afd252 100644 --- a/src/pkg/hub/kubectl_ambient_escape_test.go +++ b/src/pkg/hub/kubectl_ambient_escape_test.go @@ -126,14 +126,14 @@ func TestRealInClusterStillUsesServiceAccount(t *testing.T) { } } -// TestUnreachableRemoteClusterUnchanged pins the pre-existing pull-only +// TestUnreachableRemoteClusterUnchanged pins the pre-existing push-reported // behaviour so this change is provably additive: a non-InCluster cluster the // hub cannot reach already aimed at the sentinel, and still must. func TestUnreachableRemoteClusterUnchanged(t *testing.T) { pullOnly := &ClusterConfig{ID: "vllm-d", PullOnly: true, KubeconfigPath: "/etc/hive/kubeconfigs/vllm-d"} argv := kubectlArgsForCluster(pullOnly, "get", "ns") if got := flagValue(argv, "--kubeconfig"); got != unreachableKubeconfigSentinel { - t.Errorf("pull-only cluster --kubeconfig = %q, want sentinel %q", got, unreachableKubeconfigSentinel) + t.Errorf("push-reported cluster --kubeconfig = %q, want sentinel %q", got, unreachableKubeconfigSentinel) } reachable := &ClusterConfig{ID: "a-ks-wec2", KubeconfigPath: "/etc/hive/kubeconfigs/a-ks-wec2", Context: "wec2"} diff --git a/src/pkg/hub/orphaned_pod_reaper.go b/src/pkg/hub/orphaned_pod_reaper.go index 686cb58441..073f0572ff 100644 --- a/src/pkg/hub/orphaned_pod_reaper.go +++ b/src/pkg/hub/orphaned_pod_reaper.go @@ -231,7 +231,7 @@ func (s *HubServer) reapOrphanedPods() { if cluster == nil { continue } - // A pull-only cluster is reached only by answering its outbound + // A push-reported cluster is reached only by answering its outbound // heartbeat; the hub has no kubectl path into it. Skip rather than // burn a timeout, and count it so the gap is visible. if !cluster.KubectlReachable() { diff --git a/src/pkg/hub/perhive_env_reconcile.go b/src/pkg/hub/perhive_env_reconcile.go index 8d15a60a93..934c12723a 100644 --- a/src/pkg/hub/perhive_env_reconcile.go +++ b/src/pkg/hub/perhive_env_reconcile.go @@ -557,7 +557,7 @@ type PerHiveEnvStatus struct { ConsideredHives int `json:"considered_hives"` SkippedByStatus int `json:"skipped_by_status"` // UnreachableHives is how many hives the LAST sweep admitted by status but - // then could not read at all, because their cluster is pull-only, has no + // then could not read at all, because their cluster is push-reported, has no // registry entry, or is inside the unreachable-cluster breaker's window. // // WHY THIS COUNTER EXISTS. Every counter above is sourced from successful @@ -565,7 +565,7 @@ type PerHiveEnvStatus struct { // recording anything. A hive on an unreachable cluster therefore left NO // trace on this surface: it was absent from ObservedHives, absent from // MissingPerHiveEnv, and absent from every generation bucket. With 44 of 66 - // hosted spokes on pull-only clusters, the surface reported the 22 + // hosted spokes on push-reported clusters, the surface reported the 22 // reachable ones as a fully converged fleet while two thirds of it was // never examined. "The hub cannot see this spoke" and "this spoke is fine" // rendered identically, which is the condition under which an operator diff --git a/src/pkg/hub/pool_replenisher.go b/src/pkg/hub/pool_replenisher.go index 5b69deac25..19668a5ae8 100644 --- a/src/pkg/hub/pool_replenisher.go +++ b/src/pkg/hub/pool_replenisher.go @@ -56,7 +56,7 @@ func (s *HubServer) replenishPools() { continue // opt-in per cluster } if !cluster.KubectlReachable() { - // A pull-only cluster's infra cannot be provisioned from the hub. + // A push-reported cluster's infra cannot be provisioned from the hub. s.logger.Warn("pool replenish skipped — cluster not kubectl-reachable", "cluster", id) continue } diff --git a/src/pkg/hub/saas_auth.go b/src/pkg/hub/saas_auth.go index 34543aeca3..5e5834ad72 100644 --- a/src/pkg/hub/saas_auth.go +++ b/src/pkg/hub/saas_auth.go @@ -150,7 +150,7 @@ func (s *HubServer) trustedSpokeSelfServiceUser(r *http.Request, hiveID string) case spokeProofOK: // verified — fall through to attribution below case spokeProofUnverifiable: - return "", "not authenticated — the hub has no stored dashboard-token record for this hive and could not read its hive-secrets/dashboard-token secret (the hive's cluster is unreachable from the hub, e.g. pull-only); a spoke on a current build reports its token over the authenticated heartbeat — trigger this hive's upgrade from the hub dashboard once, and the spoke's Upgrade button will verify against the stored record from then on" + return "", "not authenticated — the hub has no stored dashboard-token record for this hive and could not read its hive-secrets/dashboard-token secret (the hive's cluster is unreachable from the hub, e.g. push-reported); a spoke on a current build reports its token over the authenticated heartbeat — trigger this hive's upgrade from the hub dashboard once, and the spoke's Upgrade button will verify against the stored record from then on" default: // spokeProofMismatch return "", "not authenticated — spoke self-service proof rejected: the spoke's DASHBOARD_AUTH_TOKEN does not match this hive's dashboard-token secret; re-sync the spoke's token" } @@ -186,7 +186,7 @@ const ( // the presented proof matched none of them. spokeProofMismatch // spokeProofUnverifiable: the hub has NO reference to check against — no - // stored DashboardTokenHash record and no readable secret (pull-only or + // stored DashboardTokenHash record and no readable secret (push-reported or // otherwise unreachable cluster). spokeProofUnverifiable ) @@ -197,7 +197,7 @@ const ( // 1. The hub's OWN stored record (SaaSHive.DashboardTokenHash — written at // provisioning when the hub mints the token, refreshed from the spoke's // authenticated heartbeat). This needs no cluster access at all, which is -// the point: hosted spokes on pull-only clusters (e.g. fmaas) are reached +// the point: hosted spokes on push-reported clusters (e.g. fmaas) are reached // only by their outbound heartbeat, and requiring a live kubectl secret // read there made every proof unverifiable by design. // 2. A live read of the hive's hive-secrets/dashboard-token secret diff --git a/src/pkg/hub/saas_bulk.go b/src/pkg/hub/saas_bulk.go index 306805607b..64df0ab923 100644 --- a/src/pkg/hub/saas_bulk.go +++ b/src/pkg/hub/saas_bulk.go @@ -283,7 +283,7 @@ func (s *HubServer) applyBulkAction(action, branch, id, username string) BulkHiv return BulkHiveResult{HiveID: id, Ok: false, Error: "unknown bulk action"} } -// bulkRestartOrUpgrade restarts or upgrades one hive, PULL ONLY — the hub never +// bulkRestartOrUpgrade restarts or upgrades one hive, heartbeat-delivered — the hub never // kubectls into the spoke's cluster. // // An upgrade arms s.heartbeatUpgrade[id] with the branch's latest SHA and marks @@ -326,7 +326,7 @@ func (s *HubServer) bulkRestartOrUpgrade(h *SaaSHive, id, username, action strin } latestSHA := reach.SHA - // PULL ONLY — no kubectl push. An upgrade is delivered by arming + // Heartbeat-delivered — no hub kubectl push. An upgrade is delivered by arming // heartbeatUpgrade below; a plain restart is delivered by arming the // durable RequestedRestartAt, which the heartbeat path turns into // resp.RestartSpoke and the spoke applies to itself. Neither needs the hub @@ -414,7 +414,7 @@ func (s *HubServer) bulkSwitchBranch(h *SaaSHive, id, username, branch string) B imageTag := branchToTag(branch) + "-latest" image := bulkHiveImageRepo + ":" + imageTag - // PULL ONLY — no `kubectl set image` push. heartbeatSwitchTag armed below + // Heartbeat-delivered — no `kubectl set image` push. heartbeatSwitchTag armed below // is the delivery: the spoke holds the hive-self-upgrade RBAC to patch its // own Deployment and applies the new tag on its next check-in, which the // doc comment above already describes as the mechanism. Dropping the push diff --git a/src/pkg/hub/saas_cluster_health.go b/src/pkg/hub/saas_cluster_health.go index a951ac33a9..52aad30ed5 100644 --- a/src/pkg/hub/saas_cluster_health.go +++ b/src/pkg/hub/saas_cluster_health.go @@ -16,6 +16,13 @@ import ( const clusterHealthCacheTTL = 30 * time.Second +const ( + clusterHealthUpgradeVersion = "v5.133.0" + clusterHealthUpgradeFixPR = "#10567" + clusterHealthUpgradeFixSHA = "db4d5c268291100fe0e067791346e88dc17fa04f" + semverCoreParts = 3 +) + // CPU and memory bar thresholds are NOT declared here. The hub serves the // cluster-health panel raw percentages and the panel colours them with its own // CLUSTER_CPU_WARN_PCT / CLUSTER_CPU_DANGER_PCT / CLUSTER_MEM_* constants, so a @@ -135,7 +142,7 @@ type PerClusterHealth struct { DataAge string `json:"data_age,omitempty"` // human-readable age or collection timestamp // StuckPods reports hive-namespace pods stuck Terminating — the residue of // nodes disappearing without draining (#5328 item 3). Nil means the hub - // could not determine it (unreachable cluster, pull-only pool, failed + // could not determine it (unreachable cluster, push-reported pool, failed // listing); a non-nil report with Total 0 means it looked and the cluster // is clean. Those must not render alike: 27 orphans accumulated for three // weeks precisely because nothing distinguished "none" from "nobody @@ -144,7 +151,7 @@ type PerClusterHealth struct { // LeakedNamespaces reports hive-hosted-* namespaces the cluster holds that // this hub has no hive record for — provisioning namespaces that were // created and never torn down (#5768). Nil means the hub could not - // determine it (unreachable cluster, pull-only pool, failed listing, or an + // determine it (unreachable cluster, push-reported pool, failed listing, or an // empty hive registry, which cannot be told apart from an unreadable one); // a non-nil report with Total 0 means it looked and the cluster is clean. // Those must not render alike, for the same reason StuckPods above draws @@ -164,6 +171,7 @@ type PerClusterHealth struct { const ( perClusterHealthStatusAwaitingHeartbeat = "awaiting_heartbeat" + perClusterHealthStatusMissingHealth = "missing_heartbeat_health" perClusterHealthStatusStaleHeartbeat = "stale_heartbeat" clusterHealthHoursPerDay = 24 ) @@ -303,7 +311,7 @@ func buildClusterHealth(s *HubServer) (*ClusterHealthResponse, error) { if res.err != nil { isPullOnly := errors.Is(res.err, errClusterPullOnly) if isPullOnly { - s.logger.Info("cluster health: pull-only cluster, using the health its spokes report over the heartbeat", "cluster", cID) + s.logger.Info("cluster health: push-reported cluster, using the health its spokes report over the heartbeat", "cluster", cID) } else { s.logger.Warn("cluster health query failed", "cluster", cID, "error", res.err) } @@ -335,8 +343,8 @@ func buildClusterHealth(s *HubServer) (*ClusterHealthResponse, error) { Summary: ClusterHealthSummary{ HiveCount: hiveCounts[cID], }, - Status: perClusterHealthStatusAwaitingHeartbeat, - Note: s.pullOnlyAwaitingHeartbeatNote(cID), + Status: s.pushReportedNoHealthStatus(cID), + Note: s.pushReportedNoHealthNote(cID), }) continue } @@ -448,7 +456,7 @@ func clusterHealthQueryTimeoutFor(cluster *ClusterConfig) time.Duration { // errClusterPullOnly marks a health query skipped because the hub cannot reach // the cluster at all. It is an EXPECTED outcome, not a fault, so callers report // it as such and use the spokes' own heartbeat-reported health instead. -var errClusterPullOnly = errors.New("cluster is pull-only: not reachable from the hub") +var errClusterPullOnly = errors.New("cluster is push-reported: not reachable from the hub") // buildSingleClusterHealth queries a single cluster for node health data. // knownHostedNamespaces is the fleet-wide set of hosted namespace names the @@ -460,7 +468,7 @@ func buildSingleClusterHealth(cluster *ClusterConfig, hiveCount int, knownHosted // Node-level health comes from kubectl, which cannot run here. This is // not a new failure mode: the caller already falls back to the health // the spokes THEMSELVES report over the heartbeat, which is exactly the - // right source for a pull-only pool. Returning the sentinel routes into + // right source for a push-reported pool. Returning the sentinel routes into // that path and keeps it out of the "query failed" warning. return PerClusterHealth{}, fmt.Errorf("%w: %s", errClusterPullOnly, cluster.ID) } @@ -826,17 +834,42 @@ func (s *HubServer) getHeartbeatHealthForCluster(clusterID string) *HeartbeatHea return entry } -func (s *HubServer) pullOnlyAwaitingHeartbeatNote(clusterID string) string { - age := s.latestRegistryHeartbeatAgeForCluster(clusterID) - if age == "" { - return "pull-only · awaiting spoke heartbeat" +func (s *HubServer) pushReportedNoHealthStatus(clusterID string) string { + if diag := s.pushReportedHeartbeatDiagnostic(clusterID); diag.heartbeatSeen { + return perClusterHealthStatusMissingHealth + } + return perClusterHealthStatusAwaitingHeartbeat +} + +func (s *HubServer) pushReportedNoHealthNote(clusterID string) string { + diag := s.pushReportedHeartbeatDiagnostic(clusterID) + if !diag.heartbeatSeen { + return "push-reported · awaiting spoke heartbeat" } - return "pull-only · awaiting spoke heartbeat (last seen " + age + ")" + ageSuffix := "" + if diag.age != "" { + ageSuffix = " (last seen " + diag.age + ")" + } + if !diag.hasVersion { + return "push-reported · heartbeat carries no node health — spoke build/version not reported; upgrade spokes to ≥ " + clusterHealthUpgradeVersion + ageSuffix + } + if diag.predatesHealthFix { + return "push-reported · heartbeat carries no node health — spoke build predates " + clusterHealthUpgradeFixPR + "; upgrade spokes to ≥ " + clusterHealthUpgradeVersion + ageSuffix + } + return "push-reported · heartbeat carries no node health — check spoke node-metrics RBAC and metrics-server" + ageSuffix } -func (s *HubServer) latestRegistryHeartbeatAgeForCluster(clusterID string) string { +type pushReportedHeartbeatDiagnostic struct { + heartbeatSeen bool + age string + hasVersion bool + predatesHealthFix bool +} + +func (s *HubServer) pushReportedHeartbeatDiagnostic(clusterID string) pushReportedHeartbeatDiagnostic { + diag := pushReportedHeartbeatDiagnostic{predatesHealthFix: true} if s == nil { - return "" + return diag } var latest time.Time s.mu.RLock() @@ -844,19 +877,61 @@ func (s *HubServer) latestRegistryHeartbeatAgeForCluster(clusterID string) strin if h.ClusterID != clusterID || h.LastHeartbeat == "" { continue } - t, err := time.Parse(time.RFC3339, h.LastHeartbeat) + heartbeatAt, err := time.Parse(time.RFC3339, h.LastHeartbeat) if err != nil { continue } - if latest.IsZero() || t.After(latest) { - latest = t + diag.heartbeatSeen = true + if latest.IsZero() || heartbeatAt.After(latest) { + latest = heartbeatAt + } + if h.Version == "" && h.GitHash == "" { + continue + } + diag.hasVersion = true + if spokeBuildAtOrAfterClusterHealthFix(h.Version, h.GitHash) { + diag.predatesHealthFix = false } } s.mu.RUnlock() - if latest.IsZero() { - return "" + if !latest.IsZero() { + diag.age = formatClusterHealthAge(time.Since(latest)) + } + return diag +} + +func spokeBuildAtOrAfterClusterHealthFix(version, gitHash string) bool { + if sameCommit(gitHash, clusterHealthUpgradeFixSHA) { + return true + } + major, minor, patch, ok := parseClusterHealthVersion(version) + if !ok { + return false + } + fixMajor, fixMinor, fixPatch, _ := parseClusterHealthVersion(clusterHealthUpgradeVersion) + if major != fixMajor { + return major > fixMajor + } + if minor != fixMinor { + return minor > fixMinor + } + return patch >= fixPatch +} + +func parseClusterHealthVersion(version string) (int, int, int, bool) { + version = strings.TrimPrefix(strings.TrimSpace(version), "v") + base, _, _ := strings.Cut(version, "-") + parts := strings.Split(base, ".") + if len(parts) < semverCoreParts { + return 0, 0, 0, false + } + major, errMajor := strconv.Atoi(parts[0]) + minor, errMinor := strconv.Atoi(parts[1]) + patch, errPatch := strconv.Atoi(parts[2]) + if errMajor != nil || errMinor != nil || errPatch != nil { + return 0, 0, 0, false } - return formatClusterHealthAge(time.Since(latest)) + return major, minor, patch, true } func formatClusterHealthAge(age time.Duration) string { diff --git a/src/pkg/hub/saas_provision.go b/src/pkg/hub/saas_provision.go index 5551d20722..62ea3d80b4 100644 --- a/src/pkg/hub/saas_provision.go +++ b/src/pkg/hub/saas_provision.go @@ -257,11 +257,13 @@ type ClusterConfig struct { // Zero means "this cluster has no hub-managed App identity" — the hub then // changes nothing about its spokes' credentials. GitHubAppID int64 `json:"github_app_id,omitempty" yaml:"github_app_id,omitempty"` - // PullOnly marks a cluster the hub CANNOT reach: its spokes connect - // outbound over the heartbeat and nothing here can kubectl into them. + // PullOnly marks a push-reported cluster the hub CANNOT reach: its + // spokes connect outbound over the heartbeat and nothing here can kubectl + // into them. The yaml/json key remains pull_only for compatibility, but + // operator-facing text should describe these clusters as push-reported. // // WHY THIS FIELD EXISTS. Registration used to require a kubeconfig_path, - // so a pull-only pool could not be registered AT ALL — the loader dropped + // so a push-reported pool could not be registered AT ALL — the loader dropped // the entry ("skipping remote cluster with no kubeconfig_path"). Everything // keyed off the cluster registry then silently misfired for its hives: // appIdentityForHive returned nil, so spokes sat on the placeholder app_id @@ -416,15 +418,15 @@ func kubectlForClusterContext(ctx context.Context, cluster *ClusterConfig, args // // It must never exist. An empty --kubeconfig makes kubectl fall back to its // AMBIENT configuration, which inside the hub pod is the hub's OWN cluster — so -// a command meant for a pull-only pool would quietly execute against the hub's +// a command meant for a push-reported pool would quietly execute against the hub's // cluster and report success for work that never happened there. Naming a path // that cannot resolve turns that silent misfire into a loud failure. -const unreachableKubeconfigSentinel = "/nonexistent/pull-only-cluster-is-not-reachable-from-the-hub" +const unreachableKubeconfigSentinel = "/nonexistent/push-reported-cluster-is-not-reachable-from-the-hub" // KubectlReachable reports whether the hub can run kubectl against this cluster. // // Callers that are about to touch a cluster with kubectl must check this and -// skip with a clear message: a pull-only cluster's spokes are reached ONLY by +// skip with a clear message: a push-reported cluster's spokes are reached ONLY by // answering their outbound heartbeat. func (c *ClusterConfig) KubectlReachable() bool { if c == nil { @@ -516,7 +518,7 @@ func loadClusters(logger *slog.Logger) map[string]ClusterConfig { // WHY FATAL RATHER THAN A DEGRADED START. NewHubServer returns no error, // so the only alternatives available here are the two this fix exists to // remove: come up with an empty registry (writes to the hub-reachable cluster silently - // off) or come up on the synthesized default (every pull-only hive + // off) or come up on the synthesized default (every push-reported hive // silently re-routed to the hub's own cluster). Both present as a // healthy hub. A hub that refuses to start is loud, is caught by the // pod's restart loop and readiness gate, and — critically — is @@ -559,7 +561,7 @@ type SaaSHive struct { // heartbeat (dashboard_token_hash) and from any successful live secret // read. Spoke-relayed upgrade requests verify their X-Hive-Proxy-Auth proof // against this first, so verification works for hives on clusters the hub - // cannot kubectl into (pull-only / unreachable) — the live secret read is + // cannot kubectl into (push-reported / unreachable) — the live secret read is // only a fallback. A hash, never the raw token: the hub must be able to // VERIFY the spoke's credential without holding a copy it could leak. DashboardTokenHash string `json:"dashboard_token_hash,omitempty"` @@ -1934,7 +1936,7 @@ func (s *HubServer) makeVanityHostServable(hiveID, vanityHost string, cluster *C // caller's "vanity host is not served" reporting truthful instead of // letting kubectl aim at the hub's own cluster and report a missing // ingress for a namespace that was never there. - return fmt.Errorf("cluster %s is pull-only: the hub cannot add %s to the spoke's ingress", cluster.ID, vanityHost) + return fmt.Errorf("cluster %s is push-reported: the hub cannot add %s to the spoke's ingress", cluster.ID, vanityHost) } if s.vanityHostServable != nil { return s.vanityHostServable(hiveID, vanityHost, cluster) @@ -2483,7 +2485,7 @@ func provisionHive(h *SaaSHive, req *CreateHiveRequest, cluster *ClusterConfig, // its hash can be recorded on the hive record below once the manifest is // applied: that stored record is what lets the hub verify spoke-relayed // upgrade proofs WITHOUT a live secret read, which is impossible on - // pull-only clusters. See SaaSHive.DashboardTokenHash. + // push-reported clusters. See SaaSHive.DashboardTokenHash. dashboardToken := func() string { b := make([]byte, dashboardTokenBytes) if _, err := cryptoRand.Read(b); err != nil { @@ -3049,7 +3051,7 @@ subjects: # domain, and a guaranteed 503 anywhere else, because that wildcard sends the # name to the HUB's router which has no backend for it. # -# The spoke is the only party that CAN read this on a pull-only cluster, where +# The spoke is the only party that CAN read this on a push-reported cluster, where # the hub has no kubectl path by design. Strictly read-only and namespace-scoped: # list/get, no write verbs, so a compromised spoke learns only its own hostname # — which it already advertises — and can neither create nor retarget routing. diff --git a/src/pkg/hub/saas_upgrade.go b/src/pkg/hub/saas_upgrade.go index 5214d5c571..ac489962a9 100644 --- a/src/pkg/hub/saas_upgrade.go +++ b/src/pkg/hub/saas_upgrade.go @@ -456,7 +456,7 @@ func (s *HubServer) handleUpgradeHive(w http.ResponseWriter, r *http.Request) { return } - // PULL ONLY — no kubectl push. The manual Upgrade button now does exactly + // Heartbeat-delivered — no kubectl push. The manual Upgrade button now does exactly // what auto-upgrade does: record the target and arm the heartbeat. The // spoke collects it on its next beat and patches its own Deployment. // @@ -940,13 +940,13 @@ func (s *HubServer) clusterRecentlyUnreachable(clusterID string) bool { if clusterID == "" { return false } - // A pull-only cluster is unreachable BY DECLARATION and permanently, so it + // A push-reported cluster is unreachable BY DECLARATION and permanently, so it // never has to be learned the expensive way. This breaker exists because // discovering unreachability costs a full dial timeout per hive per cycle — // ~90s a call, with a pool of hives serialising into tens of minutes of // blocking. Saying so in clusters.json means that price is never paid even // once, and every caller that already consults this breaker is covered - // without needing its own pull-only check. + // without needing its own push-reported check. if c, ok := s.clusters[clusterID]; ok && c.PullOnly { return true } @@ -1205,7 +1205,7 @@ func (s *HubServer) triggerAutoUpgrades() { } s.heartbeatUpgrade[h.ID] = recoverTarget s.mu.Unlock() - // PULL ONLY — the heartbeat armed above IS the delivery, as + // Heartbeat-delivered — the heartbeat armed above IS the delivery, as // the previous comment here already conceded ("the heartbeat // fallback armed above is what actually delivers the // upgrade"). The kubectl push that followed it was pure @@ -1377,9 +1377,9 @@ func (s *HubServer) triggerAutoUpgrades() { // pullonly_upgrade.go for the full measured loop. // // This is deliberately NOT gated on cluster reachability. The hub's - // kubectl path is only a fast-path optimisation, so a pull-only cluster + // kubectl path is only a fast-path optimisation, so a push-reported cluster // is irrelevant here; gating on it would silently disable auto-upgrade - // for the 40+ pull-only spokes that heartbeat perfectly well. + // for the 40+ push-reported spokes that heartbeat perfectly well. // // Refused LOUDLY, never silently: a hive with auto_upgrade=true that // simply never upgrades is indistinguishable from one already at latest, @@ -1443,7 +1443,7 @@ func (s *HubServer) triggerAutoUpgrades() { } } s.mu.Unlock() - // PULL ONLY — no kubectl push. Arming the heartbeat is the delivery: + // Heartbeat-delivered — no kubectl push. Arming the heartbeat is the delivery: // the spoke reads UpgradeTo off its next heartbeat response and patches // its own Deployment with its own ServiceAccount (cmd/hive/main.go → // self_upgrade.go). The former `rolloutRestartHive` call here was only a diff --git a/src/pkg/hub/server.go b/src/pkg/hub/server.go index 29c7eda153..714a47e30c 100644 --- a/src/pkg/hub/server.go +++ b/src/pkg/hub/server.go @@ -1204,7 +1204,7 @@ type HubServer struct { perHiveEnvSkippedByStatus int // perHiveEnvUnreachable / perHiveEnvUnreachableClusters record the LAST // sweep's hives that were admitted by status but could not be READ at all — - // pull-only cluster, no registry entry, or inside the unreachable-cluster + // push-reported cluster, no registry entry, or inside the unreachable-cluster // breaker window. Without these the sweep's skip paths left no trace, so an // unreachable spoke was indistinguishable from one that does not exist and // the surface reported a partially-observed fleet as converged. Guarded by diff --git a/src/pkg/hub/spoke/heartbeat.go b/src/pkg/hub/spoke/heartbeat.go index 8d84dc2f7f..a91202ecea 100644 --- a/src/pkg/hub/spoke/heartbeat.go +++ b/src/pkg/hub/spoke/heartbeat.go @@ -652,7 +652,7 @@ type HeartbeatPayload struct { // (SaaSHive.DashboardTokenHash) of the credential spoke-relayed upgrade // requests prove themselves with — without the hub ever needing to read // the hive-secrets secret from the spoke's cluster (impossible on - // pull-only clusters) and without the raw token ever riding the wire. The + // push-reported clusters) and without the raw token ever riding the wire. The // beat itself is authenticated by the per-hive bearer, so only the hive // can set its own record — which is sound: the record only lets that same // hive's dashboard prove it is itself. omitempty: an old spoke, or one @@ -1042,7 +1042,7 @@ type HeartbeatPayload struct { // ComponentReach carries the spoke's in-process component reach counters // (#3993, phase 2a of #3973): per (component, running commit) span counts // that increment REGARDLESS of whether an OTel exporter is configured, so - // every spoke reports — including the pull-only fleet a span backend can + // every spoke reports — including the push-reported fleet a span backend can // never see (design D1/D2). The hub stores the latest report per hive, // storage only: nothing joins, maps, or renders it until #3994. Spoke-side // the entry count is capped at tracing.MaxReachComponents and the hub @@ -1961,7 +1961,7 @@ func ingressHostExists(ctx context.Context, client *http.Client, cfg *inClusterC // for that name and answers 503. The hub then linked users at a hostname that // could never work, while the spoke's real Route was serving fine all along. // -// The spoke is the only party that can answer this on a pull-only cluster: the +// The spoke is the only party that can answer this on a push-reported cluster: the // hub has no kubectl path there by design, so it cannot read the Route. The // spoke reads its own namespace, which it can always do. //