Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions changelog.d/fixed-10607.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
- hub: call push-reporting clusters push-reported, and say why vllm-d has no node health (#10607)
1 change: 1 addition & 0 deletions src/changelog.d/changed-push-reported-clusters-10601.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
- Changed hub cluster-health wording to call heartbeat-only clusters push-reported and explain missing node health from old spokes (#10601)
6 changes: 3 additions & 3 deletions src/cmd/hive/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -1393,7 +1393,7 @@ func (b *boot) wireBootClosures() {
}
// Prefer the host our OWN Route/Ingress actually serves. The synthesised
// "<hiveID>.<hub host>" below is only correct when this spoke is fronted by
// the hub's wildcard domain; pull-only clusters must report their live host.
// the hub's wildcard domain; push-reported clusters must report their live host.
if host := spoke.SpokeServedHost(b.ctx); host != "" {
return "https://" + host
}
Expand Down Expand Up @@ -4444,7 +4444,7 @@ func (b *boot) bootHeartbeatWith(deps bootHeartbeatDeps) {
// Hash only, never the raw token: lets the hub verify this
// spoke's upgrade-proof credential without reading the
// hive-secrets secret from a cluster it may not reach
// (pull-only). Empty when no token is configured.
// (push-reported). Empty when no token is configured.
DashboardTokenHash: func() string {
if b.cfg.Dashboard.AuthToken == "" {
return ""
Expand Down Expand Up @@ -4636,7 +4636,7 @@ func (b *boot) bootHeartbeatWith(deps bootHeartbeatDeps) {
// Component reach counters (#3993, phase 2a of #3973): per
// (component, running commit) span counts aggregated in-process,
// exporter or not (D2) — the heartbeat is the only channel that
// reaches every spoke, pull-only ones included (D1). nil until
// reaches every spoke, push-reported ones included (D1). nil until
// the first span, which the hub reads as "no data", never as
// zero reach. Capped at tracing.MaxReachComponents entries.
ComponentReach: tracing.ReachSnapshot(),
Expand Down
6 changes: 3 additions & 3 deletions src/docs/design/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -72,9 +72,9 @@ that status is the thing to check before treating a page as current behaviour:
generations code and the session-cookie domain have landed
(`src/pkg/hub/hub_generations*.go`, `hub_cookie_generations.go`); the
remaining per-domain adoptions listed under "Follow-on PRs" are still pending.
- [Wrapped master delivery to pull-only spokes](master-delivery-wrapped.md) —
- [Wrapped master delivery to push-reported spokes](master-delivery-wrapped.md) —
**design only**. Read it after master-key-rotation.md: rotation is complete on
the hub and delivers nothing to the two thirds of the fleet on pull-only
the hub and delivers nothing to the two thirds of the fleet on push-reported
clusters, which the hub cannot write to by design. Records Option D — the spoke
generates a keypair, publishes the public half over its own outbound heartbeat,
and the hub seals each new master to it — and why Options A and B were
Expand Down Expand Up @@ -165,7 +165,7 @@ that status is the thing to check before treating a page as current behaviour:
The first inbound GitHub trigger: a human summons an agent by mentioning the
App on an issue or PR, mirroring the Linear agent-session path
(`pkg/linearagent`) that already exists. Poll-first transport so it works on
pull-only spokes, a mention grammar with `ask <agent>` routing under the
push-reported spokes, a mention grammar with `ask <agent>` routing under the
`linear.session_agent` resolution rule, a kick built like the Linear
responder's, replies through `Converse` on the existing watcher write path
with no new outbound surface, and seven guards each mapped to a mechanism
Expand Down
6 changes: 3 additions & 3 deletions src/docs/design/escalation-surfaces.md
Original file line number Diff line number Diff line change
Expand Up @@ -75,7 +75,7 @@ Severity is the routing key, set by the producer:
### Outbound (phase 1)

Plain SMTP submission — `net/smtp` with STARTTLS/implicit-TLS, no third-party
mail dependency. Works from pull-only clusters (outbound 465/587 only).
mail dependency. Works from push-reported clusters (outbound 465/587 only).

```yaml
email:
Expand Down Expand Up @@ -106,7 +106,7 @@ email:
### Inbound reply-to-act (phase 2, explicitly later)

Mirrors the Linear inbound model (`pkg/linearagent`: guards → ack → kick,
decoupled): an IMAP poller (outbound-only, fits pull-only clusters) watches a
decoupled): an IMAP poller (outbound-only, fits push-reported clusters) watches a
dedicated mailbox; a reply to an escalation mail can carry exactly the verbs
the mail offered (`approve`, `deny`, `kick <agent>: <prompt>`).

Expand Down Expand Up @@ -185,7 +185,7 @@ sinks are tested against local fakes (`net/smtp` test server, httptest).
## What this deliberately does not do

- No webhook *receiver* for provider callbacks (PagerDuty ack-back etc.) —
outbound-only keeps the pull-only story intact.
outbound-only keeps the push-reported story intact.
- No templating language for mail bodies; producers compose, full stop.
- No SMS provider (Twilio) — Pushover/ntfy cover the phone story without
per-message billing; revisit only on demand.
Expand Down
4 changes: 2 additions & 2 deletions src/docs/design/github-mention-triggers.md
Original file line number Diff line number Diff line change
Expand Up @@ -99,7 +99,7 @@ the same PR — never as a config key first.

The Linear path is webhook-only because Linear pushes and demands a 5-second
answer. GitHub is different in a way that matters for this fleet: most spokes
run on pull-only clusters the hub cannot write to (see
run on push-reported clusters the hub cannot write to (see
[wrapped master delivery](master-delivery-wrapped.md)), and such a spoke has no
public URL for GitHub to deliver to either. A webhook-only design would work
on the hosted hub and nowhere else.
Expand Down Expand Up @@ -300,7 +300,7 @@ agents:
mention-kicked agent without `Converse` cannot answer.
3. **Webhook accelerator.** The hub's verifier gains the three event types and
a spoke-side mount; deliveries advance the poller's watermark. Hub-to-spoke
relay for pull-only clusters is a later question that the master-delivery
relay for push-reported clusters is a later question that the master-delivery
design already frames.

## Non-goals
Expand Down
30 changes: 15 additions & 15 deletions src/docs/design/master-delivery-wrapped.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Wrapped master delivery to pull-only spokes
# Wrapped master delivery to push-reported spokes

Status: DESIGN ONLY. No implementation. This document extends
`master-key-rotation.md` and should be read after it.
Expand Down Expand Up @@ -31,7 +31,7 @@ regenerates the hub's *private* seed via `SSOSigningSeedFromMaster`

The reconcile lane is therefore an optimisation, not a dependency. It
pre-computes what the spoke would compute anyway. That is why the fleet ran
normally while the lane was dead, and why the 44 pull-only spokes work today.
normally while the lane was dead, and why the 44 push-reported spokes work today.

**The master is the one value a spoke cannot self-derive.** Delivering a *new*
master after rotation is the lane's only irreplaceable job, and on 44 of 66
Expand All @@ -41,7 +41,7 @@ spokes there is no delivery path.
returns false unconditionally. Per the operator correction of 2026-08-14,
`pull_only` is an **intentional architectural boundary on both clusters** — the
hub is not meant to hold write credentials into the heartbeat-only clusters. The
field's own doc comment already says this plainly: a pull-only cluster's spokes
field's own doc comment already says this plainly: a push-reported cluster's spokes
"connect outbound over the heartbeat and nothing here can kubectl into them"
(`src/pkg/hub/saas_provision.go:255-256`).

Expand Down Expand Up @@ -308,7 +308,7 @@ heartbeat. For any hive that is not, there is no race at all — **the attacker
pins by default, unopposed, with no timing requirement whatsoever.** The
categories are not hypothetical:

- **Not yet rolled to the new image.** Migration step 1 reaches pull-only
- **Not yet rolled to the new image.** Migration step 1 reaches push-reported
spokes via self-upgrade, which is not instantaneous or simultaneous. Between
the hub gaining the pin store and the last spoke rolling, every unrolled hive
is unopposed.
Expand Down Expand Up @@ -343,7 +343,7 @@ the estimate of how much of the fleet the rule protects on day one.
**This is unavoidable given the constraints, and it should be stated as a
residual rather than engineered around.** The hub cannot reach in to seed a
key. There is no out-of-band channel from the hub. The only pre-existing shared
secret with a pull-only spoke *is* the master. Bootstrapping trust from
secret with a push-reported spoke *is* the master. Bootstrapping trust from
something that is not the master requires something the spoke or the
provisioning process supplies — which is §5.

Expand All @@ -358,7 +358,7 @@ This is the only candidate for a non-master anchor, since the hub cannot reach
in and there is no out-of-band hub channel.

Provisioning is the moment a spoke is created, and the hub controls the
manifest even for pull-only clusters — the manifest is applied by an operator
manifest even for push-reported clusters — the manifest is applied by an operator
context, not by the hub. The template already injects per-hive secret material
(`src/pkg/hub/saas_provision.go:2567-2589`: `HeartbeatKey`, `SessionPublicKey`, `SSOPublicKey`,
`TerminalKey`, `InviteKey`), and the `/secrets` read-only projected mount
Expand Down Expand Up @@ -418,13 +418,13 @@ initiate a connection to a spoke.**
| 6 | Enrolment value added to provisioning template for new hives (§5) | operator applies manifest | new hives only | **no** |

Step 1 is the only one needing a spoke image roll, and it needs **no hub
action at all** — which is precisely why it works for pull-only clusters. The
action at all** — which is precisely why it works for push-reported clusters. The
spoke's image is updated by its existing self-upgrade path
(`HeartbeatResponse.SwitchToTag`, `src/pkg/hub/spoke/heartbeat.go:2133-2137` (field at `:1698`)), whose doc comment
records that this exists for exactly this reason: "Used for branch switches on
clusters the hub can't reach over kubectl — the spoke has in-cluster RBAC
(hive-self-upgrade role) to patch its own deployment." The delivery mechanism
for the fix is one the pull-only boundary already accommodates.
for the fix is one the push-reported boundary already accommodates.

Step 4's carrier has direct precedent. `HeartbeatResponse.PendingGateway`
(`src/pkg/hub/spoke/heartbeat.go:2167-2172`) is a **secret** delivered on the heartbeat response,
Expand Down Expand Up @@ -459,7 +459,7 @@ SafeToRetirePrevious = hasPrevious
```

One critical adjustment. `FleetFullyObserved` is defined in terms of
*Deployment-read* observation, and pull-only spokes are unreachable by
*Deployment-read* observation, and push-reported spokes are unreachable by
definition — so under Option D the 44 must become observable by a different
route: **their heartbeat publication is the observation.** A hive that is
publishing a pinned wrapping key and receiving wrapped masters is converged,
Expand All @@ -475,7 +475,7 @@ here.

The resolution is that the **denominator must stay registry-sourced** while
only the **numerator** becomes heartbeat-sourced. A paused or vanished
pull-only spoke then keeps its place in the denominator, never publishes, and
push-reported spoke then keeps its place in the denominator, never publishes, and
correctly blocks retirement. What must never be built is a denominator of
"hives that heartbeated recently". → this is the single most likely place for
a test to pass for the wrong reason; see §10.
Expand Down Expand Up @@ -607,15 +607,15 @@ anticipates:
If `HIVE_HUB_SECRET` is stripped and `HIVE_HEARTBEAT_KEY` is present, the spoke
authenticates on lane 1 (`src/pkg/hub/hub_keys.go:456-458`) and never needs the master —
fine. But after a rotation, `HIVE_HEARTBEAT_KEY` is stale. On a reachable spoke
the reconcile lane patches it. **On a pull-only spoke nothing patches it**, and
the reconcile lane patches it. **On a push-reported spoke nothing patches it**, and
the spoke cannot self-derive a fresh one without the master. So it must
authenticate with the *old* bearer, which works only while the previous
generation is still acceptable (`acceptableGenerations`,
`src/pkg/hub/hub_generations.go:222-242`, default 7 days).

This yields a workable but strictly bounded property:

- **A pull-only spoke can have `HIVE_HUB_SECRET` stripped** once it holds a
- **A push-reported spoke can have `HIVE_HUB_SECRET` stripped** once it holds a
pinned wrapping key and a valid `HIVE_HEARTBEAT_KEY`.
- **It must then receive and apply each wrapped master within
`defaultVerifyWindow`** (7 days, `src/pkg/hub/hub_generations.go:118`), because after that
Expand All @@ -624,8 +624,8 @@ This yields a workable but strictly bounded property:
verify window is permanently stranded** and requires operator intervention on
its own cluster.

That last bullet is the cost of closing RESIDUAL-1 on pull-only spokes, and it
is not small. Today a stranded pull-only spoke self-heals the moment it comes
That last bullet is the cost of closing RESIDUAL-1 on push-reported spokes, and it
is not small. Today a stranded push-reported spoke self-heals the moment it comes
back, because it holds the master and re-derives everything. After stripping,
it does not.

Expand Down Expand Up @@ -994,7 +994,7 @@ access (§6). Steps 1–5 need nothing from them.

1. F20 (fail closed on unreadable generations file) — independent, small.
2. Spoke keypair generation + publication (§3, migration step 1). Ships in the
spoke image, needs no hub action, reaches pull-only clusters via
spoke image, needs no hub action, reaches push-reported clusters via
self-upgrade.
3. Hub-side pin store + coverage counter folded into `FleetFullyObserved` (§6).
4. F19 (wire `provisionGenerationSet()` to the live set).
Expand Down
2 changes: 1 addition & 1 deletion src/docs/design/master-key-rotation.md
Original file line number Diff line number Diff line change
Expand Up @@ -549,7 +549,7 @@ generation, and strand them when retirement fires **unconditionally** at
into a 7-day-delayed outage".

So `rotateMasterSecret` now refuses unless `FleetFullyObserved` — the last sweep
saw every admitted hive with none unreachable. A pull-only cluster increments
saw every admitted hive with none unreachable. A push-reported cluster increments
`UnreachableHives` for every hive on it, so this is **false on the production hub
today and stays false** until every spoke is reachable. The wiring lands now; the
ability to fire it unlocks only when the thing that makes it safe exists.
Expand Down
2 changes: 1 addition & 1 deletion src/docs/design/pr-reach-telemetry.md
Original file line number Diff line number Diff line change
Expand Up @@ -71,7 +71,7 @@ diff's lines run." Line/function-level attribution is out of scope for phase 1
The four design decisions, accepted in #3973:

- **D1 — Aggregation is spoke-side and rides the heartbeat.** A span-backend
query can never see the pull-only spokes, and the `otel:` exporter is opt-in
query can never see the push-reported spokes, and the `otel:` exporter is opt-in
— most spokes export nowhere. A hub-side span query would report ~0 reach
for most of the fleet and call it "unused": the anchoring-rule trap, one
level up. The heartbeat is the outbound channel every spoke already opens.
Expand Down
2 changes: 1 addition & 1 deletion src/docs/documentation-map.md
Original file line number Diff line number Diff line change
Expand Up @@ -164,7 +164,7 @@ Documentation for the current Hive line (branch `v5`; the code and docs live und
- [Podman CI runner map](podman-ci-runner-map.md) — measured hosted-runner capabilities and which Podman lane goes where; SELinux is the only lane needing non-hosted infrastructure.
- [CI runner labels](ci-runner-labels.md) — how `runs-on:` picks the self-hosted fleet, why a fork must degrade to a GitHub-hosted runner, and the one variable `hivecommons` sets.
- [Gate-integrity invariants for agent lanes](design/gate-integrity-invariants.md) — proposed write-gate rules for agent lanes: no history rewrites on branches a lane did not create, no sign-off on other authors' commits, and hold-gated demotion for gate manipulation.
- [Design documents](design/README.md) — longer-form design records with the full reasoning behind a decision, indexed with a status each (shipped / partly shipped / design only / historical) so a proposal is not mistaken for current behaviour: master secret rotation, wrapped master delivery to pull-only spokes, PR reach telemetry, and the knowledge system.
- [Design documents](design/README.md) — longer-form design records with the full reasoning behind a decision, indexed with a status each (shipped / partly shipped / design only / historical) so a proposal is not mistaken for current behaviour: master secret rotation, wrapped master delivery to push-reported spokes, PR reach telemetry, and the knowledge system.
- [Discord reaction-consensus issue promotion](design/discord-issue-promotion.md) — proposed design for turning Discord reaction consensus into an audited Hive label write that composes with `project.issue_filter.require_labels` (RFC #6239).
- [GitHub @-mention triggers](design/github-mention-triggers.md) — v6 design, now shipped on the `v6` branch in all three phases (#7582, #7597, #7623), for the first inbound GitHub trigger: a human summons an agent by mentioning the App on an issue or PR, mirroring the Linear agent-session path, replying through `Converse` on the existing write path, with poll-first transport and guards mapped to mechanisms that already exist (RFC #7483). Not present on `v4` or `v5`.
- [Podman Compose-provider selection spike](podman-compose-provider-spike.md) — why `podman compose` must name its provider explicitly, and which provider needs no Docker tooling.
Expand Down
8 changes: 7 additions & 1 deletion src/docs/hub-deployment.md
Original file line number Diff line number Diff line change
Expand Up @@ -97,7 +97,13 @@ The hub's SaaS provisioner creates or records:
3. `HIVE_ID`, `POD_NAMESPACE`, `HIVE_LEVEL`, `HIVE_HUB_URL`, `HIVE_HUB_SECRET`, optional `HIVE_AUTHORIZED_USERS`, and inference endpoint env vars in the spoke Deployment.
4. `/data/saas/hives/<id>/meta.json` on the hub PVC. Heartbeats can be accepted without this record, but hub-driven upgrades and dashboard links need it.

For heartbeat-only clusters where the hub cannot run `kubectl`, follow the manifest-level workflow in `manual-provisioning.md`; this guide intentionally keeps the hub setup consistent with that battle-tested path.
For push-reported clusters where the hub cannot run `kubectl`, follow the manifest-level workflow in `manual-provisioning.md`; this guide intentionally keeps the hub setup consistent with that battle-tested path.

### Push-reported cluster entries

`/data/saas/clusters.json` keeps the historical compatibility key `pull_only`. Set `pull_only: true` when the hub cannot reach that cluster's Kubernetes API and the spokes must dial out to the hub instead. Operator-facing surfaces call these clusters **push-reported** because their node stats, dashboard token proofs, and liveness arrive in spoke heartbeats pushed from the cluster.

A push-reported entry still needs its stable identity (for example `id`, `name`, `domain`, app/forge fields), but it does not need a usable `kubeconfig_path` for hub-side reads. If node stats are absent, upgrade the spokes to a build that sends heartbeat cluster health and then check spoke node-metrics RBAC and metrics-server in that cluster.

## API surface to know

Expand Down
2 changes: 1 addition & 1 deletion src/pkg/hub/assets/dashboard.html
Original file line number Diff line number Diff line change
Expand Up @@ -9294,7 +9294,7 @@ <h2 style="font-size:1.3rem;color:var(--hub-dashboard-accent);margin:var(--sp-0)
var statusLine = '';
if (c.error) {
statusLine = '<div class="cluster-health-message error">' + esc(c.error) + '</div>';
} else if (c.status === 'awaiting_heartbeat' && c.note) {
} else if ((c.status === 'awaiting_heartbeat' || c.status === 'missing_heartbeat_health') && c.note) {
statusLine = '<div class="cluster-health-message info">' + esc(c.note) + '</div>';
} else if ((c.status === 'stale_heartbeat' || c.data_stale) && c.note) {
statusLine = '<div class="cluster-health-message warn">' + esc(c.note) + '</div>';
Expand Down
Loading
Loading