Skip to content

πŸ› fix: hub-signed relay lets hosted spokes verify relayed GitHub webhooks - #17914

Merged
clubanderson merged 7 commits into
v5from
scanner/fix-17911
Nov 12, 2026
Merged

clubanderson merged 7 commits into
v5from
scanner/fix-17911

Conversation

@hivecommons-hive

Copy link
Copy Markdown
Contributor

Closes #17911

Problem

The hub verifies each GitHub App delivery and relays it with GitHub's original X-Hub-Signature-256. The spoke has to re-verify that signature with the App webhook secret, but no hosted spoke has that secret. So hosted spokes answered every relay with 503 and ran on polling only, and event-driven review dispatch (ReviewEventDrivenEnabled(githubWebhooksConfigured())) stayed off.

Design: option 2 (per-hive hub relay key), not option 1 (ship the App secret)

  • Option 1 would copy one fleet-wide secret to every hosted spoke. The App webhook secret authenticates GitHub deliveries for the whole App. With it, any compromised spoke could sign forged deliveries and POST them to any other spoke's public /api/webhook/github. It would also turn one hub secret into one copy per spoke, so a single leak would mean rotating the secret on the GitHub App and on every spoke.
  • Option 2 limits the damage to one hive. Each hosted hive has its own random 256-bit relay key. A key held by one spoke cannot sign a relay that another spoke accepts. No spoke ever receives the App webhook secret.
  • The key travels on a per-hive authenticated channel. It is delivered only over the hive's own heartbeat, which verifyHeartbeatBearer authenticates with the per-hive bearer before anything is attached. A spoke can't claim another hive_id and pull that hive's key.

What changed

Hub

  • SaaSHive.WebhookRelayKey holds the key, generated with crypto/rand.

    • It is set when a hive is created: pool placeholder, handleCreateHive, hosted lite.
    • A hosted hive provisioned before this fix gets a key minted on its first heartbeat, so the existing fleet is covered.
    • The owner-facing handleHiveStatus response removes the key. Only the spoke's heartbeat carries it.
  • Delivery uses HeartbeatGitHubAppConfig.WebhookRelayKey, an additive field.

    • It is attached independently of the App-identity branches, the same way as SecondaryKey, and only while the spoke's reported webhook_relay_key_fingerprint (sha256, non-secret) doesn't match the record.
    • Rotation: replace or clear the key on the record. The fingerprint then stops matching and the next beat delivers the new key.
  • relayWebhookToSpoke forwards GitHub's headers unchanged. For a hive with a relay key it also adds:

    • X-Hive-Relay-Timestamp: <unix seconds>
    • X-Hive-Relay-Signature-256: sha256=<hex HMAC-SHA256(key, timestamp + "." + X-GitHub-Delivery + "." + raw body)>

    Hives the hub did not provision (no record, so no key) get the relay exactly as before.

Spoke

  • github.AdoptWebhookRelayKey stores the key in /data/webhook-relay.key (0600, atomic write), so a restarted spoke can verify relays before its first heartbeat.
    • An empty value is a no-op. It never clears a working key.
    • The key is adopted before the App-identity write guard, so a refused identity push can't drop it.
  • The heartbeat reports WebhookRelayKeyFingerprint, never the key itself.
  • api_github_webhook.go accepts a delivery when either check passes:
    1. The GitHub signature verifies with GITHUB_WEBHOOK_SECRET. This path is unchanged and has no new replay check.
    2. The hub relay signature verifies with the relay key, using a constant-time hmac.Equal. This follows the existing verifyGitHubWebhookSignature.
  • Replay protection (relay path only). These checks run after the MAC verifies:
    • A timestamp more than 5 minutes off in either direction is rejected as relay_timestamp_stale.
    • A missing delivery ID is rejected as relay_delivery_missing.
    • A delivery ID accepted within the last 10 minutes is rejected as relay_delivery_replayed. The IDs live in a per-server in-memory set capped at 4096 entries. 10 minutes is twice the skew window, so it covers every timestamp that would still verify.
    • A bad MAC is rejected as relay_signature_invalid.
    • Each rejection returns 401 with its reason, is logged, and is counted through RecordWebhookDeliveryFailure(), the same as a bad GitHub signature today.
  • With no secret and no relay key, the spoke still answers 503.
  • githubWebhooksConfigured() is now true when a relay key is present, so event-driven review dispatch turns on for hosted hives.
  • /api/status.webhooks has a new verification field: github-signed, relay (hub-signed) or none.

Tests

  • Hub:
    • The relay signature verifies with the target hive's key and not with another hive's.
    • The signature binds the delivery ID and a fresh timestamp.
    • GitHub's signature and body are forwarded unchanged.
    • A hive without a key gets the relay unsigned.
    • Heartbeat: the key is delivered until the spoke confirms it, minted for a legacy hive, not re-sent once the fingerprint matches, and re-delivered after rotation.
    • Hive status never exposes the key.
    • Hub ↔ spoke wire round trip.
  • Spoke:
    • A relay-signed delivery is accepted and applied with no secret.
    • Rejected: tampered body, another hive's key, delivery ID swapped after signing, unsigned relay, stale or future timestamp, missing delivery ID, replayed delivery ID.
    • A rejected relay degrades the cadence like a bad signature.
    • The GitHub-signed path is unchanged, including repeated delivery IDs.
    • The relay key turns githubWebhooksConfigured() and verification on.
    • The replay set stays bounded and expires entries.
  • Spoke store: adopt, redeliver, rotate, reload from disk; an empty value never clears the key; file mode is 0600; the fingerprint format matches the hub's.

Differences from the issue spec

  • Deep health / no_secret / health_deep_read: v5 at f3a7bbf has no webhooks deep-health check, no WebhookReasonNoSecret / mode_reason_code, and no health_deep_read webhook field. Those exist only on v6. On v5, parity means githubWebhooksConfigured() plus the new /api/status.webhooks.verification field. The v6 forward-port (πŸ› [v6] hosted hives reject every relayed GitHub webhook: the hub never gives the spoke a way to verify themΒ #17910) also wires the relay mode into v6's deep-health check and reason codes.
  • Rejection counting: v5 has no per-reason rejection counter. Relay rejections are counted through the existing RecordWebhookDeliveryFailure() and carry their distinct reason in the log line and the 401 body.
  • Rotation: there is no admin endpoint. Rotation means replacing or clearing the key on the hive record, and the heartbeat handles the rest. During one heartbeat interval after a rotation, relays signed with the new key are rejected until the spoke adopts it.
  • Manual redelivery: a GitHub "Redeliver" of the same delivery ID within 10 minutes of the original is rejected as a replay on the relay path. Later redeliveries are accepted.
  • Beyond the spec (security): handleHiveStatus now removes the relay key from the owner-facing record.
  • Storage: the key is kept in the hive's hub record (meta.json), as the issue asks. The hub needs the raw key to sign, so it can't store only a hash the way it does for the dashboard token.

Decision taken by hive: when a spoke has both GITHUB_WEBHOOK_SECRET and a relay key, the GitHub signature is checked first and the relay signature is the fallback, and verification reports github-signed. This is the simplest additive choice and keeps self-hosted behaviour byte-for-byte.

#17910 is the v6 forward-port of this fix.

Validation: only gofmt -l was run locally, per repo policy. CI is the verdict.

β€” hive: agent=scanner backend=copilot model=claude-fable-5.1 copilot=1.0.88 requested_by=@Danathar

…ooks

Signed-off-by: scanner <280983584+hivecommons-hive[bot]@users.noreply.github.com>
Co-authored-by: Danathar <6772335+Danathar@users.noreply.github.com>
@kubestellar-prow kubestellar-prow Bot added dco-signoff: yes Indicates the PR's author has signed the DCO. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Nov 12, 2026
@github-actions

github-actions Bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Open PR overlap check

βœ… Resolved: no conflicting open PRs or same-file overlaps remain against v5.

@hivecommons-hive hivecommons-hive Bot added author/hive This repository's own hive (the App or one of its agent lanes) authored this PR dco All commits are DCO signed-off role/trusted Author is trusted for unattended merge (trusted-author role or registry/claim tier in require_tiers) stage/050-review-pending Review swarm requested; awaiting a verdict for the current head stage/070-fixing Hive-authored PR: CI red, changes requested or needs rebase; a hive agent is fixing it and removed stage/050-review-pending Review swarm requested; awaiting a verdict for the current head labels Nov 12, 2026
@hivecommons-hive

hivecommons-hive Bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor Author

Hive status β€” updated in place; edits do not notify. Full history is in the Activity Monitor.

hive/awaiting-ci: the merge watcher refused the squash because an earlier request on head f3f5cb3 was exhausted by a GitHub API 503 (infra, not this PR). Pushed an empty signed commit to clear the stale refusal; CI pending β€” sweep will merge when green.


🐝 Hive Agent: scanner | Instance: hosted-available-oke-11-placeholder-r05x | SHA: unknown

β€” hive: agent=scanner backend=copilot model=claude-fable-5.1 copilot=1.0.88

Review findings from reviewer-2 (folded here because this head already carries its hive review):

Conclusion: the signing/verification design and implementation read correct to me; the only thing blocking this PR is a CI citation drift that the diff itself caused. One --fix run and a push should turn the checks green.

What I verified (tree at pr17914, merge-base f3a7bbfd4)

  • Sign ↔ verify agree. src/pkg/hub/webhook_relay_key.go:60-67 (webhookRelaySignature) and src/pkg/dashboard/webhook_relay.go:50-65 (verifyWebhookRelaySignature) MAC the same bytes in the same order (ts "." delivery "." body), both hex sha256= and the spoke compares with hmac.Equal. The delivery ID and timestamp are inside the MAC, so the stale/replay checks at webhook_relay.go:109-124 act on authenticated values.
  • Replay cache. firstSeen (webhook_relay.go:76-104) is only called after the MAC and timestamp verify, so unsigned traffic cannot fill it; TTL (10m) β‰₯ skew window (Β±5m) so nothing that still verifies can be replayed. Eviction is O(n) over ≀4096 entries β€” fine for this rate.
  • Fail-closed preserved. api_github_webhook.go:67-79 still 503s with neither secret nor relay key; with a secret and a failed GitHub signature it falls to the relay path only when a relay key and X-Hive-Relay-Signature-256 are present (:81-86), otherwise the old 401. Self-hosted behaviour is unchanged.
  • Key isolation. webhookRelayKeyForHive (hub/webhook_relay_key.go:81-86) reads the target hive's record, so a key held by one spoke can't verify another's relay. Delivery in hub/server.go:3362-3374 runs after verifyHeartbeatBearer (server.go:1924), and handleHiveStatus blanks the key (saas_hives_api.go:1631-1636). I checked the other listSaaSHives() consumers in saas_hives_api.go:586,827: they project into MyHiveEntry, not the raw record, so no other owner-facing path serializes WebhookRelayKey.
  • Spoke store. AdoptWebhookRelayKey (github/webhook_relay_key.go:77-98) is a no-op on empty, writes 0600 via tmp+rename, and runs before the identity write-guard in cmd/hive/main.go:5237-5242, so a refused App-identity push can't drop the key.

The CI failures

relative links and anchors resolve is red on Check api-reference source citations:

DRIFT api-reference/agents-and-controls.md:30 GET /api/review/dispatch/events -> pkg/dashboard/api_review_event_dispatch.go:16 (line does not register it)
DRIFT api-reference/auth.md:27 POST /api/github-app/recheck -> pkg/dashboard/server.go:1687 (line does not register it)
DRIFT api-reference/auth.md:28 POST /api/github-app/install-clicked -> pkg/dashboard/server.go:1688 (line does not register it)
DRIFT api-reference/auth.md:29 GET /gh-setup -> pkg/dashboard/server.go:1689 (line does not register it)

This PR adds a field to dashboard/server.go (:109-110) and an import to api_review_event_dispatch.go (:7), shifting those lines. changelog-fragment-guard fails on the same step ("Route-registering or static page files changed β€” api-reference citations must be resynced"), and native-queue-policy appears to follow from those. Fix per AGENTS.md: run bash src/scripts/check-api-reference-citations.sh --fix (and python3 src/scripts/check-docs-citations.py --fix if it also reports drift), commit, push. The changelog.d/fixed-hosted-webhook-relay-signature.md fragment itself is present and well-formed.

Open question (not a defect in this change)

hub/server.go:1923 only enforces verifyHeartbeatBearer when s.hubSecret != "". The relay key now rides that same lane as the App private keys already do, so this PR does not widen the exposure β€” noting it only so nobody reads the relay key as more protected than the keys next to it.

β€” hive reviewer (model: claude-fable-5.1)

Review findings from reviewer (folded here because this head already carries its hive review):

Read the diff against the v5 tree at 2e05246; I found no defect. This needs a human only because it adds a credential path, not because the code is wrong.

Per #17911 the design question (option 1 vs 2) is already decided β€” option 2 is what the issue recommends and what this implements, so I am not re-asking it. What remains is approver sign-off on the trust surface: webhookRelayKeyForHeartbeat (src/pkg/hub/webhook_relay_key.go:104-121) mints a key for any hosted record that lacks one and delivers it on the next authenticated beat, so the whole existing fleet changes posture on deploy with no operator step. #17915 (the v6 forward-port) is parked on the same question; one decision should cover both.

What I verified in the tree, not just the diff:

  • MAC construction/verification match and are order-correct. Hub webhookRelaySignature (src/pkg/hub/webhook_relay_key.go:60-68) and spoke verifyWebhookRelaySignature (src/pkg/dashboard/webhook_relay.go:52-66) both MAC ts + "." + delivery + "." + body, compare with hmac.Equal. verifyWebhookRelay (webhook_relay.go:106-128) checks the MAC before the timestamp window and replay cache, so unsigned traffic cannot populate the cache β€” as the comment claims.
  • Key delivery sits behind per-hive auth. handleHeartbeat runs verifyHeartbeatBearer(presentedBearer, payload.HiveID) at src/pkg/hub/server.go:1924 before webhookRelayKeyForHeartbeat at :3365, and that helper loads the record by the authenticated HiveID only. (The bearer gate is conditional on s.hubSecret != "" β€” pre-existing, and the same condition already guards App private-key delivery, so not a regression here.)
  • No leak via owner-facing APIs. Grepped every Encode(/Marshal( of a hive record in src/pkg/hub: handleHiveStatus (saas_hives_api.go:1631-1636) is the only place a whole SaaSHive is serialized to HTTP and it blanks WebhookRelayKey; handleMyHives builds its own summary. On-disk meta.json moves to 0600/0700 (saas_provision.go:2093-2112).
  • A relay-key-only heartbeat is safe on the spoke. spoke/heartbeat.go:1478 fires the callback for any non-nil GitHubAppConfig; in main.go:5234 the key is adopted before the identity guard, and prospectiveGitHubIdentity (main_github_app.go:37) returns nil for a config with no AppID/slug/URLs, so nothing else is touched and no empty PEM is written (ghCfg.PrivateKey != "" gate).
  • Fail-closed semantics preserved. api_github_webhook.go:65-94: with neither secret nor key β†’ 503 as before; with a secret, a good X-Hub-Signature-256 still short-circuits; the relay path is tried only when a relay key is held and the header is present.
  • Body matches the diff; no claimed behaviour is missing. Changelog fragment present; operator docs (runtime-and-quota.md) updated to say the secret is self-hosted-only.

One open question, not a defect: the replay cache (webhook_relay.go:69-103) is per-process. A spoke restart inside the 5-minute skew window will accept a captured relay once more. The hub key is still required to produce it, so this only matters if a relay is captured in transit; worth a line in the comment if you want it on record.

I have not approved, closed or labelled anything.

Review findings from adjudicator (folded here because this head already carries its hive review):

Correctness (delta re-review, head 9bbd5f4)

Previous security finding (relay key in world-readable meta.json) β€” fixed: src/pkg/hub/saas_provision.go:2097-2107 now creates the hive dir 0o700 and writes meta.json.tmp 0o600; TestSaveSaaSHiveKeepsRelayKeyPrivate covers both the new-dir and legacy-dir cases.

New in the delta β€” CI test (hub 1/4) is red because of it:

saas_provision.go:2100 runs _ = os.Chmod(dir, 0o700) unconditionally on every save. That doesn't just strip group/other bits β€” it also re-grants owner write on a directory that was deliberately made non-writable. Two existing tests rely on exactly that to simulate a failed save:

  • reset_assignment_sweep_branches_test.go:209 (TestHandleResetAssignmentReportsSaveFailure) β€” chmods the hive dir 0o555, expects HTTP 500; now gets 200 (status = 200, want 500).
  • reset_assignment_sweep_branches_test.go:310 (TestSweepStuckAssignmentsSkipsWedgeWhoseSaveFails) β€” same setup, expects registry ProvStatus to stay assigned; now flips to available.

Both fail in run 43224021088 (DONE 1419 tests, 1 skipped, 2 failures). These are pre-existing passing tests at the merge-base, so this is a regression introduced by the delta, not a flaky shard.

Suggested fix: only clear group/other bits and preserve the owner bits, e.g.

if fi, err := os.Stat(dir); err == nil && fi.Mode().Perm()&0o077 != 0 {
    _ = os.Chmod(dir, fi.Mode().Perm()&^0o077)
}

0o555 β†’ 0o500 keeps the dir unwritable (tests pass), legacy 0o755 β†’ 0o700 (new test still passes), and a save never silently widens a directory an operator locked down.

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Security review (head e0545a5) β€” approve, one hardening advisory

Checked: heartbeat key lane is bearer-bound to hive_id before webhookRelayKeyForHeartbeat runs (src/pkg/hub/server.go:1923-1927, :3365); handleHiveStatus scrubs the key (saas_hives_api.go:1633-1636) and no other handler encodes a whole SaaSHive; hub relay headers are an allowlist (webhook.go:180-183) so GitHub cannot inject X-Hive-Relay-*; spoke MAC is hmac.Equal and replay/skew checks run only after the MAC (dashboard/webhook_relay.go:50-66, 110-127); key file is 0600 via atomic rename (github/webhook_relay_key.go:93-97).

Advisory (low, non-blocking): MAC input has no field framing. webhookRelaySignature (src/pkg/hub/webhook_relay_key.go:63-71) and the spoke twin (src/pkg/dashboard/webhook_relay.go:50-64) MAC ts + "." + delivery + "." + body. Delivery IDs and bodies are attacker-visible, so a captured relay can be re-sent with a body prefix up to any . moved into X-GitHub-Delivery: the MAC still verifies and the new delivery ID bypasses the replay cache (webhook_relay.go:124). Impact today is bounded β€” the truncated body is never valid JSON, so HandleWebhookEvent returns 400 (api_github_webhook.go:111-114) β€” but it is a canonicalization gap a future body parser could expose. Cheap fix, both sides: length-prefix each field (len(ts) "." ts ...) or MAC sha256(delivery) and sha256(body) instead of raw bytes.

Info: the replay set is per-process memory (dashboard/server.go:110), so a legitimate relay captured inside the 5‑minute window can be replayed once across a spoke restart. Idempotent cache invalidation makes this benign; noting for completeness.

β€” hive: agent=adjudicator-2 backend=copilot model=claude-fable-5.1 copilot=1.0.88

Running check-api-reference-citations.sh --fix after the hub relay-key
code landed above the cited route registrations in server.go and
api_review_event_dispatch.go. The api_review_event_dispatch.go:16 -> :15
citation uses the "GET "+constant shape the fixer cannot rewrite, so
it is corrected by hand.

Hive-Fix-Attempt: 1/6
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: scanner <280983584+hivecommons-hive[bot]@users.noreply.github.com>
@hivecommons-hive hivecommons-hive Bot added stage/050-review-pending Review swarm requested; awaiting a verdict for the current head and removed stage/070-fixing Hive-authored PR: CI red, changes requested or needs rebase; a hive agent is fixing it labels Nov 12, 2026

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

docs-currency β€” reviewed at head 28b6512. The docs this PR touches (src/docs/env-vars.md:107, src/docs/operator-reference/runtime-and-quota.md:262-296, 681-687, changelog.d/fixed-hosted-webhook-relay-signature.md) match the new code (src/pkg/dashboard/api_github_webhook.go:65-94, src/pkg/github/webhook_cache.go:48-51). Two untouched operator docs still state the old receiver contract that this PR changes:

  1. src/docs/cel-triggers.md:145-147 β€” "Hosted hives normally receive the GitHub App's signed deliveries through the hub relay; stand-alone hives use POST /api/webhook/github. The receiver requires GITHUB_WEBHOOK_SECRET and a matching HMAC signature." After this PR a hosted spoke holds no GITHUB_WEBHOOK_SECRET and verifies the hub's X-Hive-Relay-Signature-256 with its relay key (api_github_webhook.go:67-68, :81-94). Suggested wording: "The receiver requires either GITHUB_WEBHOOK_SECRET with a matching X-Hub-Signature-256 (self-hosted) or the hub relay key delivered over the heartbeat with a matching hub relay signature (hosted); see runtime-and-quota.md."

  2. src/docs/security-model.md:444 β€” audit row "Are webhooks authenticated? GITHUB_WEBHOOK_SECRET is configured". On a hosted spoke that env var is deliberately unset by design (PR body, runtime-and-quota.md:272-275), so this check would read as "not authenticated" on every hosted hive. Suggest pointing at the new /api/status.webhooks.verification field (github-signed | relay (hub-signed) | none, webhook_cache.go:51) instead, or alongside the env var.

Both are one-line edits in files the PR already has siblings of; no behaviour concern from this perspective.

β€” hive: agent=ci-maintainer backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive

hivecommons-hive Bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor Author

Review follow-ups (below review.backlog.min_severity: P2)

Non-blocking findings below the backlog filing line are listed here instead of as issues, one comment per PR, updated in place. Tick them off as they are fixed; set review.backlog.min_severity: P3 to file every finding.

  • P3 info style β€” webhookRelayKeyForHeartbeat re-reads the hive record the handler already loaded (src/pkg/hub/webhook_relay_key.go:101): handleHeartbeat loads saasHive once at src/pkg/hub/server.go:2980 and threads it into heartbeatUpgradePolicy and the announce block, but the new s.webhookRelayKeyForHeartbeat(&payload) (server.go:336…
  • P3 low docs-currency β€” hub-deployment.md PVC table does not say meta.json now carries per-hive key material (src/docs/hub-deployment.md:25): src/docs/hub-deployment.md:25 describes /data/saas/hives/<id>/meta.json as a 'Per-hive SaaS record used by auth, upgrades, assignments, and dashboard links.' With this PR the record also holds `Web…

@github-actions

github-actions Bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Mutation testing

⏱️ The 900s mutation budget ran out, so 2 package(s) were not fully mutated and the results below do not cover them:

  • pkg/github: skipped (not started)
  • pkg/hub: skipped (not started)

❌ gremlins did not produce a report for 1 package(s), so no verdict can be derived from this run (see the job log):

  • pkg/dashboard: gremlins exited 137

1 mutant(s) in changed files are not reached by any test (NOT COVERED). A maintainer may apply the mutation/acknowledged label to accept this.

File Line Mutation
src/cmd/hive/main.go 5237 CONDITIONALS_NEGATION

Verdict: error (#17349).

@hivecommons-hive hivecommons-hive Bot added stage/070-fixing Hive-authored PR: CI red, changes requested or needs rebase; a hive agent is fixing it and removed stage/050-review-pending Review swarm requested; awaiting a verdict for the current head labels Nov 12, 2026
…ub secrets

SaaSHive.WebhookRelayKey is serialised into saasHivesDir/<id>/meta.json,
which saveSaaSHive wrote 0o644 under a 0o755 directory. The record carried
no key material before this PR; the hub's other secret stores (cluster App
keys, generation stores, session revocation) and the spoke-side relay key
file all use 0o600/0o700. A locally leaked relay key would let an attacker
forge deliveries to that hive's public /api/webhook/github.

Write meta.json 0o600, create the hive directory 0o700, and best-effort
chmod pre-existing directories so hives provisioned before this change are
tightened on their next save. Add a regression test covering both modes.

Hive-Fix-Attempt: 2/6
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: scanner <280983584+hivecommons-hive[bot]@users.noreply.github.com>
@hivecommons-hive hivecommons-hive Bot added stage/050-review-pending Review swarm requested; awaiting a verdict for the current head and removed stage/070-fixing Hive-authored PR: CI red, changes requested or needs rebase; a hive agent is fixing it stage/050-review-pending Review swarm requested; awaiting a verdict for the current head labels Nov 12, 2026
@hivecommons-hive hivecommons-hive Bot added stage/060-reviewed Review swarm verdict approve; a gate is still pending (CI running, soak not started) stage/050-review-pending Review swarm requested; awaiting a verdict for the current head and removed stage/060-reviewed Review swarm verdict approve; a gate is still pending (CI running, soak not started) labels Nov 12, 2026
…erge request

Signed-off-by: scanner <280983584+hivecommons-hive[bot]@users.noreply.github.com>
@hivecommons-hive hivecommons-hive Bot added stage/060-reviewed Review swarm verdict approve; a gate is still pending (CI running, soak not started) and removed stage/050-review-pending Review swarm requested; awaiting a verdict for the current head labels Nov 12, 2026
@hivecommons-hive hivecommons-hive Bot added stage/050-review-pending Review swarm requested; awaiting a verdict for the current head and removed stage/060-reviewed Review swarm verdict approve; a gate is still pending (CI running, soak not started) labels Nov 12, 2026

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Decision needed: @clubanderson should decide whether a per-hive hub relay key (new credential class, minted on heartbeat and persisted at /data/webhook-relay.key) is the accepted way for hosted spokes to trust relayed webhooks, given src/pkg/hub/webhook_relay_key.go falls under CODEOWNERS src/pkg/hub/key.go.

Conclusion: I read the diff against the v5 tree and the callers; I found no correctness or security defect. This introduces a new credential class and lands in a CODEOWNERS-guarded path (src/pkg/hub/*key*.go), so the trust decision belongs to an approver. Sibling for v6 is #17915.

What I verified

  • src/pkg/hub/webhook_relay_key.go:58-66 and src/pkg/dashboard/webhook_relay.go:51-66 compute the same MAC (ts "." delivery "." body); verify uses hmac.Equal and refuses an empty key, so a spoke with no key cannot be driven into the relay branch.
  • src/pkg/dashboard/webhook_relay.go:107-128: delivery ID is only remembered after MAC + skew pass, so unsigned traffic cannot fill the replay cache; cache bounded at 4096 with expiry pruning.
  • src/pkg/dashboard/api_github_webhook.go:81-94: the relay branch is reached only when relayKey != "" and the relay header is present; a self-hosted spoke with GITHUB_WEBHOOK_SECRET and no relay header behaves exactly as before.
  • Key exposure: the only endpoint that serializes a full SaaSHive is handleHiveStatus (src/pkg/hub/saas_hives_api.go:1631-1636), and it now blanks WebhookRelayKey; handleMyHives builds hiveEntry views, not records. saveSaaSHive tightens meta.json to 0600 and strips group/other from the dir without widening owner bits (saas_provision.go:2093-2105).
  • Heartbeat: handleHeartbeat attaches the key the same way SecondaryKey already is (src/pkg/hub/server.go:3362-3375); on the spoke the callback applies fields under the existing "empty means unchanged" contract, and AdoptWebhookRelayKey runs before the identity write-guard (src/cmd/hive/main.go:5237-5241), so a refused identity push does not drop the key.

Design points for the approver (not defects)

  1. When a spoke holds both a secret and a relay key, a relay-signed delivery is accepted even if GitHub's X-Hub-Signature-256 fails (api_github_webhook.go:81). Intentional per the body (hub already verified), but it means the hub is a trusted signer on self-hosted spokes that also hold the secret.
  2. Rotation is manual (edit/clear the record); there is no rotation command or expiry. Fine for a first landing, worth a follow-up issue.

I have not approved or changed anything.

β€” hive: agent=reviewer-2 backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive hivecommons-hive Bot added stage/085-needs-decision A human decision is owed: sentinel alert, hold/needs-decision label, human-only rule or parked issue and removed stage/050-review-pending Review swarm requested; awaiting a verdict for the current head labels Nov 12, 2026
@hivecommons-hive

hivecommons-hive Bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor Author

Needs a decision, not a merge:

  • the reviewer swarm returned requires_human @2e05246
  • the review found: the reviewer marked this change high-risk
  • reviewer question: @clubanderson should sign off on the new credential surface: the hub now auto-mints a long-lived per-hive HMAC relay key for every hosted hive (including the existing fleet, on its next heartbeat) and that key alone authenticates POSTs to the spoke's public /api/webhook/github.
  • the automerge lane will not merge this unattended

Head: verdict requires_human @2e05246, 72/73 checks green, 1 pending, DCO yes, soak 0s of 10m0s.

What to do: approve this head (2e05246) to merge as-is, or leave a changes requested review (or reply /hive decision <what to change>) and the hive will push the fix.

How to release it:

  • After reading the verdict, a trusted merger can submit a GitHub Approve review for the current head and the lane clears the hold on its next pass; /hive approve replies on a PR are acknowledged but not consumed.

Clear the cause and the hive re-evaluates on its next pass; the stage moves to stage/090-ready-for-human-merge, stage/095-pre-merge-soak or the merge queue on its own. Approval does not bypass CI, hold labels or other merge gates.

Updated 2026-11-12 15:07 UTC.

@clubanderson clubanderson left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Decision (as org owner, on behalf of the open question on #17915): option 2 is accepted β€” a hub-minted, per-hive 256-bit relay key, delivered only over the hive's own bearer-authenticated heartbeat and never the App webhook secret, is the trust mechanism for hub-relayed GitHub deliveries on hosted spokes. Also accepted: GitHub signature checked first with the relay signature as fallback; rotation by replacing/clearing the record field (an admin rotation endpoint can be a follow-up); redelivery inside the 10-minute window treated as a replay on the relay path.

Reviewed head 2e05246: hub and spoke MAC the same timestamp.delivery.body, hmac.Equal on the spoke, MAC verified before the replay cache is touched, cache bounded and pruned, key delivered after verifyHeartbeatBearer and gated on fingerprint mismatch, handleHiveStatus strips it, un-hosted spokes and the GITHUB_WEBHOOK_SECRET path are byte-for-byte unchanged, and saveSaaSHive now writes the record rw------- under rwx------ with the pre-existing-dir tightening (9bbd5f4/2ff9d29/f3f5cb3, tests at webhook_relay_key_test.go). Fragment present and correctly named.

One follow-up please (πŸ“–, separate PR on v5): port the docs hunks of v6 commit 4ad3e30 (review-swarm.md event-driven sentence, backup-restore.md note, api-reference verification field) so v5's docs match v6's.

/lgtm
/approve

@clubanderson

Copy link
Copy Markdown
Member

/lgtm
/approve

@kubestellar-prow kubestellar-prow Bot added lgtm Indicates that a PR is ready to be merged. labels Nov 12, 2026
@kubestellar-prow

Copy link
Copy Markdown
Contributor

LGTM label has been added.

DetailsGit tree hash: 2433bb44651810df604779530af05aadc7eb3a37

1 similar comment
@kubestellar-prow

Copy link
Copy Markdown
Contributor

LGTM label has been added.

DetailsGit tree hash: 2433bb44651810df604779530af05aadc7eb3a37

@kubestellar-prow

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: clubanderson

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@kubestellar-prow kubestellar-prow Bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Nov 12, 2026
@kubestellar-prow

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: clubanderson

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@clubanderson
clubanderson merged commit b962347 into v5 Nov 12, 2026
102 of 110 checks passed
@kubestellar-prow
kubestellar-prow Bot deleted the scanner/fix-17911 branch November 12, 2026 15:32
@hivecommons-hive hivecommons-hive Bot removed role/trusted Author is trusted for unattended merge (trusted-author role or registry/claim tier in require_tiers) stage/085-needs-decision A human decision is owed: sentinel alert, hold/needs-decision label, human-only rule or parked issue labels Nov 12, 2026
hivecommons-hive Bot added a commit that referenced this pull request Nov 13, 2026
…ooks (#17914)

* πŸ› fix: hub-signed relay lets hosted spokes verify relayed GitHub webhooks

Signed-off-by: scanner <280983584+hivecommons-hive[bot]@users.noreply.github.com>
Co-authored-by: Danathar <6772335+Danathar@users.noreply.github.com>

* πŸ“ docs: refresh api-reference citations shifted by relay-key change

Running check-api-reference-citations.sh --fix after the hub relay-key
code landed above the cited route registrations in server.go and
api_review_event_dispatch.go. The api_review_event_dispatch.go:16 -> :15
citation uses the "GET "+constant shape the fixer cannot rewrite, so
it is corrected by hand.

Hive-Fix-Attempt: 1/6
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: scanner <280983584+hivecommons-hive[bot]@users.noreply.github.com>

* πŸ”’ fix: store hive meta.json with the relay key rw------- like other hub secrets

SaaSHive.WebhookRelayKey is serialised into saasHivesDir/<id>/meta.json,
which saveSaaSHive wrote 0o644 under a 0o755 directory. The record carried
no key material before this PR; the hub's other secret stores (cluster App
keys, generation stores, session revocation) and the spoke-side relay key
file all use 0o600/0o700. A locally leaked relay key would let an attacker
forge deliveries to that hive's public /api/webhook/github.

Write meta.json 0o600, create the hive directory 0o700, and best-effort
chmod pre-existing directories so hives provisioned before this change are
tightened on their next save. Add a regression test covering both modes.

Hive-Fix-Attempt: 2/6
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: scanner <280983584+hivecommons-hive[bot]@users.noreply.github.com>

* πŸ› fix: tighten hive dir perms without widening a read-only dir

saveSaaSHive forced the hive directory to 0700, which re-added owner-write
to directories the reset-assignment tests lock to 0555 to simulate a failed
persist, so TestHandleResetAssignmentReportsSaveFailure and
TestSweepStuckAssignmentsSkipsWedgeWhoseSaveFails no longer saw the failure.
Strip only group/other bits so legacy 0755 dirs still become 0700 while a
deliberately read-only dir keeps failing the write.

Signed-off-by: scanner <280983584+hivecommons-hive[bot]@users.noreply.github.com>

* ♻️ test: reuse useTempSaaSHivesDir in saveSaaSHive perm tests

TestSaveSaaSHiveKeepsRelayKeyPrivate and TestSaveSaaSHiveNeverWidensReadOnlyDir
swapped saasHivesDir for t.TempDir() inline even though the file already
defines useTempSaaSHivesDir for exactly that. Use the helper like the
sibling tests do.

Hive-Fix-Attempt: 3/6
Signed-off-by: scanner <280983584+hivecommons-hive[bot]@users.noreply.github.com>

* chore: retrigger merge watcher after GitHub 503 exhausted the prior merge request

Signed-off-by: scanner <280983584+hivecommons-hive[bot]@users.noreply.github.com>

---------

Signed-off-by: scanner <280983584+hivecommons-hive[bot]@users.noreply.github.com>
Co-authored-by: scanner <280983584+hivecommons-hive[bot]@users.noreply.github.com>
Co-authored-by: Danathar <6772335+Danathar@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. author/hive This repository's own hive (the App or one of its agent lanes) authored this PR copilot dco All commits are DCO signed-off dco-signoff: yes Indicates the PR's author has signed the DCO. lgtm Indicates that a PR is ready to be merged. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

πŸ› [v5] hosted hives reject every relayed GitHub webhook: the hub never gives the spoke a way to verify them

1 participant