Repository navigation
π fix: hub-signed relay lets hosted spokes verify relayed GitHub webhooks - #17914
Conversation
β¦ooks Signed-off-by: scanner <280983584+hivecommons-hive[bot]@users.noreply.github.com> Co-authored-by: Danathar <6772335+Danathar@users.noreply.github.com>
Open PR overlap checkβ
Resolved: no conflicting open PRs or same-file overlaps remain against |
|
Hive status β updated in place; edits do not notify. Full history is in the Activity Monitor. hive/awaiting-ci: the merge watcher refused the squash because an earlier request on head f3f5cb3 was exhausted by a GitHub API 503 (infra, not this PR). Pushed an empty signed commit to clear the stale refusal; CI pending β sweep will merge when green. π Hive Agent: β hive: agent=scanner backend=copilot model=claude-fable-5.1 copilot=1.0.88 Review findings from Conclusion: the signing/verification design and implementation read correct to me; the only thing blocking this PR is a CI citation drift that the diff itself caused. One What I verified (tree at
|
There was a problem hiding this comment.
Security review (head e0545a5) β approve, one hardening advisory
Checked: heartbeat key lane is bearer-bound to hive_id before webhookRelayKeyForHeartbeat runs (src/pkg/hub/server.go:1923-1927, :3365); handleHiveStatus scrubs the key (saas_hives_api.go:1633-1636) and no other handler encodes a whole SaaSHive; hub relay headers are an allowlist (webhook.go:180-183) so GitHub cannot inject X-Hive-Relay-*; spoke MAC is hmac.Equal and replay/skew checks run only after the MAC (dashboard/webhook_relay.go:50-66, 110-127); key file is 0600 via atomic rename (github/webhook_relay_key.go:93-97).
Advisory (low, non-blocking): MAC input has no field framing. webhookRelaySignature (src/pkg/hub/webhook_relay_key.go:63-71) and the spoke twin (src/pkg/dashboard/webhook_relay.go:50-64) MAC ts + "." + delivery + "." + body. Delivery IDs and bodies are attacker-visible, so a captured relay can be re-sent with a body prefix up to any . moved into X-GitHub-Delivery: the MAC still verifies and the new delivery ID bypasses the replay cache (webhook_relay.go:124). Impact today is bounded β the truncated body is never valid JSON, so HandleWebhookEvent returns 400 (api_github_webhook.go:111-114) β but it is a canonicalization gap a future body parser could expose. Cheap fix, both sides: length-prefix each field (len(ts) "." ts ...) or MAC sha256(delivery) and sha256(body) instead of raw bytes.
Info: the replay set is per-process memory (dashboard/server.go:110), so a legitimate relay captured inside the 5βminute window can be replayed once across a spoke restart. Idempotent cache invalidation makes this benign; noting for completeness.
β hive: agent=adjudicator-2 backend=copilot model=claude-fable-5.1 copilot=1.0.88
Running check-api-reference-citations.sh --fix after the hub relay-key code landed above the cited route registrations in server.go and api_review_event_dispatch.go. The api_review_event_dispatch.go:16 -> :15 citation uses the "GET "+constant shape the fixer cannot rewrite, so it is corrected by hand. Hive-Fix-Attempt: 1/6 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: scanner <280983584+hivecommons-hive[bot]@users.noreply.github.com>
There was a problem hiding this comment.
docs-currency β reviewed at head 28b6512. The docs this PR touches (src/docs/env-vars.md:107, src/docs/operator-reference/runtime-and-quota.md:262-296, 681-687, changelog.d/fixed-hosted-webhook-relay-signature.md) match the new code (src/pkg/dashboard/api_github_webhook.go:65-94, src/pkg/github/webhook_cache.go:48-51). Two untouched operator docs still state the old receiver contract that this PR changes:
-
src/docs/cel-triggers.md:145-147β "Hosted hives normally receive the GitHub App's signed deliveries through the hub relay; stand-alone hives usePOST /api/webhook/github. The receiver requiresGITHUB_WEBHOOK_SECRETand a matching HMAC signature." After this PR a hosted spoke holds noGITHUB_WEBHOOK_SECRETand verifies the hub'sX-Hive-Relay-Signature-256with its relay key (api_github_webhook.go:67-68,:81-94). Suggested wording: "The receiver requires eitherGITHUB_WEBHOOK_SECRETwith a matchingX-Hub-Signature-256(self-hosted) or the hub relay key delivered over the heartbeat with a matching hub relay signature (hosted); see runtime-and-quota.md." -
src/docs/security-model.md:444β audit row "Are webhooks authenticated?GITHUB_WEBHOOK_SECRETis configured". On a hosted spoke that env var is deliberately unset by design (PR body,runtime-and-quota.md:272-275), so this check would read as "not authenticated" on every hosted hive. Suggest pointing at the new/api/status.webhooks.verificationfield (github-signed|relay (hub-signed)|none,webhook_cache.go:51) instead, or alongside the env var.
Both are one-line edits in files the PR already has siblings of; no behaviour concern from this perspective.
β hive: agent=ci-maintainer backend=copilot model=claude-fable-5.1 copilot=1.0.88
Review follow-ups (below
|
Mutation testingβ±οΈ The 900s mutation budget ran out, so 2 package(s) were not fully mutated and the results below do not cover them:
β gremlins did not produce a report for 1 package(s), so no verdict can be derived from this run (see the job log):
1 mutant(s) in changed files are not reached by any test (NOT COVERED). A maintainer may apply the
Verdict: |
β¦ub secrets SaaSHive.WebhookRelayKey is serialised into saasHivesDir/<id>/meta.json, which saveSaaSHive wrote 0o644 under a 0o755 directory. The record carried no key material before this PR; the hub's other secret stores (cluster App keys, generation stores, session revocation) and the spoke-side relay key file all use 0o600/0o700. A locally leaked relay key would let an attacker forge deliveries to that hive's public /api/webhook/github. Write meta.json 0o600, create the hive directory 0o700, and best-effort chmod pre-existing directories so hives provisioned before this change are tightened on their next save. Add a regression test covering both modes. Hive-Fix-Attempt: 2/6 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: scanner <280983584+hivecommons-hive[bot]@users.noreply.github.com>
β¦erge request Signed-off-by: scanner <280983584+hivecommons-hive[bot]@users.noreply.github.com>
There was a problem hiding this comment.
Decision needed: @clubanderson should decide whether a per-hive hub relay key (new credential class, minted on heartbeat and persisted at /data/webhook-relay.key) is the accepted way for hosted spokes to trust relayed webhooks, given src/pkg/hub/webhook_relay_key.go falls under CODEOWNERS src/pkg/hub/key.go.
Conclusion: I read the diff against the v5 tree and the callers; I found no correctness or security defect. This introduces a new credential class and lands in a CODEOWNERS-guarded path (src/pkg/hub/*key*.go), so the trust decision belongs to an approver. Sibling for v6 is #17915.
What I verified
src/pkg/hub/webhook_relay_key.go:58-66andsrc/pkg/dashboard/webhook_relay.go:51-66compute the same MAC (ts "." delivery "." body); verify useshmac.Equaland refuses an empty key, so a spoke with no key cannot be driven into the relay branch.src/pkg/dashboard/webhook_relay.go:107-128: delivery ID is only remembered after MAC + skew pass, so unsigned traffic cannot fill the replay cache; cache bounded at 4096 with expiry pruning.src/pkg/dashboard/api_github_webhook.go:81-94: the relay branch is reached only whenrelayKey != ""and the relay header is present; a self-hosted spoke withGITHUB_WEBHOOK_SECRETand no relay header behaves exactly as before.- Key exposure: the only endpoint that serializes a full
SaaSHiveishandleHiveStatus(src/pkg/hub/saas_hives_api.go:1631-1636), and it now blanksWebhookRelayKey;handleMyHivesbuildshiveEntryviews, not records.saveSaaSHivetightensmeta.jsonto 0600 and strips group/other from the dir without widening owner bits (saas_provision.go:2093-2105). - Heartbeat:
handleHeartbeatattaches the key the same waySecondaryKeyalready is (src/pkg/hub/server.go:3362-3375); on the spoke the callback applies fields under the existing "empty means unchanged" contract, andAdoptWebhookRelayKeyruns before the identity write-guard (src/cmd/hive/main.go:5237-5241), so a refused identity push does not drop the key.
Design points for the approver (not defects)
- When a spoke holds both a secret and a relay key, a relay-signed delivery is accepted even if GitHub's
X-Hub-Signature-256fails (api_github_webhook.go:81). Intentional per the body (hub already verified), but it means the hub is a trusted signer on self-hosted spokes that also hold the secret. - Rotation is manual (edit/clear the record); there is no rotation command or expiry. Fine for a first landing, worth a follow-up issue.
I have not approved or changed anything.
β hive: agent=reviewer-2 backend=copilot model=claude-fable-5.1 copilot=1.0.88
|
Needs a decision, not a merge:
Head: verdict requires_human @2e05246, 72/73 checks green, 1 pending, DCO yes, soak 0s of 10m0s. What to do: approve this head ( How to release it:
Clear the cause and the hive re-evaluates on its next pass; the stage moves to Updated 2026-11-12 15:07 UTC. |
clubanderson
left a comment
There was a problem hiding this comment.
Decision (as org owner, on behalf of the open question on #17915): option 2 is accepted β a hub-minted, per-hive 256-bit relay key, delivered only over the hive's own bearer-authenticated heartbeat and never the App webhook secret, is the trust mechanism for hub-relayed GitHub deliveries on hosted spokes. Also accepted: GitHub signature checked first with the relay signature as fallback; rotation by replacing/clearing the record field (an admin rotation endpoint can be a follow-up); redelivery inside the 10-minute window treated as a replay on the relay path.
Reviewed head 2e05246: hub and spoke MAC the same timestamp.delivery.body, hmac.Equal on the spoke, MAC verified before the replay cache is touched, cache bounded and pruned, key delivered after verifyHeartbeatBearer and gated on fingerprint mismatch, handleHiveStatus strips it, un-hosted spokes and the GITHUB_WEBHOOK_SECRET path are byte-for-byte unchanged, and saveSaaSHive now writes the record rw------- under rwx------ with the pre-existing-dir tightening (9bbd5f4/2ff9d29/f3f5cb3, tests at webhook_relay_key_test.go). Fragment present and correctly named.
One follow-up please (π, separate PR on v5): port the docs hunks of v6 commit 4ad3e30 (review-swarm.md event-driven sentence, backup-restore.md note, api-reference verification field) so v5's docs match v6's.
/lgtm
/approve
|
/lgtm |
|
LGTM label has been added. DetailsGit tree hash: 2433bb44651810df604779530af05aadc7eb3a37 |
1 similar comment
|
LGTM label has been added. DetailsGit tree hash: 2433bb44651810df604779530af05aadc7eb3a37 |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: clubanderson The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: clubanderson The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
β¦ooks (#17914) * π fix: hub-signed relay lets hosted spokes verify relayed GitHub webhooks Signed-off-by: scanner <280983584+hivecommons-hive[bot]@users.noreply.github.com> Co-authored-by: Danathar <6772335+Danathar@users.noreply.github.com> * π docs: refresh api-reference citations shifted by relay-key change Running check-api-reference-citations.sh --fix after the hub relay-key code landed above the cited route registrations in server.go and api_review_event_dispatch.go. The api_review_event_dispatch.go:16 -> :15 citation uses the "GET "+constant shape the fixer cannot rewrite, so it is corrected by hand. Hive-Fix-Attempt: 1/6 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: scanner <280983584+hivecommons-hive[bot]@users.noreply.github.com> * π fix: store hive meta.json with the relay key rw------- like other hub secrets SaaSHive.WebhookRelayKey is serialised into saasHivesDir/<id>/meta.json, which saveSaaSHive wrote 0o644 under a 0o755 directory. The record carried no key material before this PR; the hub's other secret stores (cluster App keys, generation stores, session revocation) and the spoke-side relay key file all use 0o600/0o700. A locally leaked relay key would let an attacker forge deliveries to that hive's public /api/webhook/github. Write meta.json 0o600, create the hive directory 0o700, and best-effort chmod pre-existing directories so hives provisioned before this change are tightened on their next save. Add a regression test covering both modes. Hive-Fix-Attempt: 2/6 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: scanner <280983584+hivecommons-hive[bot]@users.noreply.github.com> * π fix: tighten hive dir perms without widening a read-only dir saveSaaSHive forced the hive directory to 0700, which re-added owner-write to directories the reset-assignment tests lock to 0555 to simulate a failed persist, so TestHandleResetAssignmentReportsSaveFailure and TestSweepStuckAssignmentsSkipsWedgeWhoseSaveFails no longer saw the failure. Strip only group/other bits so legacy 0755 dirs still become 0700 while a deliberately read-only dir keeps failing the write. Signed-off-by: scanner <280983584+hivecommons-hive[bot]@users.noreply.github.com> * β»οΈ test: reuse useTempSaaSHivesDir in saveSaaSHive perm tests TestSaveSaaSHiveKeepsRelayKeyPrivate and TestSaveSaaSHiveNeverWidensReadOnlyDir swapped saasHivesDir for t.TempDir() inline even though the file already defines useTempSaaSHivesDir for exactly that. Use the helper like the sibling tests do. Hive-Fix-Attempt: 3/6 Signed-off-by: scanner <280983584+hivecommons-hive[bot]@users.noreply.github.com> * chore: retrigger merge watcher after GitHub 503 exhausted the prior merge request Signed-off-by: scanner <280983584+hivecommons-hive[bot]@users.noreply.github.com> --------- Signed-off-by: scanner <280983584+hivecommons-hive[bot]@users.noreply.github.com> Co-authored-by: scanner <280983584+hivecommons-hive[bot]@users.noreply.github.com> Co-authored-by: Danathar <6772335+Danathar@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Closes #17911
Problem
The hub verifies each GitHub App delivery and relays it with GitHub's original
X-Hub-Signature-256. The spoke has to re-verify that signature with the App webhook secret, but no hosted spoke has that secret. So hosted spokes answered every relay with 503 and ran on polling only, and event-driven review dispatch (ReviewEventDrivenEnabled(githubWebhooksConfigured())) stayed off.Design: option 2 (per-hive hub relay key), not option 1 (ship the App secret)
/api/webhook/github. It would also turn one hub secret into one copy per spoke, so a single leak would mean rotating the secret on the GitHub App and on every spoke.verifyHeartbeatBearerauthenticates with the per-hive bearer before anything is attached. A spoke can't claim anotherhive_idand pull that hive's key.What changed
Hub
SaaSHive.WebhookRelayKeyholds the key, generated with crypto/rand.handleCreateHive, hosted lite.handleHiveStatusresponse removes the key. Only the spoke's heartbeat carries it.Delivery uses
HeartbeatGitHubAppConfig.WebhookRelayKey, an additive field.SecondaryKey, and only while the spoke's reportedwebhook_relay_key_fingerprint(sha256, non-secret) doesn't match the record.relayWebhookToSpokeforwards GitHub's headers unchanged. For a hive with a relay key it also adds:X-Hive-Relay-Timestamp: <unix seconds>X-Hive-Relay-Signature-256: sha256=<hex HMAC-SHA256(key, timestamp + "." + X-GitHub-Delivery + "." + raw body)>Hives the hub did not provision (no record, so no key) get the relay exactly as before.
Spoke
github.AdoptWebhookRelayKeystores the key in/data/webhook-relay.key(0600, atomic write), so a restarted spoke can verify relays before its first heartbeat.WebhookRelayKeyFingerprint, never the key itself.api_github_webhook.goaccepts a delivery when either check passes:GITHUB_WEBHOOK_SECRET. This path is unchanged and has no new replay check.hmac.Equal. This follows the existingverifyGitHubWebhookSignature.relay_timestamp_stale.relay_delivery_missing.relay_delivery_replayed. The IDs live in a per-server in-memory set capped at 4096 entries. 10 minutes is twice the skew window, so it covers every timestamp that would still verify.relay_signature_invalid.RecordWebhookDeliveryFailure(), the same as a bad GitHub signature today.githubWebhooksConfigured()is now true when a relay key is present, so event-driven review dispatch turns on for hosted hives./api/status.webhookshas a newverificationfield:github-signed,relay (hub-signed)ornone.Tests
githubWebhooksConfigured()andverificationon.Differences from the issue spec
no_secret/health_deep_read: v5 at f3a7bbf has no webhooks deep-health check, noWebhookReasonNoSecret/mode_reason_code, and nohealth_deep_readwebhook field. Those exist only on v6. On v5, parity meansgithubWebhooksConfigured()plus the new/api/status.webhooks.verificationfield. The v6 forward-port (π [v6] hosted hives reject every relayed GitHub webhook: the hub never gives the spoke a way to verify themΒ #17910) also wires the relay mode into v6's deep-health check and reason codes.RecordWebhookDeliveryFailure()and carry their distinct reason in the log line and the 401 body.handleHiveStatusnow removes the relay key from the owner-facing record.meta.json), as the issue asks. The hub needs the raw key to sign, so it can't store only a hash the way it does for the dashboard token.Decision taken by hive: when a spoke has both
GITHUB_WEBHOOK_SECRETand a relay key, the GitHub signature is checked first and the relay signature is the fallback, andverificationreportsgithub-signed. This is the simplest additive choice and keeps self-hosted behaviour byte-for-byte.#17910 is the v6 forward-port of this fix.
Validation: only
gofmt -lwas run locally, per repo policy. CI is the verdict.β hive: agent=scanner backend=copilot model=claude-fable-5.1 copilot=1.0.88 requested_by=@Danathar