Repository navigation
π fix: [v6] hub-signed relay lets hosted spokes verify relayed GitHub webhooks - #17915
Conversation
β¦ooks Signed-off-by: scanner <280983584+hivecommons-hive[bot]@users.noreply.github.com> Co-authored-by: Danathar <6772335+Danathar@users.noreply.github.com>
|
Hive status β updated in place; edits do not notify. Full history is in the Activity Monitor. hive/awaiting-ci: pushed api-reference citation resync (the links/changelog-guard failures were citation drift from the server.go route shift). CI pending β sweep will merge when green. π Hive Agent: β hive: agent=scanner backend=copilot model=claude-fable-5.1 copilot=1.0.88 Review findings from Read the diff against Two things need a push before this can land: 1. Both red checks are the same citation drift (blocking). The comment edit at 2. The new hub-side secret lands in a world-readable file (medium). Not a defect, just noting: Review findings from Security perspective (head a8a6ac0) β no in-scope blocker. Verified: the heartbeat lane that delivers One low advisory, non-blocking:
Review findings from Correctness review (head a8a6ac0) β relay signing/verification, replay cache, key delivery and persistence check out against #17910; the hub and spoke MACs agree byte-for-byte ( One advisory (low):
|
Open PR overlap checkBase branch: Conflicts are checked with every PR merged onto the current tip of
|
There was a problem hiding this comment.
Docs-currency review (head b1685f3). The three touched docs are accurate against the code, but three operator-facing docs this change invalidates were not updated.
-
src/docs/review-swarm.md:733β still saysreview.event_drivenunset "means on exactly whenGITHUB_WEBHOOK_SECRETis set". After this PR the gate isgithubWebhooksConfigured()βgithub.WebhookVerification() != WebhookVerificationNone(src/pkg/dashboard/api_review_event_dispatch.go:29-31), so it is also on with a hub relay key and no secret. Turning event-driven dispatch on for hosted hives is the headline outcome of #17910, and this sentence now tells a hosted operator the opposite. Fix: "β¦on exactly when the spoke can verify webhooks (GITHUB_WEBHOOK_SECRET, or a hub relay key on a hosted hive β/api/status.webhooks.verification != none)". -
src/docs/api-reference.md:37andsrc/docs/api-reference/health-status-events.md:28β the/api/health/deepwebhooksfield list (status,detail,mode,mode_reason_code,rejected_1h) omits the newverificationkey added atsrc/pkg/dashboard/server.go:4012, and the cause list ("noGITHUB_WEBHOOK_SECRET, signature mismatch, undecodable payload") omits the four relay rejections now surfaced as warnings (src/pkg/dashboard/health_webhooks.go:36-41). -
src/docs/webhooks.md:242β "signature_failedmeans the secret does not match the sender's". On a hosted spoke with no secret, a relay that arrives withoutX-Hive-Relay-Signature-256is also recorded assignature_failed(src/pkg/dashboard/api_github_webhook.go:86-91), i.e. an unsigned/unprovisioned relay, not a secret mismatch. One clause covering that case would keep the reason-code table truthful.
Advisory (not blocking): /data/webhook-relay.key is not in the spoke backup manifest (src/docs/backup-restore.md:182-190); that is fine since the hub re-delivers on the next heartbeat when the fingerprint is missing, but worth a one-line note there so a restore operator does not go looking for it.
β hive: agent=reviewer-2 backend=copilot model=claude-fable-5.1 copilot=1.0.88
Review follow-ups (below
|
β¦ason aliases Review follow-ups for #17915: - review-swarm.md: event_driven default keys off webhook verification (secret or hub relay key), not only GITHUB_WEBHOOK_SECRET - api-reference: /api/health/deep webhooks object now lists verification and the relay_* warn causes - webhooks.md: signature_failed also covers an unsigned relay on a hosted spoke without a secret - backup-restore.md: note /data/webhook-relay.key is intentionally not archived (hub re-delivers it) - pkg/dashboard: use github.WebhookReasonRelay* directly instead of package-local aliases Hive-Fix-Attempt: 1/6 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: scanner <280983584+hivecommons-hive[bot]@users.noreply.github.com>
Signed-off-by: scanner <280983584+hivecommons-hive[bot]@users.noreply.github.com>
There was a problem hiding this comment.
Decision needed: @Danathar should decide whether a hub-minted per-hive relay credential, stored in plaintext on the hive record (meta.json) and delivered to spokes over the heartbeat, is acceptable as the trust mechanism for the public /api/webhook/github endpoint on hosted spokes.
Conclusion: I read the signing/verification code, the heartbeat delivery path and the callers on both sides and found no correctness defect. This introduces a new credential and a new authentication path on a public endpoint, so a maintainer should sign off on the trust model rather than me. The only red CI is mechanical docs-citation drift.
What I verified (v6 @ 4ad3e30)
- Signing and verification agree byte-for-byte:
pkg/hub/webhook_relay_key.go:60-68andpkg/dashboard/webhook_relay.go:43-58both MACtimestamp "." delivery "." body;hmac.Equalon the spoke. - The MAC is checked before the timestamp is parsed or the delivery ID is remembered (
webhook_relay.go:104-119), so unsigned traffic cannot fill the replay cache. Cache is bounded at 4096 and pruned on insert (webhook_relay.go:71-96). X-GitHub-Deliveryis in the hub's forwarded header set (pkg/hub/webhook.go:181), so the spoke'sdeliverybinding sees the same value the hub signed.- Key delivery runs after
verifyHeartbeatBearer(pkg/hub/server.go:1928β:3376), so a spoke is only ever handed its own hive's key. Note the bearer check is gated ons.hubSecret != ""β pre-existing, and the same gate the App private-key lanes already rely on, so not a regression. handleHiveStatuscopies the record and blanksWebhookRelayKeybefore encoding (saas_hives_api.go:1637-1640). The list endpoints atsaas_hives_api.go:587/828/2241project into entry structs rather than encodingSaaSHive, so I found no other owner-facing path that leaks it.- Self-hosted path unchanged: with
GITHUB_WEBHOOK_SECRETset and no relay key, a bad GitHub signature still fails atapi_github_webhook.go:86-92exactly as before. - Body claims I checked against the diff all hold (fingerprint-gated delivery, 0600 atomic write, empty-key no-op, legacy-fleet minting on first beat).
Why CI is red
changelog-fragment-guard / Docs Link Check / native-queue-policy are all the same failure: the citation guard reports DRIFT for ~24 src/docs/api-reference*.md lines pointing at pkg/dashboard/server.go and api_review_event_dispatch.go. Cause: the new webhookRelaySeen field at server.go:110 and the import change in api_review_event_dispatch.go:3-9 shifted line numbers. Per AGENTS.md, run python3 src/scripts/check-docs-citations.py --fix and push β no hand edits. The fragment changelog.d/fixed-hosted-webhook-relay-signature.md itself is present and well-formed.
Open question for the decision above (not a defect): the relay key lives in plaintext in the hive record alongside the other per-hive material, and the hub needs the raw key to sign. The PR body states this and chooses it deliberately. I am flagging it only because a new long-lived credential on a public endpoint is a trust-policy call, not an engineering one.
I have not approved or changed anything.
β hive: agent=reviewer backend=copilot model=claude-fable-5.1 copilot=1.0.88
|
Needs a decision, not a merge:
Head: verdict requires_human @b4fa5f1, 70/72 checks green, 2 pending, DCO yes, soak 0s of 10m0s. What to do: approve this head ( How to release it:
Clear the cause and the hive re-evaluates on its next pass; the stage moves to |
|
Decision recorded by @clubanderson for head |
β¦ub secrets SaaSHive.WebhookRelayKey is serialised into saasHivesDir/<id>/meta.json, which saveSaaSHive wrote 0o644 under a 0o755 directory. The record carried no key material before this PR; the hub's other secret stores (cluster App keys, generation stores, session revocation) and the spoke-side relay key file all use 0o600/0o700. A locally leaked relay key would let an attacker forge deliveries to that hive's public /api/webhook/github. Write meta.json 0o600, create the hive directory 0o700, and best-effort chmod pre-existing directories so hives provisioned before this change are tightened on their next save. Add a regression test covering both modes. Hive-Fix-Attempt: 2/6 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: scanner <280983584+hivecommons-hive[bot]@users.noreply.github.com> (cherry picked from commit 9bbd5f4) Signed-off-by: scanner <280983584+hivecommons-hive[bot]@users.noreply.github.com> Hive-Fix-Attempt: 2/6
saveSaaSHive forced the hive directory to 0700, which re-added owner-write to directories the reset-assignment tests lock to 0555 to simulate a failed persist, so TestHandleResetAssignmentReportsSaveFailure and TestSweepStuckAssignmentsSkipsWedgeWhoseSaveFails no longer saw the failure. Strip only group/other bits so legacy 0755 dirs still become 0700 while a deliberately read-only dir keeps failing the write. Signed-off-by: scanner <280983584+hivecommons-hive[bot]@users.noreply.github.com> (cherry picked from commit 2ff9d29) Signed-off-by: scanner <280983584+hivecommons-hive[bot]@users.noreply.github.com> Hive-Fix-Attempt: 2/6
TestSaveSaaSHiveKeepsRelayKeyPrivate and TestSaveSaaSHiveNeverWidensReadOnlyDir swapped saasHivesDir for t.TempDir() inline even though the file already defines useTempSaaSHivesDir for exactly that. Use the helper like the sibling tests do. Hive-Fix-Attempt: 3/6 Signed-off-by: scanner <280983584+hivecommons-hive[bot]@users.noreply.github.com> (cherry picked from commit f3f5cb3) Signed-off-by: scanner <280983584+hivecommons-hive[bot]@users.noreply.github.com> Hive-Fix-Attempt: 2/6
There was a problem hiding this comment.
Style/maintainability pass at a8a6ac0 β no blocker; one advisory.
PR body omits an on-disk permission change. src/pkg/hub/saas_provision.go:2090-2105 now creates hive record dirs 0o700 (was 0o755), strips group/other bits from pre-existing dirs on every save (:2097), and writes meta.json 0o600 (was 0o644). The body's "What changed" / "Storage" sections only say the key lives in meta.json. Anything outside the hub process that read hive records (backups, sidecars, another uid) silently loses access after the first save. Please add this to the PR body so operators see it in the changelog trail.
Minor, no action needed: the relay wire format (header names, MAC layout, sha256: fingerprint) is implemented three times β src/pkg/hub/webhook_relay_key.go:34-70, src/pkg/dashboard/webhook_relay.go:30-57, src/pkg/github/webhook_relay_key.go:122-130. That matches the existing hub/spoke mirrored-type convention, and the cross-package tests pin it, so it's consistent with the tree β just something to keep in sync on any future change.
β hive: agent=reviewer-2 backend=copilot model=claude-fable-5.1 copilot=1.0.88
clubanderson
left a comment
There was a problem hiding this comment.
Hardening commits now match v5 (#17914); v5/v6 pair equivalent. /lgtm /approve
|
/lgtm |
|
LGTM label has been added. DetailsGit tree hash: 78a0fd1ffd53dc0812b0262622e7d897b02c0649 |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: clubanderson The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
Thank you for your contribution! Your PR has been merged. We'd love to hear how your experience was: share feedback |
Closes #17910
Forward-port of the v5 fix (PR #17914) for #17911, cherry-picked onto v6 @ c984347. Conflicts were resolved so v6's additions are kept:
Rejected1h,LastRejectedAt,annotateModeLocked, and per-reasonRecordWebhookDeliveryFailure(reason).Problem
The hub verifies each GitHub App delivery and relays it with GitHub's original
X-Hub-Signature-256. The spoke has to re-verify that signature with the App webhook secret, but no hosted spoke has that secret. So hosted spokes answered every relay with 503 and ran on polling only, and event-driven review dispatch (ReviewEventDrivenEnabled(githubWebhooksConfigured())) stayed off.Design: option 2 (per-hive hub relay key), not option 1 (ship the App secret)
/api/webhook/github. It would also turn one hub secret into one copy per spoke, so a single leak would mean rotating the secret on the GitHub App and on every spoke.verifyHeartbeatBearerauthenticates with the per-hive bearer before anything is attached. A spoke can't claim anotherhive_idand pull that hive's key.What changed
Hub
SaaSHive.WebhookRelayKeyholds the key, generated with crypto/rand.handleCreateHive, hosted lite.handleHiveStatusresponse removes the key. Only the spoke's heartbeat carries it.Delivery uses
HeartbeatGitHubAppConfig.WebhookRelayKey, an additive field.SecondaryKey, and only while the spoke's reportedwebhook_relay_key_fingerprint(sha256, non-secret) doesn't match the record.relayWebhookToSpokeforwards GitHub's headers unchanged. For a hive with a relay key it also adds:X-Hive-Relay-Timestamp: <unix seconds>X-Hive-Relay-Signature-256: sha256=<hex HMAC-SHA256(key, timestamp + "." + X-GitHub-Delivery + "." + raw body)>Hives the hub did not provision (no record, so no key) get the relay exactly as before.
Spoke
github.AdoptWebhookRelayKeystores the key in/data/webhook-relay.key(0600, atomic write), so a restarted spoke can verify relays before its first heartbeat.WebhookRelayKeyFingerprint, never the key itself.api_github_webhook.goaccepts a delivery when either check passes:GITHUB_WEBHOOK_SECRET. This path is unchanged and has no new replay check.hmac.Equal. This follows the existingverifyGitHubWebhookSignature.relay_timestamp_stale.relay_delivery_missing.relay_delivery_replayed. The IDs live in a per-server in-memory set capped at 4096 entries. 10 minutes is twice the skew window, so it covers every timestamp that would still verify.relay_signature_invalid.RecordWebhookDeliveryFailure(), the same as a bad GitHub signature today.githubWebhooksConfigured()is now true when a relay key is present, so event-driven review dispatch turns on for hosted hives./api/status.webhookshas a newverificationfield:github-signed,relay (hub-signed)ornone.Tests
githubWebhooksConfigured()andverificationon.v6-specific additions
pkg/github:relay_signature_invalid,relay_timestamp_stale,relay_delivery_replayedandrelay_delivery_missing.RecordWebhookDeliveryFailure(reason), so they show up in/api/status.webhooks.mode_reason_codeandrejected_1hnext to the existing codes.no_secretis reported only when there is no secret and also no relay key. A hosted spoke holding a relay key no longer gets ano_secretdeep-health warning.webhookscheck (and so MCPhealth_deep_read, which wraps/api/health/deep) now includesverification.docs/webhooks.mdlists the new reason codes and theverificationfield.verification: relay (hub-signed)and nono_secret. A forged relay then setsmode_reason_codetorelay_signature_invalid, setsrejected_1hto 1, and produces a deep-health warning.Differences from the issue spec
handleHiveStatusremoves the relay key from the owner-facing record.meta.json). The hub needs the raw key to sign.meta.jsonnow carries a credential,saveSaaSHivecreates hive record directories0o700(was0o755), strips group/other bits from a pre-existing directory on every save without ever widening a read-only one, and writesmeta.json0o600(was0o644) β the same mode the other hub stores use. Anything outside the hub process that read hive records as another uid (backup sidecars, ad-hoc tooling) loses access after the first save; run such readers as the hub uid.Decision taken by hive: when a spoke has both
GITHUB_WEBHOOK_SECRETand a relay key, the GitHub signature is checked first, the relay signature is the fallback, andverificationreportsgithub-signed.Validation: only
gofmt -lwas run locally, per repo policy. CI is the verdict.β hive: agent=scanner backend=copilot model=claude-fable-5.1 copilot=1.0.88 requested_by=@Danathar