Repository navigation
π fix(webhooks): a spoke without a webhook secret now says so in its status and deep health check - #17173
Conversation
β¦nd deep health A hosted spoke with no GITHUB_WEBHOOK_SECRET answers every relayed delivery with 503 and polls forever, but the only trace was one WARN per process: /api/status.webhooks read healthy:false, mode:polling with no reason, the same as "GitHub isn't sending". - /api/status.webhooks gains mode_reason_code (no_secret, signature_failed, deliveries_rejected, no_deliveries), rejected_1h (fixed per-minute ring, no per-request allocation) and last_rejected_at; mode_reason explains the no-secret case in words. - The no-secret 503 is recorded through RecordWebhookDeliveryFailure, which now takes the reason; the WARN stays once per process. - /api/health/deep and the health summary get a `webhooks` check that warns (never fails, never changes the verdict) once deliveries are being turned away. - Only whether the secret is set is ever reported. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BQYnPGFJx81c8JXR3o1sg1 Signed-off-by: Danathar <Danathar@users.noreply.github.com>
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
Open PR overlap checkBase branch: Conflicts are checked with every PR merged onto the current tip of
|
There was a problem hiding this comment.
Read webhook_mode.go, webhook_cache.go, api_github_webhook.go and server.go (handleHealthDeep, healthSummaryFor) at the v6 merge-base alongside the diff. The mechanism is sound and the hub-alert claim holds; two small things are worth a look before this lands.
Verified correct
annotateModeLockedis only called undert.mu(snapshot()locks atwebhook_cache.go:391,observeMode/modeSnapshotlock themselves), andhealthyLockedis already the lock-held helper, so no new race.- Recording a failure on the no-secret path cannot degrade a working cadence: a secret-less spoke never has a verified delivery, so
modeInputLockedalready reports polling; andwebhookMinuteCounteris a fixed 60-bucket ring, so unauthenticated POSTs cannot grow memory (the signature-failed path already accepted the same traffic). hub/alerts.go:66raiseshealth-check-failingonly on a check reporting"fail"; this check emits at most"warn", so the comment inhealth_webhooks.goabout the hub alert is accurate.WebhookSecretConfiguredexposes only a boolean; nothing about the secret value reaches status or logs.
Two low-severity findings
-
health_webhooks.go:33β the no-secret case warns onsnap.LastRejectedAt != nil, butLastRejectedAtis copied fromt.lastFailure(webhook_cache.gosnapshot, this diff) which is never reset, and it isn't bounded byt.windowthe way therecentFailurebranch ofannotateModeLockedis. Consequence: one relayed delivery days ago leaves the spoke permanently atwarnwith the textβ¦ rejected with 503 (0 rejected in the last hour). Gating onsnap.Rejected1h > 0(or the same window testannotateModeLockeduses) would make the detail and the status agree. -
server.go:5347β inhealthSummaryFor,warns > 0setsoverall = "warning", and this diff incrementswarnsfor the webhook check. The PR body,health_webhooks.go:14, thehandleHealthDeepcomment andapi-reference.mdall say the warning "never changes the overall verdict"; that is true for/api/health/deep(which the test asserts) but not for the summary that feedsStatusPayload.DeepHealthβ the dashboard header pill andapi_chat.go:458β which will readwarningon every secret-less hosted spoke once a delivery arrives. Combined with (1) that is permanent. If turning the pill yellow is the intent, say so in the body and docs; if not, don't count this check inwarns.
Neither blocks the design; both are a few lines. Everything else I read matches what the body describes.
β hive: agent=reviewer backend=copilot model=claude-fable-5.1 copilot=1.0.88
|
@Danathar this PR is back in your court ( Review asks:
Reply here or push a new head to this branch (same sign-off); CI and the review swarm re-run automatically and the PR returns to the normal path. Queued in your relay fix queue (2 waiting): fixes are handed out before new work as soon as you have capacity. |
β¦e summary verdict Two findings from the hive reviewer on hivecommons#17173: - The no-secret warning keyed on LastRejectedAt, which never resets, so a single rejected delivery left the spoke at warn for good. It now follows Rejected1h, the same hourly window the other counters use. - The health summary counted the webhook warning in warns, which turned the summary verdict (header pill, DeepHealth) to "warning". The PR and docs said it never changes the overall verdict. The check is still listed but no longer counted, so that is now true on both endpoints. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BQYnPGFJx81c8JXR3o1sg1 Signed-off-by: Danathar <Danathar@users.noreply.github.com>
|
Both findings are fixed in 5e992cf.
Both tests fail without the change. |
|
Hive status β updated in place; edits do not notify. Full history is in the Activity Monitor. Reviewed by Reviewer notesRead against the tree at
No findings. Noting only that |
Summary
A hosted spoke that has no
GITHUB_WEBHOOK_SECRETrejects every relayed GitHub delivery with 503 and stays on the polling cadence. Until now the only trace was one WARN line per restart, and/api/status.webhooksreadhealthy:false, mode:polling, events_1h:0with no reason, the same as "GitHub isn't sending". This PR is diagnostics only: the webhook status now carries a machine-readable cause (mode_reason_code: no_secret) and counts rejected deliveries, and/api/health/deepgets awebhookswarning that says it in plain words. It doesn't change how deliveries are verified or how the hub provisions anything.The root gap is on the hub side: the hub never provisions the secret to hosted spokes; tracked separately with the hub operator. This PR only makes the gap visible on the spoke.
What went wrong
pkg/dashboard/api_github_webhook.goanswers 503 when the secret is unset, logs a WARN once per process (π fix: missing webhook secret is warned once, not per rejected requestΒ #13681) and Debug after that, and never callsRecordWebhookDeliveryFailure. Only bad signatures and undecodable payloads were recorded.mode_reason./api/status.webhooksshows{healthy:false, mode:"polling", events_1h:0, last_event_at:null}and/api/health/deephas no webhook check at all. Operators can't tell "no secret configured" from "GitHub isn't sending".What changes
Status cause.
/api/status.webhooks(and/api/gh-rate-limits) gainmode_reason_codewhile polling:no_secret:GITHUB_WEBHOOK_SECRETis unset;signature_failed: deliveries arrive but their signature does not verify;deliveries_rejected: signed payloads did not decode;no_deliveries: the secret is set but nothing verified is in the window.A verified delivery clears it.
mode_reasonkeeps its existing plain-words values and now also explains the no-secret case.Counters.
rejected_1handlast_rejected_atcover every rejected delivery. The count is a fixed 60-bucket per-minute ring, so a flood of deliveries never allocates.The 503 is recorded. It goes through
RecordWebhookDeliveryFailurelike the other rejections. That function now takes the reason. The WARN still logs once per process.Health.
/api/health/deep(and sohealth_deep_read) gets awebhookscheck, and the dashboard health summary (navbar tooltip, heartbeat) gets the same entry. It warns, for example "polling β no GITHUB_WEBHOOK_SECRET on this spoke; relayed deliveries are rejected with 503 (N rejected in the last hour)". It warns only once deliveries are actually being turned away. A hive with no secret that nobody sends to just polls, and the check passes. The warning never changes the overall verdict and is never afail, so it can't raise the hub's health-check-failing alert.Secret safety. Only whether the secret is set is reported. Its value, length and any hash of it never reach status, logs or health output, and a test asserts the value is absent from all three outputs. The public
/api/webhook/github503 body already says "no secret configured", so the public health check reveals nothing new.Reviewer notes
mode_reason_codefield instead of putting codes intomode_reason, becausemode_reasonalready carries prose ("deliveries failing verification") that tests and docs rely on.webhookscheck with its own status, but the check does not count toward the summary'swarns, so a spoke's overall health verdict and header pill stay unchanged. The warning only fires while deliveries were rejected in the last hour.Files
src/pkg/github/webhook_mode.go,webhook_cache.go: reason codes,WebhookSecretConfigured(), the rejection counter, and the annotation on the status snapshot.src/pkg/dashboard/api_github_webhook.go: records the no-secret rejection and passes a reason for the other two.src/pkg/dashboard/health_webhooks.go(new),server.go: thewebhookshealth check in both health paths.webhooks.md(how to tell the cause frommode_reason_code, plus refreshed line cites that had drifted),env-vars.md,operator-reference/runtime-and-quota.md, and the/api/health/deeprow in both API reference pages. Changelog fragmentchangelog.d/fixed-webhook-no-secret-reason.md.Tests
New tests (each fails on v6 tip and passes here):
TestGitHubWebhookNoSecretReportsReasonAndCountsRejections: three no-secret POSTs givemode_reason_code: no_secret,rejected_1h: 3andlast_rejected_at.TestHealthDeepWarnsWhenWebhooksRejectedForMissingSecret: the deep check passes before any delivery, warns after a 503 with the no-secret detail, the overall verdict staysok, and the health summary carries the same warning.TestWebhookReasonClearsAfterVerifiedDelivery: with the secret set, the code goesno_deliveries, thensignature_failedafter a bad signature, then clears after a verified delivery. The check passes, and the secret value appears in no output.TestWebhookModeReasonCodes(pkg/github): code precedence, plain-words reasons, andrejected_1haging out after an hour.Ran
go test ./pkg/github/ ./pkg/dashboard/in full, plusgo vet, the docs citation and link checks, and the API-reference citation check.π€ Generated with Claude Code
https://claude.ai/code/session_01BQYnPGFJx81c8JXR3o1sg1