Skip to content

πŸ› fix(webhooks): a spoke without a webhook secret now says so in its status and deep health check - #17173

Merged
hivecommons-hive[bot] merged 3 commits into
hivecommons:v6from
Danathar:fix/webhook-no-secret-reason
Nov 8, 2026
Merged

hivecommons-hive[bot] merged 3 commits into
hivecommons:v6from
Danathar:fix/webhook-no-secret-reason

Conversation

@Danathar

@Danathar Danathar commented Nov 8, 2026 •

Copy link
Copy Markdown
Contributor

Summary

A hosted spoke that has no GITHUB_WEBHOOK_SECRET rejects every relayed GitHub delivery with 503 and stays on the polling cadence. Until now the only trace was one WARN line per restart, and /api/status.webhooks read healthy:false, mode:polling, events_1h:0 with no reason, the same as "GitHub isn't sending". This PR is diagnostics only: the webhook status now carries a machine-readable cause (mode_reason_code: no_secret) and counts rejected deliveries, and /api/health/deep gets a webhooks warning that says it in plain words. It doesn't change how deliveries are verified or how the hub provisions anything.

The root gap is on the hub side: the hub never provisions the secret to hosted spokes; tracked separately with the hub operator. This PR only makes the gap visible on the spoke.

What went wrong

  • pkg/dashboard/api_github_webhook.go answers 503 when the secret is unset, logs a WARN once per process (πŸ› fix: missing webhook secret is warned once, not per rejected requestΒ #13681) and Debug after that, and never calls RecordWebhookDeliveryFailure. Only bad signatures and undecodable payloads were recorded.
  • The webhook mode starts in polling and only moves after a verified delivery, so a spoke without a secret polls forever with an empty mode_reason.
  • On the live projectbluefin hosted hive, /api/status.webhooks shows {healthy:false, mode:"polling", events_1h:0, last_event_at:null} and /api/health/deep has no webhook check at all. Operators can't tell "no secret configured" from "GitHub isn't sending".

What changes

  • Status cause. /api/status.webhooks (and /api/gh-rate-limits) gain mode_reason_code while polling:

    • no_secret: GITHUB_WEBHOOK_SECRET is unset;
    • signature_failed: deliveries arrive but their signature does not verify;
    • deliveries_rejected: signed payloads did not decode;
    • no_deliveries: the secret is set but nothing verified is in the window.

    A verified delivery clears it. mode_reason keeps its existing plain-words values and now also explains the no-secret case.

  • Counters. rejected_1h and last_rejected_at cover every rejected delivery. The count is a fixed 60-bucket per-minute ring, so a flood of deliveries never allocates.

  • The 503 is recorded. It goes through RecordWebhookDeliveryFailure like the other rejections. That function now takes the reason. The WARN still logs once per process.

  • Health. /api/health/deep (and so health_deep_read) gets a webhooks check, and the dashboard health summary (navbar tooltip, heartbeat) gets the same entry. It warns, for example "polling β€” no GITHUB_WEBHOOK_SECRET on this spoke; relayed deliveries are rejected with 503 (N rejected in the last hour)". It warns only once deliveries are actually being turned away. A hive with no secret that nobody sends to just polls, and the check passes. The warning never changes the overall verdict and is never a fail, so it can't raise the hub's health-check-failing alert.

  • Secret safety. Only whether the secret is set is reported. Its value, length and any hash of it never reach status, logs or health output, and a test asserts the value is absent from all three outputs. The public /api/webhook/github 503 body already says "no secret configured", so the public health check reveals nothing new.

Reviewer notes

  • I used a new mode_reason_code field instead of putting codes into mode_reason, because mode_reason already carries prose ("deliveries failing verification") that tests and docs rely on.
  • The health summary (navbar tooltip, heartbeat) lists the webhooks check with its own status, but the check does not count toward the summary's warns, so a spoke's overall health verdict and header pill stay unchanged. The warning only fires while deliveries were rejected in the last hour.

Files

  • src/pkg/github/webhook_mode.go, webhook_cache.go: reason codes, WebhookSecretConfigured(), the rejection counter, and the annotation on the status snapshot.
  • src/pkg/dashboard/api_github_webhook.go: records the no-secret rejection and passes a reason for the other two.
  • src/pkg/dashboard/health_webhooks.go (new), server.go: the webhooks health check in both health paths.
  • Docs: webhooks.md (how to tell the cause from mode_reason_code, plus refreshed line cites that had drifted), env-vars.md, operator-reference/runtime-and-quota.md, and the /api/health/deep row in both API reference pages. Changelog fragment changelog.d/fixed-webhook-no-secret-reason.md.

Tests

New tests (each fails on v6 tip and passes here):

  • TestGitHubWebhookNoSecretReportsReasonAndCountsRejections: three no-secret POSTs give mode_reason_code: no_secret, rejected_1h: 3 and last_rejected_at.
  • TestHealthDeepWarnsWhenWebhooksRejectedForMissingSecret: the deep check passes before any delivery, warns after a 503 with the no-secret detail, the overall verdict stays ok, and the health summary carries the same warning.
  • TestWebhookReasonClearsAfterVerifiedDelivery: with the secret set, the code goes no_deliveries, then signature_failed after a bad signature, then clears after a verified delivery. The check passes, and the secret value appears in no output.
  • TestWebhookModeReasonCodes (pkg/github): code precedence, plain-words reasons, and rejected_1h aging out after an hour.

Ran go test ./pkg/github/ ./pkg/dashboard/ in full, plus go vet, the docs citation and link checks, and the API-reference citation check.

πŸ€– Generated with Claude Code

https://claude.ai/code/session_01BQYnPGFJx81c8JXR3o1sg1

…nd deep health

A hosted spoke with no GITHUB_WEBHOOK_SECRET answers every relayed
delivery with 503 and polls forever, but the only trace was one WARN per
process: /api/status.webhooks read healthy:false, mode:polling with no
reason, the same as "GitHub isn't sending".

- /api/status.webhooks gains mode_reason_code (no_secret,
  signature_failed, deliveries_rejected, no_deliveries), rejected_1h
  (fixed per-minute ring, no per-request allocation) and
  last_rejected_at; mode_reason explains the no-secret case in words.
- The no-secret 503 is recorded through RecordWebhookDeliveryFailure,
  which now takes the reason; the WARN stays once per process.
- /api/health/deep and the health summary get a `webhooks` check that
  warns (never fails, never changes the verdict) once deliveries are
  being turned away.
- Only whether the secret is set is ever reported.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BQYnPGFJx81c8JXR3o1sg1
Signed-off-by: Danathar <Danathar@users.noreply.github.com>
@kubestellar-prow kubestellar-prow Bot added the dco-signoff: yes Indicates the PR's author has signed the DCO. label Nov 8, 2026
@kubestellar-prow

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign hanthor for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@kubestellar-prow kubestellar-prow Bot added the size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. label Nov 8, 2026
@github-actions

github-actions Bot commented Nov 8, 2026 •

Copy link
Copy Markdown
Contributor

Open PR overlap check

Base branch: v6

Conflicts are checked with every PR merged onto the current tip of v6, so a PR that only needs a rebase is not reported as conflicting with its siblings.

PR Title Relation Files
#17130 ✨ feat(github): the hive files grouped issues from GitHub code scanning alerts (opt-in per repo) same files src/docs/api-reference/health-status-events.md
src/pkg/dashboard/server.go
#17025 🌱 Forward-merge v5 into v6 (1 commits) same files src/docs/env-vars.md

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Read webhook_mode.go, webhook_cache.go, api_github_webhook.go and server.go (handleHealthDeep, healthSummaryFor) at the v6 merge-base alongside the diff. The mechanism is sound and the hub-alert claim holds; two small things are worth a look before this lands.

Verified correct

  • annotateModeLocked is only called under t.mu (snapshot() locks at webhook_cache.go:391, observeMode/modeSnapshot lock themselves), and healthyLocked is already the lock-held helper, so no new race.
  • Recording a failure on the no-secret path cannot degrade a working cadence: a secret-less spoke never has a verified delivery, so modeInputLocked already reports polling; and webhookMinuteCounter is a fixed 60-bucket ring, so unauthenticated POSTs cannot grow memory (the signature-failed path already accepted the same traffic).
  • hub/alerts.go:66 raises health-check-failing only on a check reporting "fail"; this check emits at most "warn", so the comment in health_webhooks.go about the hub alert is accurate.
  • WebhookSecretConfigured exposes only a boolean; nothing about the secret value reaches status or logs.

Two low-severity findings

  1. health_webhooks.go:33 β€” the no-secret case warns on snap.LastRejectedAt != nil, but LastRejectedAt is copied from t.lastFailure (webhook_cache.go snapshot, this diff) which is never reset, and it isn't bounded by t.window the way the recentFailure branch of annotateModeLocked is. Consequence: one relayed delivery days ago leaves the spoke permanently at warn with the text … rejected with 503 (0 rejected in the last hour). Gating on snap.Rejected1h > 0 (or the same window test annotateModeLocked uses) would make the detail and the status agree.

  2. server.go:5347 β€” in healthSummaryFor, warns > 0 sets overall = "warning", and this diff increments warns for the webhook check. The PR body, health_webhooks.go:14, the handleHealthDeep comment and api-reference.md all say the warning "never changes the overall verdict"; that is true for /api/health/deep (which the test asserts) but not for the summary that feeds StatusPayload.DeepHealth β€” the dashboard header pill and api_chat.go:458 β€” which will read warning on every secret-less hosted spoke once a delivery arrives. Combined with (1) that is permanent. If turning the pill yellow is the intent, say so in the body and docs; if not, don't count this check in warns.

Neither blocks the design; both are a few lines. Everything else I read matches what the body describes.

β€” hive: agent=reviewer backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive

hivecommons-hive Bot commented Nov 8, 2026 •

Copy link
Copy Markdown
Contributor

@Danathar this PR is back in your court (stage/072-returned-to-contributor). The hive will not merge it until the head changes.

Review asks:

  • the review swarm requested changes

Reply here or push a new head to this branch (same sign-off); CI and the review swarm re-run automatically and the PR returns to the normal path.

Queued in your relay fix queue (2 waiting): fixes are handed out before new work as soon as you have capacity.

…e summary verdict

Two findings from the hive reviewer on hivecommons#17173:
- The no-secret warning keyed on LastRejectedAt, which never resets, so a
  single rejected delivery left the spoke at warn for good. It now follows
  Rejected1h, the same hourly window the other counters use.
- The health summary counted the webhook warning in warns, which turned the
  summary verdict (header pill, DeepHealth) to "warning". The PR and docs
  said it never changes the overall verdict. The check is still listed but
  no longer counted, so that is now true on both endpoints.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BQYnPGFJx81c8JXR3o1sg1
Signed-off-by: Danathar <Danathar@users.noreply.github.com>
@Danathar

Danathar commented Nov 8, 2026

Copy link
Copy Markdown
Contributor Author

Both findings are fixed in 5e992cf.

  1. The no-secret warning now follows Rejected1h instead of LastRejectedAt, so a rejection older than an hour no longer keeps the spoke at warn. New test: TestWebhookHealthNoSecretWarnsOnlyWithinTheHour.
  2. healthSummaryFor still lists the webhooks check but doesn't count it in warns, so the summary verdict (header pill, DeepHealth) no longer changes, which matches the docs. TestHealthDeepWarnsWhenWebhooksRejectedForMissingSecret now checks that too.

Both tests fail without the change. ./pkg/dashboard/ and ./pkg/github/ pass, and vet is clean. I updated the PR body to drop the amber-pill note.

@hivecommons-hive

hivecommons-hive Bot commented Nov 8, 2026 •

Copy link
Copy Markdown
Contributor

Hive status β€” updated in place; edits do not notify. Full history is in the Activity Monitor.

Reviewed by reviewer-2 β€” no findings.

Reviewer notes

Read against the tree at 5e992cf. Diagnostics-only as described; looks correct to me.

  • src/pkg/dashboard/api_github_webhook.go:63 and src/pkg/github/webhook_mode.go:86 both read strings.TrimSpace(os.Getenv("GITHUB_WEBHOOK_SECRET")) at call time, so the handler's 503 branch and WebhookSecretConfigured() can't disagree about whether a secret exists.
  • webhook_cache.go webhookMinuteCounter: the ring resets a bucket when its minute changes and total sums only minutes in (m-60, m] β€” correct hour window, no allocation. annotateModeLocked is called from snapshot(), which holds t.mu (webhook_cache.go:430-433).
  • RecordWebhookDeliveryFailure gained a parameter; the three call sites in api_github_webhook.go:78,84,105 are the only non-test callers in src/.
  • Body claim checked: the webhooks entry added to handleHealthDeep (server.go:3981-3990) does not touch failCount/overall, and in healthSummaryFor (server.go:5345-5350) it is appended without incrementing warns β€” so the overall verdict really is unchanged.

No findings. Noting only that WebhookSecretConfigured() is an os.Getenv on every status snapshot; cheap, and consistent with how the handler already reads it.

Evidence bundle `#171734970366` sha256:0196c8eb1a50a947c65335562f8799a596f12a20e872c873abca87eb4fd3d601 β€” download from the dashboard or `hivectl review evidence #17173`.

@hivecommons-hive
hivecommons-hive Bot merged commit 5f056fe into hivecommons:v6 Nov 8, 2026
68 of 69 checks passed
@hivecommons-hive

Copy link
Copy Markdown
Contributor

merged by Hive trusted-author auto-merge: author @Danathar holds role owner and GitHub permission maintain on repo; CI green at 4970366.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

dco-signoff: yes Indicates the PR's author has signed the DCO. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant