Repository navigation
Description
Target: v6.
On hold: @andy2578 wants time to think through the design. Agents, please don't work on this.
The problem in one paragraph
Hosted hives never get GitHub webhooks. GitHub sends each webhook to the hub, and the hub passes it on to the right hive. But a hive will only accept a webhook if it has the webhook secret (GITHUB_WEBHOOK_SECRET), and the hub never gives hosted hives that secret. So every hive turns every webhook away (HTTP 503), and falls back to checking GitHub on a timer ("polling") instead.
Why it matters
Polling uses a lot more GitHub API calls than webhooks do. The projectbluefin hive (knuckle) keeps hitting its limit of 7,000 calls an hour. Working webhooks would save it roughly 500–1,000 calls an hour. That is an estimate, not a measurement.
How we know
- Knuckle has turned away every single webhook since at least Oct 31: zero accepted, ever. Its status says
webhooks: polling, 0 events in the last hour. - kubestellar-console showed the same thing earlier ([selfreport] unclassified_error_repeated: github webhook rejected: no webhook secret configured (set GITHUB_WEBHOOK_SECRE… #13921), and so did knuckle in [selfreport] unclassified_error_repeated: github webhook rejected: no webhook secret configured (set GITHUB_WEBHOOK_SECRE… #13666. Both got marked as done when 🐛 fix: missing webhook secret is warned once, not per rejected request #13681 merged, but that change only made the warning quieter. The cause was never fixed.
- Nobody noticed, because a hive only logs one quiet line about it per restart.
Why the obvious fix isn't safe
The quick fix would be to give every hosted hive the hub's own webhook secret. But then every hive, including ones run by other people, would hold a secret the hub trusts, and could use it to send the hub fake webhooks.
Options
- Give each hive the hub's secret. Simple, but unsafe for the reason above. Not recommended.
- The hub signs forwarded webhooks with a separate key for each hive. Each hive only trusts its own key, so no hive can impersonate GitHub to the hub. This needs changes in the hub (forwarding and setting up new hives) and a way to give existing hives their key.
- Hosted hives stay on polling, by design. Write that down, and have the hub stop forwarding webhooks to them, since every forward currently fails.
Related
- 🐛 fix(webhooks): a spoke without a webhook secret now says so in its status and deep health check #17173 makes the problem visible: a hive's status page says "no webhook secret, using polling" instead of staying silent. It doesn't change how webhooks are delivered.
- Other API-budget work: 🐛 fix(github): routine since= reads get free 304s within the hour, and the hive names the sweep behind each REST call #17101, 🐛 fix(github): the ETag cache keeps the hive's repeat reads, so they come back as free 304s instead of being charged #17172, 🐛 fix(github): the un-park sweep reuses comment and timeline reads of items whose updated_at has not moved #17174 and 🐛 fix(dashboard): the metrics collector reads each repo's GitHub metadata at most once an hour instead of every five minutes #17175.
Technical details (code paths on v6)
-
GitHub delivers to the hub:
POST /api/github/webhook(pkg/hub/server.go:1756). -
The hub checks the signature with its one App webhook secret (
pkg/hub/webhook.go:74-91,loadWebhookSecretat:502). -
For PR, check, issue and push events,
relayWebhookToSpoke(webhook.go:96-99,:135) forwards the body unchanged, with the original signature, to<dashboard>/api/webhook/github. It picks the hive by repository name. -
The hive returns 503 when
GITHUB_WEBHOOK_SECRETis unset (pkg/dashboard/api_github_webhook.go:63-75). -
Nothing sets that variable for hosted hives:
- the
hive-secretsSecret in provisioning (pkg/hub/saas_provision.go:3293-3317); - the hive's env block (
:3524onward); pushGitHubConfigToSpoke(webhook.go:328);desiredPerHiveEnv(perhive_env_reconcile.go:111).
Yet
src/docs/env-vars.md:105says a hive needs it. - the
-
Upgrades only change the image (
kubectl set image,saas_upgrade.go:801). Existing hives would therefore get a new key only through the env reconcile, or by being set up again. -
The hub should be logging
spoke rejected webhook relay … status 503(webhook.go:209) for every delivery to every hosted hive.
Out of scope: the GitHub App's webhook URL and event list; self-hosted hives, which set their own secret.
Activity
[-][v6] Hosted spokes never receive GitHub webhooks: GITHUB_WEBHOOK_SECRET is never provisioned (relay design decision)[/-][+][v6] Hosted hives never get GitHub webhooks (the hub never gives them the webhook secret) — design decision[/+]📖 docs: hosted hive webhooks via the hub relay key (#730)
clubanderson commentedon Nov 16, 2026
Hi @Danathar, option 2 from this issue has shipped: the hub signs each forwarded webhook with a per-hive relay key, so hosted hives no longer need their own GitHub secret. It landed in #17914 (v5), #17915 (v6) and #18024 (owner-facing setup in Settings).
Would you try it on one of your hives?
webhooksblock in/api/statuson your hive:verificationshould readrelay (hub-signed)and show a recent last event. On v6 the same fields are also in/api/health/deep.If you see
503 webhooks disabled: no secret configured, the guide's troubleshooting table covers it. If anything misbehaves, please comment here or open a new issue.Docs: GitHub webhooks and the hub relay key (synced into the docs site by hivecommons/docs#730), plus the source at src/docs/webhooks.md.
Closing as completed: #17914, #17915, #18024 and hivecommons/docs#730 cover the design decision, the code and the docs. Reopen or file a new issue if the trial turns up problems.
github-actions commentedon Nov 16, 2026
@Danathar this issue was closed by @clubanderson as completed.
If this was closed by a merged PR, the fix only takes effect once it ships in a tagged release and your deployment updates to it — check your running version against the fix before concluding the fix failed.
If you are still seeing the problem, comment
/reopenand it will be reopened for you — you do not need write access. Please add what you are still observing (running version, logs, steps) so it is not closed the same way again.