Skip to content

[v6] Hosted hives never get GitHub webhooks (the hub never gives them the webhook secret) — design decision #17182

Closed

Description

@Danathar

Target: v6.

On hold: @andy2578 wants time to think through the design. Agents, please don't work on this.

The problem in one paragraph

Hosted hives never get GitHub webhooks. GitHub sends each webhook to the hub, and the hub passes it on to the right hive. But a hive will only accept a webhook if it has the webhook secret (GITHUB_WEBHOOK_SECRET), and the hub never gives hosted hives that secret. So every hive turns every webhook away (HTTP 503), and falls back to checking GitHub on a timer ("polling") instead.

Why it matters

Polling uses a lot more GitHub API calls than webhooks do. The projectbluefin hive (knuckle) keeps hitting its limit of 7,000 calls an hour. Working webhooks would save it roughly 500–1,000 calls an hour. That is an estimate, not a measurement.

How we know

Why the obvious fix isn't safe

The quick fix would be to give every hosted hive the hub's own webhook secret. But then every hive, including ones run by other people, would hold a secret the hub trusts, and could use it to send the hub fake webhooks.

Options

  1. Give each hive the hub's secret. Simple, but unsafe for the reason above. Not recommended.
  2. The hub signs forwarded webhooks with a separate key for each hive. Each hive only trusts its own key, so no hive can impersonate GitHub to the hub. This needs changes in the hub (forwarding and setting up new hives) and a way to give existing hives their key.
  3. Hosted hives stay on polling, by design. Write that down, and have the hub stop forwarding webhooks to them, since every forward currently fails.

Related

Technical details (code paths on v6)
  1. GitHub delivers to the hub: POST /api/github/webhook (pkg/hub/server.go:1756).

  2. The hub checks the signature with its one App webhook secret (pkg/hub/webhook.go:74-91, loadWebhookSecret at :502).

  3. For PR, check, issue and push events, relayWebhookToSpoke (webhook.go:96-99, :135) forwards the body unchanged, with the original signature, to <dashboard>/api/webhook/github. It picks the hive by repository name.

  4. The hive returns 503 when GITHUB_WEBHOOK_SECRET is unset (pkg/dashboard/api_github_webhook.go:63-75).

  5. Nothing sets that variable for hosted hives:

    • the hive-secrets Secret in provisioning (pkg/hub/saas_provision.go:3293-3317);
    • the hive's env block (:3524 onward);
    • pushGitHubConfigToSpoke (webhook.go:328);
    • desiredPerHiveEnv (perhive_env_reconcile.go:111).

    Yet src/docs/env-vars.md:105 says a hive needs it.

  6. Upgrades only change the image (kubectl set image, saas_upgrade.go:801). Existing hives would therefore get a new key only through the env reconcile, or by being set up again.

  7. The hub should be logging spoke rejected webhook relay … status 503 (webhook.go:209) for every delivery to every hosted hive.

Out of scope: the GitHub App's webhook URL and event list; self-hosted hives, which set their own secret.

Activity

  1. added
    help wantedDenotes an issue that needs help from a contributor. Must meet "help wanted" guidelines.
    on Nov 8, 2026
  2. changed the title [-][v6] Hosted spokes never receive GitHub webhooks: GITHUB_WEBHOOK_SECRET is never provisioned (relay design decision)[/-] [+][v6] Hosted hives never get GitHub webhooks (the hub never gives them the webhook secret) — design decision[/+] on Nov 8, 2026
  3. clubanderson commented on Nov 16, 2026

    @clubanderson
    Member

    Hi @Danathar, option 2 from this issue has shipped: the hub signs each forwarded webhook with a per-hive relay key, so hosted hives no longer need their own GitHub secret. It landed in #17914 (v5), #17915 (v6) and #18024 (owner-facing setup in Settings).

    Would you try it on one of your hives?

    1. Open Settings → Forge App → GitHub webhooks and generate the webhook relay key.
    2. Point the GitHub App (or a repo/org webhook) at the hub as the guide describes.
    3. Check the webhooks block in /api/status on your hive: verification should read relay (hub-signed) and show a recent last event. On v6 the same fields are also in /api/health/deep.

    If you see 503 webhooks disabled: no secret configured, the guide's troubleshooting table covers it. If anything misbehaves, please comment here or open a new issue.

    Docs: GitHub webhooks and the hub relay key (synced into the docs site by hivecommons/docs#730), plus the source at src/docs/webhooks.md.

    Closing as completed: #17914, #17915, #18024 and hivecommons/docs#730 cover the design decision, the code and the docs. Reopen or file a new issue if the trial turns up problems.

  4. github-actions commented on Nov 16, 2026

    @github-actions
    Contributor

    @Danathar this issue was closed by @clubanderson as completed.

    If this was closed by a merged PR, the fix only takes effect once it ships in a tagged release and your deployment updates to it — check your running version against the fix before concluding the fix failed.

    If you are still seeing the problem, comment /reopen and it will be reopened for you — you do not need write access. Please add what you are still observing (running version, logs, steps) so it is not closed the same way again.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    help wantedDenotes an issue that needs help from a contributor. Must meet "help wanted" guidelines.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions