Skip to content

πŸ› dashboard: a hosted spoke's JSON 502 reaches the owner instead of the ingress page - #18601

Merged
kubestellar-prow[bot] merged 1 commit into
v5from
fix/hosted-ingress-502-passthrough
Nov 17, 2026
Merged

kubestellar-prow[bot] merged 1 commit into
v5from
fix/hosted-ingress-502-passthrough

Conversation

@clubanderson

Copy link
Copy Markdown
Member

Fixes #18598 (the masking half; the meetings logging half is #18600).

Problem

The hosted ingress sets `custom-http-errors: "502,503"` and serves the error-pages backend for those codes. It cannot see the upstream body, so a 502 the spoke answered (e.g. `quorum meeting.join: blocked: the bot has no permission to connect to that voice channel`) became `{"ok":false,"error":"hive unavailable","detail":"This hive is restarting..."}`. The ingress access log showed the spoke answering in tens of milliseconds before the error-pages backend replaced it. #17037 hit the same thing and was fixed per handler; there are ~95 `StatusBadGateway` sites in `pkg/dashboard`.

Change

`withHostedGatewayPassthrough` (new `hosted_gateway_status.go`), wired into both dashboard handler chains outside `authenticate`:

  • Applies only when the request carries the hub's `X-Hive-Proxy-Auth` proof (it came through the hosted ingress) and the path is `/api/`.
  • A handler's 502 with a JSON content type goes out as 424 Failed Dependency, which the ingress passes through with its body, plus `X-Hive-Status: 502`.
  • When the spoke is unreachable the ingress produces the status itself, so the generic page still shows. Self-hosted hives, non-API paths, non-JSON 502s and all other statuses are unchanged. 503 is deliberately left alone because the UI reads spoke 503s ("not enabled") by status.
  • The writer keeps `Flush`, `Hijack`, `Unwrap` and `ReadFrom` for streaming and websocket routes.

Note: `inbox-recorder.js` retries on 502; a spoke-answered 502 on a hosted hive now arrives as 424 and is not retried. The spoke did answer, so that matches what the failure is.

Tests

`hosted_gateway_status_test.go`: a hosted JSON 502 becomes 424 with the body and header kept; self-hosted, non-API, non-JSON, 503, 409 and implicit 200 are unchanged; the served `Handler()` chain never lets a hosted JSON 502 through; writer pass-throughs.

πŸ€– Generated with Claude Code

…he ingress page

The hosted ingress intercepts 502/503 and serves its generic 'hive
unavailable' page, so a 502 the spoke answered with its own error read as
an outage. On a request carrying the hub's X-Hive-Proxy-Auth proof, a JSON
502 from an /api/ handler now goes out as 424 (passed through by the
ingress) with X-Hive-Status: 502. Unreachable spokes still get the generic
page; 503 is unchanged because the UI reads spoke 503s by status.

Refs #18598

Signed-off-by: Andy Anderson <andy@clubanderson.com>
@kubestellar-prow kubestellar-prow Bot added dco-signoff: yes Indicates the PR's author has signed the DCO. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Nov 17, 2026
@github-actions

github-actions Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Open PR overlap check

βœ… Resolved: no conflicting open PRs or same-file overlaps remain against v5.

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Security perspective, head 639b145: one in-scope trust-boundary finding (inline).

The X-Hive-Proxy-Auth proof is only tested for presence (hosted_gateway_status.go:40), but the wrapper sits outside authenticate, which does the real secureCompare against s.authToken (server.go:2170-2172). The comment says this "grants nothing", but it does change what an unauthenticated caller can read. isPublicPath leaves /api/contribute, /api/gh-user-auth/*, /api/v1, /api/leaderboard and others open (server.go:2545+). Today a 502 from those handlers, which can carry internal upstream error text, is replaced by the ingress's generic page. Any anonymous client who adds X-Hive-Proxy-Auth: x gets the 424 and the full body instead.

Fix: validate the proof the same way authenticate does.

β€” hive: agent=adjudicator-4 backend=claude model=claude-sonnet-5-5 claude=2.1.325

// to an API route. The proof header's presence is enough here: this only
// picks the status an error is sent under, it grants nothing.
func isHostedGatewayRequest(r *http.Request) bool {
return r.Header.Get(proxyAuthHeader) != "" && strings.HasPrefix(r.URL.Path, "/api/")

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Presence-only check lets any anonymous caller send a junk X-Hive-Proxy-Auth and unmask 502 bodies on public /api/ routes (contribute, gh-user-auth, v1...), which the ingress used to hide. Validate the proof like authenticate does (server.go:2170):

Suggested change
return r.Header.Get(proxyAuthHeader) != "" && strings.HasPrefix(r.URL.Path, "/api/")
func (s *Server) isHostedGatewayRequest(r *http.Request) bool {
proof := r.Header.Get(proxyAuthHeader)
return proof != "" && s.authToken != "" && secureCompare(proof, s.authToken) && strings.HasPrefix(r.URL.Path, "/api/")
}

and call s.isHostedGatewayRequest(r) at line 46. The test must then set the proof to the server's auth token.

@clubanderson clubanderson added lgtm Indicates that a PR is ready to be merged. approved Indicates a PR has been approved by an approver from all required OWNERS files. labels Nov 17, 2026
@kubestellar-prow

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

Approval requirements bypassed by manually added approval.

This pull-request has been approved by:

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@hivecommons-hive

Copy link
Copy Markdown
Contributor

Hive status β€” updated in place; edits do not notify. Full history is in the Activity Monitor.

Review findings from adjudicator-3 (folded here because this head already carries its hive review):

Correctness review at head 639b145 (diff plus jsonError in api.go, withServerTiming in server.go, and the retry list in static-src/js/inbox-recorder.js).

The rewrite mechanism is sound. jsonError sets Content-Type before WriteHeader, so the JSON check in hostedGatewayStatusWriter.WriteHeader sees it. Flush, Hijack, Unwrap and ReadFrom are kept. The wrapper only applies when X-Hive-Proxy-Auth is present and the path is /api/.

1. In-scope (medium): the 502β†’424 rewrite breaks client retry of transient spoke 502s.

  • src/pkg/dashboard/static-src/js/inbox-recorder.js has INBOX_REC_RETRY_STATUSES = [0, 408, 429, 500, 502, 503, 504], and statuses outside that list stop the recording.
  • On a hosted hive, a transient JSON 502 from an upload handler now arrives as 424, which is not in the list. A retryable hiccup then aborts the recording.
  • The PR body acknowledges the change but treats it as acceptable. It is not: the same 502 had been retried before this PR.
  • Requested fix: add 424 to that list, or have the client treat X-Hive-Status: 502 as 502. The first option is the smaller change:
    const INBOX_REC_RETRY_STATUSES = [0, 408, 424, 429, 500, 502, 503, 504];
  • Please also check other dashboard JS that branches on status === 502 or >= 500. 424 is a 4xx, so those paths now classify a spoke error as a client error. I could not exhaustively search the static JS here.

2. In-scope (low): the wiring test does not prove the rewrite happens in the served chain.

  • TestHostedGatewayPassthroughWiredAndStreams sends a bogus proof ("proof") through s.Handler() and only asserts the status is not 502.
  • If authenticate rejects the proof with 401/403 first, the test passes whether or not withHostedGatewayPassthrough is wired. Its own comment concedes this.
  • Requested fix: send the real s.authToken as the proof, plus the owner identity headers other tests in this package use (e.g. dashboard_owner_authz_test.go). Then assert 424 and X-Hive-Status: 502.

3. Out-of-scope (info): the issue's logging half is not in this PR.

@kubestellar-prow
kubestellar-prow Bot merged commit 735ae86 into v5 Nov 17, 2026
105 checks passed
@kubestellar-prow
kubestellar-prow Bot deleted the fix/hosted-ingress-502-passthrough branch November 17, 2026 03:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. dco-signoff: yes Indicates the PR's author has signed the DCO. lgtm Indicates that a PR is ready to be merged. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

πŸ› Discord meeting join failures reach the owner as "hive unavailable" and leave no log line

1 participant