Skip to content

πŸ› fix(dashboard): Let hive decide preview says why it failed instead of "hive unavailable" - #16782

Merged
clubanderson merged 1 commit into
v5from
fix/decide-plan-unavailable
Nov 5, 2026
Merged

clubanderson merged 1 commit into
v5from
fix/decide-plan-unavailable

Conversation

@clubanderson

Copy link
Copy Markdown
Member

Problem

On the hosted dashboard, Let hive decide on a PR row opened the preview and immediately showed Could not preview #16776 (hive unavailable); Retry failed the same way. Seen on #16736 earlier too, several times a day.

Root cause (verified live on hosted-available-oke-11-placeholder-r05x)

  1. handleRepoItemDecideDelegatedPlan answered every planning failure as 502 (src/pkg/dashboard/api_repo_item_decide_delegated_plan.go, jsonError(w, err.Error(), http.StatusBadGateway)), including non-transport ones: hivecommons/hive#16736 is not open (merged after the row loaded) and reading checks for …#16776: GET …/check-runs?per_page=100: 500 [] (api.github.com's intermittent 500 on a 70-check head β€” reproduced 1-in-3 with gh api; the spoke's logs show the same 500s on check-runs and commit status for that head at the time).
  2. The hosted hive-api-xhr Ingress has custom-http-errors: "401,502,503" (src/pkg/hub/saas_provision.go), so ingress-nginx intercepted the hive's 502 and replaced its JSON body with src/deploy/k8s/error-pages.yaml's {"ok":false,"error":"hive unavailable",…}. The ingress access log shows upstream 502 with ~4 s upstream time on each click β€” the hive answered; the gateway discarded its answer. Same trap as πŸ› bug: lack of SSO looks like crashed applicationΒ #16652's SSO 503.
  3. The plan is ~8 sequential GitHub reads with no retry on 5xx, and nothing was logged server-side, so the failure left no trace.
  4. The dialog printed the error string bare, with no hive, cause or next step.

Calling the endpoint on the pod directly (port-forward + proxy proof) returned 200 for #16776 and 502 {"error":"hivecommons/hive#16736 is not open"} for #16736 β€” the hive was up throughout.

Fix

  • Retry transient GitHub failures (github.IsRetryableError: 5xx, 429, rate limit, timeout) twice with 750 ms β†’ 1.5 s backoff before giving up.
  • Never answer 502/503 from the plan: not-open β†’ 409, GitHub read failed / no GitHub client / plan failed β†’ 424, with a structured body {ok:false, error, reason, retryable, attempts, elapsed_ms, hive, hint}.
  • Warn log on every failed plan with the real error, reason, attempts and duration.
  • Dialog: names the hive that answered, explains the failure (merged/closed since the row loaded β†’ refresh, no Retry; GitHub failed after N attempts over Xs β†’ Retry; no GitHub client; gateway 502 β†’ restarting, retry shortly; network), and always states that Let hive do that stays off until a preview loads β€” the hive never records a decision it could not preview. In-app only; no native dialogs.

Tests

  • TestDelegatedPlanRetriesTransientGitHubFailure β€” one check-runs 500 is retried to a 200; past the budget β†’ 424 reason: github, attempts: 3, retryable.
  • TestDelegatedPlanNotOpenIsConflictNotBadGateway β€” closed item β†’ 409 reason: not-open.
  • TestDelegatedPlanFailureStatusesAreNeverIntercepted β€” no classification yields 502/503.
  • TestStageTrackerDelegatePreviewSkeletonPrefetch β€” the gateway / not-open / github / network renderings and the Retry rule.

Refs #16776.

…of "hive unavailable"

The preview answered every failure as 502 β€” a PR merged since the row
loaded, an intermittent GitHub 500 on check-runs β€” and the hosted
Ingress intercepts 502/503 and swaps the body for its generic outage
page, so the owner read "hive unavailable" for a hive that was up.

- retry transient GitHub failures twice with backoff before giving up
- answer not-open as 409 and unrecoverable reads as 424, never 502/503,
  with {reason, retryable, attempts, elapsed_ms, hive, hint}
- log a Warn with the real error on every failed plan
- the dialog names the hive, explains the failure (merged since the row
  loaded / GitHub failed after N attempts / no GitHub client / gateway
  502), hides Retry when it cannot help, and says Let hive do that
  stays off until a preview loads

Refs #16776

Signed-off-by: Andrew Anderson <andy@clubanderson.com>
@kubestellar-prow kubestellar-prow Bot added the dco-signoff: yes Indicates the PR's author has signed the DCO. label Nov 5, 2026
@kubestellar-prow

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign kellyaa for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@kubestellar-prow kubestellar-prow Bot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Nov 5, 2026
@github-actions

github-actions Bot commented Nov 5, 2026

Copy link
Copy Markdown
Contributor

Open PR overlap check

Base branch: v5

Conflicts are checked with every PR merged onto the current tip of v5, so a PR that only needs a rebase is not reported as conflicting with its siblings.

PR Title Relation Files
#16769 πŸ› fix: keep navigation label visible on phones same files src/pkg/dashboard/static/index.html

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks correct to me. I read the handler, the classifier and the helpers it leans on at the PR head rather than the diff alone:

  • api_repo_item_decide_delegated_plan.go:371 returns gatherDelegatedInput's error unwrapped, so errors.As(err, &notOpen) in delegatedPlanFailureFor reaches the *delegatedItemNotOpenError built at api_repo_item_decide_delegated.go:637 β†’ 409, matching TestDelegatedPlanNotOpenIsConflictNotBadGateway.
  • pkg/github/github_retryable.go:23-45 (IsRetryableError) uses errors.Is/As, so the %w-wrapped check-runs 500 is classified retryable and the loop in planDelegatedDecisionWithRetry stops at delegatedPlanRetries (2 extra attempts, 750msβ†’1.5s). On ctx cancellation delegatedPlanRetrySleep returns ctx.Err() and the loop exits with the original error β€” no spin, no lost cause.
  • The non-502/503 choice is justified by what the tree says: api_auth.go:312 and api_contribute.go:1190 already document the hosted Ingress swallowing those statuses, and the new delegatedPlanFailure.status is only ever 409 or 424.
  • tracker-actions.js stageTrackerDelegatePlanFailure treats a 502–504 with no reason (gateway HTML β†’ r.json() fails β†’ {}) as gateway, and stageTrackerDelegatePlanRetryable withholds Retry for not-open/config, consistent with Retryable:false on the Go side.

The body's four root-cause claims each correspond to a concrete change in the diff; no gap between narrative and code.

β€” hive: agent=reviewer-2 backend=copilot model=claude-fable-5.1 copilot=1.0.88

@clubanderson
clubanderson merged commit b1d367f into v5 Nov 5, 2026
87 of 91 checks passed
@hivecommons-hive

Copy link
Copy Markdown
Contributor

Hive status β€” updated in place; edits do not notify. Full history is in the Activity Monitor.

Review findings from reviewer (folded here because this head already carries its hive review):

Read the Go and JS changes against the tree at 5e9fd1e and the callers before the body. Looks correct to me; the body's claims match the diff.

What I verified

  • planDelegatedDecisionWithRetry (api_repo_item_decide_delegated_plan.go:179-199): retries only when ghpkg.IsRetryableError is true (pkg/github/github_retryable.go:23-45: deadline, net.Error, rate-limit, 429, 5xx), stops after delegatedPlanRetries (so attempts maxes at 3), and returns the original error β€” not ctx.Err() β€” when the sleep is cancelled, so the classifier still sees the GitHub failure. Correct.
  • delegatedItemNotOpenError.Error() (:208) returns the same "<ref> is not open" string as the fmt.Errorf it replaces, so the execute path (decideDelegatedOne, api_repo_item_decide_delegated.go:496-499) audits the identical text; only the plan handler changes behaviour. Good β€” no silent change to the audit trail.
  • delegatedPlanFailureFor classification order (:265-290): errors.As not-open β†’ errors.Is no-client β†’ retryable β†’ any *gh.ErrorResponse β†’ default. Statuses are 409/424 only, and TestDelegatedPlanFailureStatusesAreNeverIntercepted pins that against the Ingress custom-http-errors: "401,502,503" trap. The jsonStatusResponse/publicHiveName helpers it uses exist (api.go:684, social_cards.go:460).
  • JS stageTrackerDelegatePlanFailure treats a bodiless 502–504 as gateway/retryable and stageTrackerDelegatePlanRetryable honours server retryable:false and never offers Retry for not-open/config. Consistent with the server contract.
  • The three Go tests named in the body exist in the diff; CI is green with only tide pending.

One info-level note (not blocking)

The comment in tracker-actions.js (stageTrackerDelegatePlanErrorContent doc, "#16776: that message was a merged PR once…") and the body's Refs #16776 point at the stream-watchdog PR you happened to click on, not at an issue tracking this bug. A future reader following that link lands on an unrelated ffmpeg change. If there is no tracking issue, consider dropping the ref from the code comment so it does not read as a tracker.

@kubestellar-prow
kubestellar-prow Bot deleted the fix/decide-plan-unavailable branch November 5, 2026 16:38
@github-actions

github-actions Bot commented Nov 5, 2026

Copy link
Copy Markdown
Contributor

Thank you for your contribution! Your PR has been merged.

We'd love to hear how your experience was: share feedback

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

dco-signoff: yes Indicates the PR's author has signed the DCO. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant