Skip to content

πŸ› fix: stop reporting GitHub 502/503/504 fetch failures as repeated runtime errors - #15366

Merged
hivecommons-hive[bot] merged 2 commits into
v5from
scanner/fix-15365
Oct 29, 2026
Merged

hivecommons-hive[bot] merged 2 commits into
v5from
scanner/fix-15365

Conversation

@hivecommons-hive

Copy link
Copy Markdown
Contributor

Closes #15365

A GitHub-side 503 on fetch-pr in the self-authored automerge sweep is transient and retried on the next sweep tick; it is not a hive defect. The slog feed was counting the repeated WARN as unclassified_error_repeated, exactly like the rate-limit storm in #15106 (the message names the caller, so each site would file its own issue). This applies the same exclusion for gateway-class statuses (502/503/504); the WARN stays in local logs. 500 deliberately stays reportable, per the existing rate-limit storm test.

β€” hive: agent=scanner backend=copilot model=claude-fable-5.1 copilot=1.0.88

…untime errors

A GitHub-side 503 on fetch-pr is transient and retried on the next sweep tick; the slog feed counted it as unclassified_error_repeated like the rate-limit storm in #15106. Apply the same exclusion for gateway-class statuses; 500 stays reportable.

Fixes #15365

Signed-off-by: scanner <280983584+hivecommons-hive[bot]@users.noreply.github.com>
@kubestellar-prow kubestellar-prow Bot added the dco-signoff: yes Indicates the PR's author has signed the DCO. label Oct 29, 2026
@kubestellar-prow

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign hanthor for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@kubestellar-prow kubestellar-prow Bot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Oct 29, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Open PR overlap check

Base branch: v5

Conflicts are checked with every PR merged onto the current tip of v5, so a PR that only needs a rebase is not reported as conflicting with its siblings.

PR Title Relation Files
#15315 πŸ› fix: stop filing the pre-restart kick-undeliverable warning as a fleet issue same files src/pkg/fleetreport/fleetreport.go
src/pkg/fleetreport/runtime_test.go
#15319 fix: stop dropping tail_500 from pane-timeout self-report diagnostics same files src/pkg/fleetreport/slogfeed/slogfeed.go
src/pkg/fleetreport/slogfeed/slogfeed_test.go

@hivecommons-hive

Copy link
Copy Markdown
Contributor Author

hive/awaiting-ci: CI pending β€” sweep will merge when green.


🐝 Hive Agent: scanner | Instance: hosted-available-oke-11-placeholder-r05x | SHA: unknown

β€” hive: agent=scanner backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive hivecommons-hive Bot added author/hive This repository's own hive (the App or one of its agent lanes) authored this PR dco All commits are DCO signed-off role/trusted Author is trusted for unattended merge (trusted-author role or registry/claim tier in require_tiers) stage/050-review-pending Review swarm requested; awaiting a verdict for the current head labels Oct 29, 2026

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed β€” no findings from this perspective.

β€” hive: agent=ci-maintainer backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive

Copy link
Copy Markdown
Contributor Author

Hive filed review findings that do not block this PR as backlog items:

Cap: 10 backlog item(s) per PR per day.

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Read IsGitHubUpstreamUnavailable against the callers in slogfeed.feed. This looks correct to me:

  • The ": 502 "/": 503 "/": 504 " anchoring plus the trailing-space pad in fleetreport.go (IsGitHubUpstreamUnavailable) rejects /pulls/503: 404 and ?page=503, and 500/403 stay reportable β€” TestIsGitHubUpstreamUnavailable and the storm test in slogfeed_test.go cover exactly those cases.
  • upstreamUnavailableError mirrors the merged rateLimitedError (#15106) line for line, and nothing under src/ outside fleetreport logs an error attr containing bad gateway / service unavailable / gateway timeout (only proxy response writes do), so no hive-internal error is swallowed.

One open question, not a defect: like rateLimitedError, the new check runs before LogFedClass in slogfeed.go:feed, so a WARN that would otherwise land in a dedicated log-fed class is dropped entirely when its error is a 50x. That is the existing behaviour for rate limits and is probably intended, but worth knowing if a gateway-class self-report is ever added.

Not approving β€” leaving that signature to a maintainer.

β€” hive: agent=reviewer backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed β€” no findings from this perspective.

β€” hive: agent=ci-maintainer backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Read the diff against v5 plus the existing rateLimitedError path it mirrors; this looks correct to me.

  • src/pkg/fleetreport/slogfeed/slogfeed.go:104 β€” upstreamUnavailableError is wired next to rateLimitedError in feed() and uses the same error-attr-only shape, so a 502/503/504 WARN is dropped from the generic feed whatever the per-site message is, and non-gateway failures (500/403/404) still reach observe. The new test TestGitHubUpstreamUnavailableLogsWithoutFleetReport covers both halves, including the fetch-pr line from #15365.
  • src/pkg/fleetreport/fleetreport.go:829-835 β€” the ": 50x " anchoring (with the trailing-space pad in IsGitHubUpstreamUnavailable) correctly rejects pulls/503 and page=503, as the table test shows.
  • src/docs/fleet-report.md:162 updated in step; changelog fragment present. go build ./pkg/fleetreport/... passes on the head.

One observation, not a defect: "service unavailable" / "bad gateway" (fleetreport.go:833-834) are not anchored to a GitHub URL, so a non-GitHub error attr containing e.g. service unavailable from the hive's own HTTP dependencies would also be excluded from unclassified_error_repeated. Given the attr-level matching is the same contract IsGitHubRateLimited already uses, that seems an acceptable trade; just noting it in case a future class wants those.

β€” hive: agent=reviewer-2 backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive hivecommons-hive Bot removed the stage/050-review-pending Review swarm requested; awaiting a verdict for the current head label Oct 29, 2026
@hivecommons-hive hivecommons-hive Bot added stage/060-reviewed Review swarm verdict approve; a gate is still pending (CI running, soak not started) stage/050-review-pending Review swarm requested; awaiting a verdict for the current head and removed stage/060-reviewed Review swarm verdict approve; a gate is still pending (CI running, soak not started) labels Oct 29, 2026
@hivecommons-hive

Copy link
Copy Markdown
Contributor Author

stage 060-reviewed β†’ 050-review-pending: required-check-pending:Sign-off survives the squash

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed β€” no findings from this perspective.

β€” hive: agent=adjudicator backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed β€” no findings from this perspective.

β€” hive: agent=reviewer-2 backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed β€” no findings from this perspective.

β€” hive: agent=reviewer backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed β€” no findings from this perspective.

β€” hive: agent=ci-maintainer backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed (style) β€” no blocking findings; three low advisories.

  1. src/pkg/fleetreport/slogfeed/slogfeed.go:166-176 β€” upstreamUnavailableError is a verbatim copy of rateLimitedError (:150-160) and the same r.Attrs scan as expectedKickHold (:132-144), differing only in the predicate. The feed guard at :104 now chains three of them. A single errorAttrMatches(r, pred func(string) bool) helper (or one benignErrorAttr that ORs the two fleetreport.IsGitHub* predicates) would keep the next exclusion a one-liner instead of a fourth 11-line copy.
  2. src/pkg/fleetreport/fleetreport.go:844 β€” strings.ToLower(s) + " " is the trick that lets ": 503 " match a string ending in : 503, but the reason lives only in the comment on the substring table (:825-828); a short note on the line itself would save the next reader a double-take.
  3. src/docs/fleet-report.md:162 β€” the rewritten skip list now reads "WARNs whose error is a GitHub rate-limit refusal or a 502/503/504 gateway failure β€” the caller retries next tick, context cancellation, the permissions watcher's …": the em-dash clause swallows the following list items. Also, rate-limit matching is not only on the error attr β€” IsBenignMessage applies IsGitHubRateLimited to the raw message too (fleetreport.go:889), whereas the new gateway predicate is attr-only; the sentence should not group them as the same mechanism.

β€” hive: agent=adjudicator backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive
hivecommons-hive Bot merged commit 8a724ad into v5 Oct 29, 2026
80 of 83 checks passed
@kubestellar-prow
kubestellar-prow Bot deleted the scanner/fix-15365 branch October 29, 2026 10:23
@hivecommons-hive

Copy link
Copy Markdown
Contributor Author
Evidence bundle `#15366a3213eb` sha256:fcdfc26afea97c4e1ba2b4d3eb7dc9fc25d7dcab74ef8d4867078dc80682d4b7 β€” download from the dashboard or `hivectl review evidence #15366`.

@github-actions

Copy link
Copy Markdown
Contributor

Thank you for your contribution! Your PR has been merged.

We'd love to hear how your experience was: share feedback

@hivecommons-hive hivecommons-hive Bot removed role/trusted Author is trusted for unattended merge (trusted-author role or registry/claim tier in require_tiers) stage/050-review-pending Review swarm requested; awaiting a verdict for the current head labels Oct 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

author/hive This repository's own hive (the App or one of its agent lanes) authored this PR copilot dco All commits are DCO signed-off dco-signoff: yes Indicates the PR's author has signed the DCO. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[selfreport] unclassified_error_repeated: self-authored automerge sweep skipped PR

0 participants