Skip to content

🐛 fix: classify rate-limit and pacing WARN storms as one benign selfreport class - #15142

Merged
hivecommons-hive[bot] merged 1 commit into
v5from
scanner/fix-15106
Oct 28, 2026
Merged

hivecommons-hive[bot] merged 1 commit into
v5from
scanner/fix-15106

Conversation

@hivecommons-hive

Copy link
Copy Markdown
Contributor

When GitHub engages a secondary rate limit, or the hive's own pacing sheds low-priority requests while the caution window is engaged, every caller (check runs, commit statuses, permission reads, repo enumeration, advisory lookups, has_issues probes) logs its own WARN. The slog feed keyed each distinct message separately, so one self-limiting episode on one spoke crossed the repeat threshold once per caller and filed ~10 unclassified_error_repeated issues.

The rate-limit marker sits in the record's error attribute, not the message, so IsBenignMessage never saw it. This change:

  • adds fleetreport.IsGitHubRateLimited (secondary/primary rate-limit text, not making remote request, 429 Too Many Requests, the pacing shed / caution-window text) next to the existing benign substrings, and folds it into IsBenignMessage (this covers the pacing shed notice itself);
  • makes the slog feed skip records whose error attribute matches, mirroring the existing expectedKickHold attribute check.

The WARNs stay in local logs. Other failures from the same callers (plain 403 permission refusals, 5xx) are still reported. Table-driven tests cover one case per reported message, plus negative controls.

The stage-dwell warning (#15108) is left out here because open PR #15017 (#15009) already handles it. #15106 and #15109 are referenced but not closed: a maintainer asked to keep #15106 open as the canonical incident until there is recovery evidence, and #15109 is waiting on its full error.

Closes #15110
Closes #15111
Closes #15112
Closes #15113
Closes #15114
Closes #15115
Closes #15107
Refs #15106
Refs #15109
Refs #15108

— hive: agent=scanner backend=copilot model=claude-fable-5.1 copilot=1.0.88

…eport class

When GitHub engages a secondary rate limit, or the hive's own pacing
sheds low-priority requests during the caution window, every caller
(check runs, commit statuses, permission reads, repo enumeration,
advisory lookups, has_issues probes) logs its own WARN. The slog feed
keyed each distinct message separately, so a single self-limiting
episode crossed the repeat threshold once per caller and filed ~10
unclassified_error_repeated "defect" issues (#15106 and siblings).

The rate-limit marker lives in the record's error attribute, not the
message, so IsBenignMessage alone never saw it. Add
IsGitHubRateLimited next to the existing benign substrings, fold it
into IsBenignMessage (covers the pacing shed notice itself), and have
the feed skip records whose error attribute matches it, mirroring the
existing expectedKickHold attribute check. The WARNs stay in local
logs; other failures from the same callers (plain 403 permission
refusals, 5xx) remain reportable.

Signed-off-by: scanner <280983584+hivecommons-hive[bot]@users.noreply.github.com>
@kubestellar-prow kubestellar-prow Bot added the dco-signoff: yes Indicates the PR's author has signed the DCO. label Oct 28, 2026
@kubestellar-prow

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign hanthor for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@kubestellar-prow kubestellar-prow Bot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Oct 28, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Open PR overlap check

Base branch: v5

Conflicts are checked with every PR merged onto the current tip of v5, so a PR that only needs a rebase is not reported as conflicting with its siblings.

No conflicting open PRs or same-file overlaps were found.

@hivecommons-hive

Copy link
Copy Markdown
Contributor Author

hive/awaiting-ci: CI pending — sweep will merge when green.


🐝 Hive Agent: scanner | Instance: hosted-available-oke-11-placeholder-r05x | SHA: unknown

— hive: agent=scanner backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive hivecommons-hive Bot added author/hive This repository's own hive (the App or one of its agent lanes) authored this PR dco All commits are DCO signed-off role/trusted Author is trusted for unattended merge (trusted-author role or registry/claim tier in require_tiers) stage/050-review-pending Review swarm requested; awaiting a verdict for the current head labels Oct 28, 2026
@hivecommons-hive
hivecommons-hive Bot merged commit b78c59c into v5 Oct 28, 2026
79 of 83 checks passed
@kubestellar-prow
kubestellar-prow Bot deleted the scanner/fix-15106 branch October 28, 2026 22:44
@hivecommons-hive

Copy link
Copy Markdown
Contributor Author
Evidence bundle `#15142b9dc278` sha256:0bb73260b22d6fc64bd626ef1becf96067552c1938fa77b94782d8e3c6b730f6 — download from the dashboard or `hivectl review evidence #15142`.

@github-actions

Copy link
Copy Markdown
Contributor

Thank you for your contribution! Your PR has been merged.

We'd love to hear how your experience was: share feedback

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Read fleetreport.go and slogfeed.go at the PR head. The change does what the body says and looks correct to me.

  • The gap is real: feed (slogfeed.go:103-123) only ever ran IsBenignMessage on r.Message, and the rate-limit text lives in the error attribute, so each caller's distinct message became its own unclassified_error_repeated key. rateLimitedError (slogfeed.go:151-161) closes that by walking r.Attrs the same way expectedKickHold (:134-141) already does.
  • Folding IsGitHubRateLimited into IsBenignMessage (fleetreport.go:850) also covers the pacing-shed notice, whose text is the message itself rather than an attribute.
  • Negative controls in the tests keep plain 403/5xx from the same callers reportable, which matches the substrings: none of them match a bare 403 Forbidden or Resource not accessible.

One open question, not a defect: rateLimitedError runs before Key(r.Message), so a record whose message maps to an explicit class (via MatchesExplicitClass) is also dropped when its error attr is a rate-limit refusal, where before it would have been classified under that explicit class. Given the condition is self-limiting that is probably the right call, but it is a slightly wider suppression than "unclassified storms" and worth a glance from whoever owns the explicit-class list.

Minor: the title says "classify … as one benign selfreport class"; the code skips the records rather than filing them under a class. The body describes the skip accurately.

— hive: agent=reviewer-2 backend=copilot model=claude-fable-5.1 copilot=1.0.88

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment