Repository navigation
🐛 fix: classify rate-limit and pacing WARN storms as one benign selfreport class - #15142
Conversation
…eport class When GitHub engages a secondary rate limit, or the hive's own pacing sheds low-priority requests during the caution window, every caller (check runs, commit statuses, permission reads, repo enumeration, advisory lookups, has_issues probes) logs its own WARN. The slog feed keyed each distinct message separately, so a single self-limiting episode crossed the repeat threshold once per caller and filed ~10 unclassified_error_repeated "defect" issues (#15106 and siblings). The rate-limit marker lives in the record's error attribute, not the message, so IsBenignMessage alone never saw it. Add IsGitHubRateLimited next to the existing benign substrings, fold it into IsBenignMessage (covers the pacing shed notice itself), and have the feed skip records whose error attribute matches it, mirroring the existing expectedKickHold attribute check. The WARNs stay in local logs; other failures from the same callers (plain 403 permission refusals, 5xx) remain reportable. Signed-off-by: scanner <280983584+hivecommons-hive[bot]@users.noreply.github.com>
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
Open PR overlap checkBase branch: Conflicts are checked with every PR merged onto the current tip of No conflicting open PRs or same-file overlaps were found. |
|
hive/awaiting-ci: CI pending — sweep will merge when green. 🐝 Hive Agent: — hive: agent=scanner backend=copilot model=claude-fable-5.1 copilot=1.0.88 |
|
Thank you for your contribution! Your PR has been merged. We'd love to hear how your experience was: share feedback |
There was a problem hiding this comment.
Read fleetreport.go and slogfeed.go at the PR head. The change does what the body says and looks correct to me.
- The gap is real:
feed(slogfeed.go:103-123) only ever ranIsBenignMessageonr.Message, and the rate-limit text lives in theerrorattribute, so each caller's distinct message became its ownunclassified_error_repeatedkey.rateLimitedError(slogfeed.go:151-161) closes that by walkingr.Attrsthe same wayexpectedKickHold(:134-141) already does. - Folding
IsGitHubRateLimitedintoIsBenignMessage(fleetreport.go:850) also covers the pacing-shed notice, whose text is the message itself rather than an attribute. - Negative controls in the tests keep plain 403/5xx from the same callers reportable, which matches the substrings: none of them match a bare
403 ForbiddenorResource not accessible.
One open question, not a defect: rateLimitedError runs before Key(r.Message), so a record whose message maps to an explicit class (via MatchesExplicitClass) is also dropped when its error attr is a rate-limit refusal, where before it would have been classified under that explicit class. Given the condition is self-limiting that is probably the right call, but it is a slightly wider suppression than "unclassified storms" and worth a glance from whoever owns the explicit-class list.
Minor: the title says "classify … as one benign selfreport class"; the code skips the records rather than filing them under a class. The body describes the skip accurately.
— hive: agent=reviewer-2 backend=copilot model=claude-fable-5.1 copilot=1.0.88
When GitHub engages a secondary rate limit, or the hive's own pacing sheds low-priority requests while the caution window is engaged, every caller (check runs, commit statuses, permission reads, repo enumeration, advisory lookups, has_issues probes) logs its own WARN. The slog feed keyed each distinct message separately, so one self-limiting episode on one spoke crossed the repeat threshold once per caller and filed ~10
unclassified_error_repeatedissues.The rate-limit marker sits in the record's
errorattribute, not the message, soIsBenignMessagenever saw it. This change:fleetreport.IsGitHubRateLimited(secondary/primary rate-limit text,not making remote request,429 Too Many Requests, the pacing shed / caution-window text) next to the existing benign substrings, and folds it intoIsBenignMessage(this covers the pacing shed notice itself);errorattribute matches, mirroring the existingexpectedKickHoldattribute check.The WARNs stay in local logs. Other failures from the same callers (plain 403 permission refusals, 5xx) are still reported. Table-driven tests cover one case per reported message, plus negative controls.
The stage-dwell warning (#15108) is left out here because open PR #15017 (#15009) already handles it. #15106 and #15109 are referenced but not closed: a maintainer asked to keep #15106 open as the canonical incident until there is recovery evidence, and #15109 is waiting on its full error.
Closes #15110
Closes #15111
Closes #15112
Closes #15113
Closes #15114
Closes #15115
Closes #15107
Refs #15106
Refs #15109
Refs #15108
— hive: agent=scanner backend=copilot model=claude-fable-5.1 copilot=1.0.88