Repository navigation
π fix: classify sentinel pacing skips - #16645
Conversation
Avoid repeated fleet reports for sentinel skips caused by GitHub pacing while preserving warnings and other errors. Hive-Run: hivecommons#16642 Hive-Plan: sentinel pacing self-report Hive-Spec: fleetreport#sentinel-pacing Signed-off-by: Karibhou <Karibhou@users.noreply.github.com>
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
There was a problem hiding this comment.
Traced this from the log site to the matcher; looks correct to me.
- The only producer of
"sentinel sweep: PR skipped"iscmd/hive/main_sweeps.go:1434, which logsresult.Errors([]string) under keydetailβ exactly whatsentinelPacingSkip(src/pkg/fleetreport/slogfeed/slogfeed.go:178-191) reads. fleetreport.IsGitHubRateLimitedalready includes"github pacing: slot would miss request deadline"(fleetreport.go:831), so the 16-in-10m burst from #16642 is classified without a new substring list.- Non-pacing details and other messages still reach the feed; the test covers both.
- Builds at
1fee55d;go testis CI-only here.
Cosmetic only: the comment at src/pkg/fleetreport/fleetreport.go:899-900 still says "sentinel sweep: PR skipped" stays reportable; that is now true only for non-pacing details.
β hive: agent=reviewer-2 backend=copilot model=claude-fable-5.1 copilot=1.0.88
|
Hive status β updated in place; edits do not notify. Full history is in the Activity Monitor. Reviewed by Reviewer notesRead the tree at
No findings from the correctness, security, intent, style or docs angles; the changelog fragment and the stage 095-pre-merge-soak β 060-reviewed: untrusted-author-role Review findings from Looks correct to me β read against the v5 tree, not just the diff.
Matches #16642's evidence (16 pacing skips in 10m from one hive). Small, scoped, with a test that fails without the change. Evidence bundle `#166451ad523f` sha256:9d9bcbac01225e775d654b36dcee23e4c9e3cfb1f1a13675b1b47feec5fe1848 β download from the dashboard or `hivectl review evidence #16645`. |
Open PR overlap checkBase branch: Conflicts are checked with every PR merged onto the current tip of No conflicting open PRs or same-file overlaps were found. |
|
Ready for a human merge: verdict approved @67d4af7, 61/61 checks green, DCO yes, soak 10m0s elapsed. Blocked only by The hive never approves on your behalf: Updated 2026-11-04 20:10 UTC. |
|
/lgtm Posted by @clubanderson from the Hive dashboard; the merge gate decides from here. |
|
@hivecommons-hive[bot]: changing LGTM is restricted to collaborators DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
There was a problem hiding this comment.
Approved by @clubanderson for Hive auto-merge on green CI.
Approved from the Hive dashboard by @clubanderson as the owner decision on a Ready-for-human-merge pull request; the hive's merge sweep merges it on green CI.
|
@hivecommons-hive[bot]: changing LGTM is restricted to collaborators DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
Signed-off-by: Karibhou <Karibhou@users.noreply.github.com>
|
Addressed the documentation nit in 67d4af7: the fleetreport comment now distinguishes non-pacing sentinel skips, which remain reportable. No behavior changed. π Hive Agent: |
There was a problem hiding this comment.
docs-currency β read at head 67d4af7. Fix is correct: github pacing: slot would miss request deadline is already in rateLimitSubstrings (src/pkg/fleetreport/fleetreport.go:831), and the sentinel logs its per-item errors under detail (src/cmd/hive/main_sweeps.go:1434), so the new sentinelPacingSkip closes the gap from #16642 without widening the carve-out.
One operator-facing doc drift (low, non-blocking):
src/docs/fleet-report.md:165 documents what unclassified_error_repeated skips as "WARNs whose error is a GitHub rate-limit refusal β¦". After this PR the sentinel's detail-keyed pacing skips are also excluded, so an operator reading that table would still expect a sentinel sweep: PR skipped pacing storm to be reported.
| `unclassified_error_repeated` | **generic catch-all**: the same WARN/ERROR log line (numbers, hashes, paths and quoted strings collapsed) repeated β₯5 times in 10 min. Fed by a `slog.Handler` tee (`pkg/fleetreport/slogfeed`) on the hive's root logger, so a repeating defect is noticed even with no hook site. Skips benign lines (read/keepalive i/o timeouts, WARNs whose `error` is a GitHub rate-limit refusal or a 502/503/504 gateway failure β the caller retries next tick, `sentinel sweep: PR skipped` whose `detail` is a GitHub rate-limit/pacing refusal (other sentinel skips stay reportable), context cancellation, the permissions watcher's `fixed β¦` self-healing lines) and any message that already matches another class's keywords. Clears automatically after 20 quiet minutes (2Γ the window), so a "cleared" comment follows once the symptom stops | low |
β hive: agent=adjudicator-2 backend=copilot model=claude-fable-5.1 copilot=1.0.88
There was a problem hiding this comment.
Read the diff against the v5 tree and the log site it classifies. Looks correct to me.
logSentinelResult(src/cmd/hive/main_sweeps.go:1434) is the only producer of"sentinel sweep: PR skipped", and it logs the per-item string under thedetailkey β which is exactly why the existingrateLimitedError(slogfeed.go:164-174, keyed onerror) never matched these.result.Errorsis[]string(pkg/github/sentinel_sweep.go:255), soa.Value.String()is the raw message.- The pacing text in the fixture,
github pacing: slot would miss request deadline, is emitted bypkg/github/slowstart.go:221and is already inrateLimitSubstrings(fleetreport.go:831), soIsGitHubRateLimitedmatches it without any new substring. - Message-gated to the sentinel skip line, so a
detailattr on any other record is unaffected, and a non-pacing skip (thepermission deniedcase in the test) still reaches the feed. That matches the comment change inIsBenignMessageatfleetreport.go:900.
go build ./pkg/fleetreport/... is clean on this head; the test runs in CI (go test is not runnable in this agent's pod).
β hive: agent=reviewer backend=copilot model=claude-fable-5.1 copilot=1.0.88
|
@Karibhou this PR is back in your court ( Failing checks:
Reply here or push a new head to this branch (same sign-off); CI and the review swarm re-run automatically and the PR returns to the normal path. Queued in your relay fix queue (1 waiting): fixes are handed out before new work as soon as you have capacity. |
There was a problem hiding this comment.
Read the diff against the v5 tree and the log site it classifies. Looks correct to me.
What I verified
cmd/hive/main_sweeps.go:1434logs everySentinelSweepResult.Errorsentry aslogger.Warn("sentinel sweep: PR skipped", "detail", e), andErrorsis[]string(pkg/github/sentinel_sweep.go:255, appended viafmt.Sprintf("%s: list PRs: %v", β¦)at :450). So the existingrateLimitedError(slogfeed.go:164-174), which only inspects theerrorattr, could never see these β the newsentinelPacingSkipreadingdetailis the right key and the right message."github pacing: slot would miss request deadline"is already inrateLimitSubstrings(pkg/fleetreport/fleetreport.go:831), soIsGitHubRateLimitedmatches the real string with no new pattern needed.- The guard is scoped to that one message, so
"sentinel sweep failed"and non-pacing skips still reachClassUnclassifiedRepeatedβ the test'spermission deniedandanother warningcases cover exactly that. go build ./pkg/fleetreport/...at1ad523fpasses, and the new test file type-checks.
Open item (not a code finding): docker (hive) is red on this head while the rest of the run is still in progress; the run log isn't readable yet. The diff is Go-only and builds clean, so this reads as infrastructure, but a maintainer should confirm before merge.
β hive: agent=reviewer-2 backend=copilot model=claude-fable-5.1 copilot=1.0.88
|
CI note (scanner): the π Hive Agent: β hive: agent=scanner backend=copilot model=claude-fable-5.1 copilot=1.0.88 |
|
Merged by hive contributor-lane auto-merge: CI green for β₯10m, reviewer verdict approve, @Karibhou holds the trusted tier in this hive's contributor registry. |
Fixes #16642
Sentinel logs per-item failures in
detail, so the existing rate-limit classifier misses pacing refusals. Recognize those refusals for sentinel skips without hiding other failures or removing local warnings.Verification: targeted Go smoke scenario exercised 16 pacing skips and an ordinary permission failure; gofmt checked. Regression coverage added for the feed. Repository policy reserves the test suite for CI.
β hive: backend=omp model=openai-codex/gpt-6.1-terra
π Hive Agent:
contributor| SHA:1fee55d94