Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions changelog.d/fixed-18589-open-pr-guard-label.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
- The open-PR guard now writes the `has-open-pr` label on its own time budget, remembers a failed write per issue instead of retrying it on every claim pass, logs refusals (no write access, issue gone or locked) at INFO, and reports repeated transient failures under a new `open_pr_guard_label_failed` fleet-report class instead of `unclassified_error_repeated` (#18589).
1 change: 1 addition & 0 deletions src/docs/fleet-report.md
Original file line number Diff line number Diff line change
Expand Up @@ -170,6 +170,7 @@ title is `[selfreport] <class>: <short message>`.
| `tmux_session_check_failed` | the agent's tmux session `has-session` check failed (`tmux has-session failed`, `crux/agent/manager.go`) — typically the session was lost or the tmux server is unhealthy, repeated ≥5 times in 10 min. Log-fed: same repeat threshold and idle clear as `agent_pane_timeout` | medium |
| `transcript_scan_slow` | a Copilot transcript activity scan took longer than its interval (`transcript activity scan outlived its interval`, `crux/agent/busy_visibility.go`) — agent liveness reads may lag. The WARN itself is debounced at the source (`transcriptScanPace`, `crux/agent/transcript_scan_pace.go`): it fires once per slow episode, only after 3 consecutive passes outlive the interval, and re-arms only after a pass finishes under 80% of the interval, so one sustained overrun logs one line and never trips this class; it is reported only when ≥5 distinct slow→fast→slow episodes occur within 10 min. Log-fed: same ≥5-in-10-min repeat threshold and 20-minute idle clear as `agent_pane_timeout`. Operator diagnostics: check the agent's transcript/session volume; it is bounded by concurrency, so repeated episodes indicate slow disk or very large session directories | low |
| `kick_freshness_withheld` | a kick was withheld because every issue/PR ref failed the pre-kick freshness check (`kick withheld: every ref failed the freshness check`, `cmd/hive/kick_freshness.go`). The kick is retried on the next cycle; reported only when the same withholding repeats ≥5 times in 10 min (refs persistently unreadable). Diagnostics count stale snapshots, closed items, deferred reads and other read errors for the last withheld kick, without including refs or raw errors. See local per-ref logs for details. Deferred API-budget refusals, refs in a stale repo and refs the re-read found closed or merged log at INFO, so a kick withheld only by those never trips it; a stale repo is reported where it is produced (`forgestate: repo cache is stale; serving last-good items (marked stale)`, `GitHub refused repo enumeration transiently; serving last-good snapshot for this repo (marked stale)`, both WARN in `pkg/github/client.go`). Its main job is keeping the residual WARN out of `unclassified_error_repeated`. Log-fed: same repeat threshold and idle clear as `agent_pane_timeout` | low |
| `open_pr_guard_label_failed` | the pre-dispatch open-PR guard refused a claim but could not put `has-open-pr` on the issue (`open-PR guard: could not label issue` / `could not ensure label`, `pkg/github/open_pr_links.go`). The label is a tracker signal only: the claim is refused either way. Each issue is retried at most once per 15 minutes after a transient failure (timeout, rate limit, 5xx); a refusal (403/404/410/422: no write access, issue gone or locked) is remembered for 6 hours and logs at INFO, so it never feeds this class. Log-fed: ≥5 WARNs in 10 minutes (hivecommons/hive#18589) | low |
| `models_endpoint_empty` | the dashboard's inference model discovery (`/api/inference/{backend}/models`) found no models from any configured endpoint (`no models found from any endpoint`, `pkg/dashboard/api_models.go`) and fell back to the static aliases, flagged unverified in the UI. The source logs it at WARN once per backend per hour (DEBUG on repeats, re-armed when discovery succeeds), so this fires only when the condition flaps ≥5 times in 10 min (gateway key rejected, endpoint down). Log-fed: same repeat threshold and idle clear as `agent_pane_timeout` | low |
| `unclassified_error_repeated` | **generic catch-all**: the same WARN/ERROR log line (numbers, hashes, paths and quoted strings collapsed) repeated ≥5 times in 10 min. Fed by a `slog.Handler` tee (`pkg/fleetreport/slogfeed`) on the hive's root logger, so a repeating defect is noticed even with no hook site. Skips benign lines (read/keepalive i/o timeouts, WARNs whose `error` is a GitHub rate-limit refusal or a 502/503/504 gateway failure — the caller retries next tick, context cancellation, the permissions watcher's `fixed …` self-healing lines) and any message that already matches another class's keywords. Clears automatically after 20 quiet minutes (2× the window), so a "cleared" comment follows once the symptom stops | low |

Expand Down
2 changes: 2 additions & 0 deletions src/pkg/fleetreport/runtime.go
Original file line number Diff line number Diff line change
Expand Up @@ -90,6 +90,7 @@ const (
ClassInferenceBackendAuthFailure = "inference_backend_auth_failure"
ClassSHAPollCommitMsgUnavail = "sha_poll_commit_message_unavailable"
ClassKickFreshnessWithheld = "kick_freshness_withheld"
ClassOpenPRGuardLabelFailed = "open_pr_guard_label_failed"
ClassModelsEndpointEmpty = "models_endpoint_empty"
ClassAgentLoginRequired = "agent_login_required"
ClassPRRequestHeadUnpushed = "pr_request_head_unpushed"
Expand Down Expand Up @@ -217,6 +218,7 @@ var Classes = map[string]ClassSpec{
ClassInferenceUpstreamError: {Component: "inference-gateway", Severity: "medium", MinCount: unclassifiedRepeatCount, Window: unclassifiedRepeatWindow, IdleClear: unclassifiedIdleClear, LogFed: true, Keywords: []string{"inference upstream error", "inference upstream failed", "inference translate request failed"}},
ClassSHAPollCommitMsgUnavail: {Component: "github-client", Severity: "low", MinCount: unclassifiedRepeatCount, Window: unclassifiedRepeatWindow, IdleClear: unclassifiedIdleClear, LogFed: true, Keywords: []string{"sha poll: commit message fetch non-200"}},
ClassKickFreshnessWithheld: {Component: "agent-runtime", Severity: "low", MinCount: unclassifiedRepeatCount, Window: unclassifiedRepeatWindow, IdleClear: unclassifiedIdleClear, LogFed: true, Keywords: []string{"kick withheld: every ref failed the freshness check"}},
ClassOpenPRGuardLabelFailed: {Component: "github-client", Severity: "low", MinCount: unclassifiedRepeatCount, Window: unclassifiedRepeatWindow, IdleClear: unclassifiedIdleClear, LogFed: true, Keywords: []string{"open-pr guard: could not label issue", "open-pr guard: could not ensure label"}},
ClassModelsEndpointEmpty: {Component: "dashboard", Severity: "low", MinCount: unclassifiedRepeatCount, Window: unclassifiedRepeatWindow, IdleClear: unclassifiedIdleClear, LogFed: true, Keywords: []string{"no models found from any endpoint"}},
ClassAgentLoginRequired: {Component: "agent-runtime", Severity: "medium", MinCount: unclassifiedRepeatCount, Window: unclassifiedRepeatWindow, IdleClear: unclassifiedIdleClear, LogFed: true, Keywords: []string{"login required detected", "watchdog: agent credential failure"}},
ClassPRRequestHeadUnpushed: {Component: "github-relay", Severity: "medium", MinCount: unclassifiedRepeatCount, Window: unclassifiedRepeatWindow, IdleClear: unclassifiedIdleClear, LogFed: true, Keywords: []string{"head branch is not on the remote"}},
Expand Down
3 changes: 3 additions & 0 deletions src/pkg/fleetreport/runtime_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -467,6 +467,7 @@ func TestEveryRuntimeClassTriggerAndNonTriggerThresholds(t *testing.T) {
{class: ClassInferenceUpstreamError, message: "inference upstream error"},
{class: ClassSHAPollCommitMsgUnavail, message: "SHA poll: commit message fetch non-200"},
{class: ClassKickFreshnessWithheld, message: "kick withheld: every ref failed the freshness check"},
{class: ClassOpenPRGuardLabelFailed, message: "open-PR guard: could not label issue"},
{class: ClassModelsEndpointEmpty, message: "no models found from any endpoint"},
{class: ClassAgentLoginRequired, message: "login required detected"},
{class: ClassPRRequestHeadUnpushed, message: "pr-request watcher: head branch is not on the remote"},
Expand Down Expand Up @@ -1267,6 +1268,8 @@ func TestRecurringRuntimeMessagesAreClassified(t *testing.T) {
"issue-claims: GitHub side effect failed": ClassIssueClaimEffectFailed,
"SHA poll: commit message fetch non-200": ClassSHAPollCommitMsgUnavail,
"kick withheld: every ref failed the freshness check": ClassKickFreshnessWithheld,
"open-PR guard: could not label issue": ClassOpenPRGuardLabelFailed,
"open-PR guard: could not ensure label": ClassOpenPRGuardLabelFailed,
"no models found from any endpoint": ClassModelsEndpointEmpty,
"login required detected": ClassAgentLoginRequired,
"pr-request watcher: head branch is not on the remote": ClassPRRequestHeadUnpushed,
Expand Down
106 changes: 99 additions & 7 deletions src/pkg/github/open_pr_links.go
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ import (
"errors"
"fmt"
"log/slog"
"net/http"
"sort"
"strings"
"sync"
Expand Down Expand Up @@ -49,6 +50,22 @@ const (
// for its repo: a surge of claims fails open at once instead of each one
// re-running a listing that just timed out and re-logging the warning.
openPRLinkFailureTTL = time.Minute

// openPRLabelTimeout bounds the has-open-pr label write on its own
// budget. It used to share the caller's deadline (15s from the
// contribute lane) with the open-PR listing ahead of it, so a slow
// listing of a large repo left the label write an already-spent
// context and every guard pass failed with `context deadline exceeded`
// (hivecommons/hive#18589).
openPRLabelTimeout = 10 * time.Second
// openPRLabelRetryBackoff is how long a transient label failure
// (timeout, rate limit, 5xx) keeps the guard from re-trying that issue:
// the label is a tracker signal, and a claim surge on the same issue
// must not turn one slow write into a WARN per pass.
openPRLabelRetryBackoff = 15 * time.Minute
// openPRLabelRefusedBackoff is the same for a refusal that will not fix
// itself (no write access, issue gone or locked).
openPRLabelRefusedBackoff = 6 * time.Hour
)

// errOpenPRListingBackedOff marks an error replayed from the failure cache.
Expand Down Expand Up @@ -93,14 +110,17 @@ type openPRLinkCache struct {
repos map[string]*openPRLinkIndex
// failures holds each repo's last listing failure for openPRLinkFailureTTL.
failures map[string]openPRLinkFailure
// labelBackoff holds, per claimKey, when the guard may next try to
// label that issue after a failed has-open-pr write.
labelBackoff map[string]time.Time
// disabled is scanner.skip_issues_with_open_pr: false.
disabled bool
now func() time.Time
}

func (c *Client) openPRLinks() *openPRLinkCache {
c.openPRLinkOnce.Do(func() {
c.openPRLinkState = &openPRLinkCache{repos: map[string]*openPRLinkIndex{}, failures: map[string]openPRLinkFailure{}, now: time.Now}
c.openPRLinkState = &openPRLinkCache{repos: map[string]*openPRLinkIndex{}, failures: map[string]openPRLinkFailure{}, labelBackoff: map[string]time.Time{}, now: time.Now}
})
return c.openPRLinkState
}
Expand Down Expand Up @@ -297,18 +317,90 @@ func (c *Client) openPRLinkBlockReason(ctx context.Context, repo string, number

// markIssueHasOpenPR applies HasOpenPRLabel once. Idempotent on the labels
// the caller just read; a label failure is logged, never fatal — the claim
// is refused either way.
// is refused either way. The write runs on its own budget, and a failed
// write is remembered per issue so later passes skip it quietly until the
// backoff lapses (hivecommons/hive#18589). A refusal that cannot fix itself
// logs at INFO; a transient failure logs one WARN per backoff window.
func (c *Client) markIssueHasOpenPR(ctx context.Context, repo string, number int, labels []string, link OpenPRLink) {
for _, l := range labels {
if strings.EqualFold(strings.TrimSpace(l), HasOpenPRLabel) {
return
}
}
if err := c.EnsureIssueLabel(ctx, repo, HasOpenPRLabel, hasOpenPRLabelColor, hasOpenPRLabelDescription); err != nil && c.logger != nil {
c.logger.Warn("open-PR guard: could not ensure label", slog.String("label", HasOpenPRLabel), slog.String("repo", repo), slog.String("error", err.Error()))
key := claimKey(c.canonicalRepo(repo), number)
cache := c.openPRLinks()
cache.mu.Lock()
until, backedOff := cache.labelBackoff[key]
backedOff = backedOff && cache.now().Before(until)
cache.mu.Unlock()
if backedOff {
return
}

labelCtx, cancel := context.WithTimeout(context.WithoutCancel(ctx), openPRLabelTimeout)
defer cancel()
step := "could not ensure label"
err := c.EnsureIssueLabel(labelCtx, repo, HasOpenPRLabel, hasOpenPRLabelColor, hasOpenPRLabelDescription)
if err == nil || !errors.Is(err, context.DeadlineExceeded) {
// A failed lookup or create does not mean the label is missing:
// the add below still lands when it exists (or the forge creates
// it), so only a spent budget skips it.
if addErr := c.AddLabels(labelCtx, repo, number, []string{HasOpenPRLabel}); addErr != nil {
step, err = "could not label issue", addErr
} else {
err = nil
}
}

cache.mu.Lock()
now := cache.now()
for k, t := range cache.labelBackoff {
if !now.Before(t) {
delete(cache.labelBackoff, k)
}
}
if err == nil {
delete(cache.labelBackoff, key)
cache.mu.Unlock()
return
}
if err := c.AddLabels(ctx, repo, number, []string{HasOpenPRLabel}); err != nil && c.logger != nil {
c.logger.Warn("open-PR guard: could not label issue", slog.String("label", HasOpenPRLabel),
slog.String("repo", repo), slog.Int("issue", number), slog.Int("pr", link.PRNumber), slog.String("error", err.Error()))
reason, refused := openPRLabelFailureReason(err)
backoff := openPRLabelRetryBackoff
if refused {
backoff = openPRLabelRefusedBackoff
}
cache.labelBackoff[key] = now.Add(backoff)
cache.mu.Unlock()

if c.logger == nil {
return
}
attrs := []any{slog.String("label", HasOpenPRLabel), slog.String("repo", repo), slog.Int("issue", number),
slog.Int("pr", link.PRNumber), slog.String("reason", reason), slog.Duration("retry_after", backoff), slog.String("error", err.Error())}
if refused {
c.logger.Info("open-PR guard: "+step+" (refused; not retrying this issue for a while)", attrs...)
return
}
c.logger.Warn("open-PR guard: "+step, attrs...)
}

// openPRLabelFailureReason names a has-open-pr write failure for the log and
// reports whether it is a refusal that will not fix itself on retry: no
// write access (403), the repo or issue is gone (404/410), or the forge
// rejected the write — a locked issue answers 422.
func openPRLabelFailureReason(err error) (reason string, refused bool) {
switch {
case isRateLimitErr(err):
return "rate-limited", false
case errors.Is(err, context.DeadlineExceeded):
return "timeout", false
case githubStatusError(err, http.StatusForbidden):
return "forbidden", true
case githubStatusError(err, http.StatusNotFound), githubStatusError(err, http.StatusGone):
return "not-found", true
case githubStatusError(err, http.StatusUnprocessableEntity):
return "unprocessable", true
default:
return "error", false
}
}
Loading
Loading