From e017fae8081f012df4d14dcd3311e81e91259c6d Mon Sep 17 00:00:00 2001 From: Danathar Date: Sun, 8 Nov 2026 14:20:45 -0500 Subject: [PATCH] =?UTF-8?q?=F0=9F=90=9B=20[v6]=20contributor-first:=20a=20?= =?UTF-8?q?relay=20in=20a=20disabled=20tier=20or=20with=20no=20matching=20?= =?UTF-8?q?work=20does=20not=20cover=20a=20lane?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit RoleCoverage counted a relay as covering its role when the hub would not hand it work (revoked, or its trust tier in hub.disabled_tiers) and when every ready it sent was answered no_matching_work. Both stay connected, so the governor kept the resident agent's kicks withheld while nobody worked the lane. Seen on hive-wild-mole on 2026-11-08: a relay covered scanner while selection answered no_matching_work every 30 s. Same class as #17233 and #17268. - Factor revoked / disabled tier into tierAccessGateReason (disabled half: tierDisabled). resumeGateReason keeps its reasons and order; fresh selection's disabled_tiers loop now calls tierDisabled. RoleCoverage reads the tier under c.mu and evaluates with h.mu released. - Record the idle run on the connection: lastIdleReasonSince beside lastIdleReason, set on task_unavailable in the ready and yank paths, no_matching_work and repo_unmintable folded into one run. Cleared on the selection commit, standby dispatch, resume adoption and sendFixTask. - RoleCoverage skips a relay holding no task that has been answered no_matching_work / repo_unmintable for coverageNoWorkWindow (2 min from the first answer; the relay retries every 30 s and the governor still applies contributor_first_grace on top). - Docs: RoleCoverage comment, agent-configuration.md coverage list, governor header, one drifted citation; changelog fragment. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01BQYnPGFJx81c8JXR3o1sg1 Signed-off-by: Danathar --- ...d-disabled-tier-no-work-relay-not-cover.md | 1 + src/docs/agent-configuration.md | 18 +- src/docs/design/agent-state-inventory.md | 2 +- src/pkg/dashboard/contribute_coverage.go | 106 ++++++- src/pkg/dashboard/contribute_coverage_test.go | 260 ++++++++++++++++++ src/pkg/dashboard/contribute_fixqueue.go | 2 + src/pkg/dashboard/contribute_select.go | 77 ++++-- .../dashboard/contribute_standby_dispatch.go | 2 +- src/pkg/dashboard/contribute_ws.go | 17 +- src/pkg/governor/contributor_first.go | 3 +- 10 files changed, 447 insertions(+), 41 deletions(-) create mode 100644 changelog.d/fixed-disabled-tier-no-work-relay-not-cover.md diff --git a/changelog.d/fixed-disabled-tier-no-work-relay-not-cover.md b/changelog.d/fixed-disabled-tier-no-work-relay-not-cover.md new file mode 100644 index 0000000000..cd7dac2d92 --- /dev/null +++ b/changelog.d/fixed-disabled-tier-no-work-relay-not-cover.md @@ -0,0 +1 @@ +- Contributor-first lanes are no longer left unworked by a relay the hub will not hand work to, or has no work for. A relay whose trust tier is in `hub.disabled_tiers` (or whose access is revoked) is refused every task, and a relay whose requests keep being answered `no_matching_work` (or `repo_unmintable`) is connected and idle, but both still counted as covering their role, so the governor kept the resident agent's kicks withheld while nobody worked the lane. Seen on hive-wild-mole on 2026-11-08: a relay covered the scanner lane while every request for work it sent, about every 30 seconds, was answered `no_matching_work`. The hub now leaves a revoked or disabled-tier relay out of coverage, using the same check that refuses it work, and leaves out a relay with no task that has been answered "no work for you" for two minutes. A task or fix-pr task handed to the relay counts it again at once. diff --git a/src/docs/agent-configuration.md b/src/docs/agent-configuration.md index a454f76006..5d5ec4cbd8 100644 --- a/src/docs/agent-configuration.md +++ b/src/docs/agent-configuration.md @@ -1382,8 +1382,11 @@ the resident agent would flap on and off with every socket drop. A relay counts as covering the role only when all of these hold: +- it is acting as the role: its effective role, after any owner "Acting as" + assignment, is that role; - its heartbeat is fresh; -- it is not in a failure-streak pause; +- it is signed in as a contributor with a GitHub login; +- contributions are not suspended (`hub.contribute_suspended`); - it is not in standby mode (`just contribute-standby`). A standby relay never asks for ordinary work; it only takes what you dispatch to it by hand; - it has not reported a quota hold. A relay whose quota guard is holding stops @@ -1391,13 +1394,24 @@ A relay counts as covering the role only when all of these hold: tells the hub (`quota_hold_v1`, see [the contributor relay guide](contributor-relay.md#a-held-relay-tells-the-hub-quota_hold_v1)); an older relay is still counted while held; +- it is not in a failure-streak pause; - it has not used up its hourly or daily task allowance (`hub.tier_limits..max_per_hour` / `max_per_day` for the contributor's trust tier). The hub refuses such a relay work (`hourly_limit` / `daily_limit`) until the window rolls, so it covers nothing in the meantime. Coverage and task assignment use the same check, so they always agree on which limits apply and how they are counted; -- contributions are not suspended (`hub.contribute_suspended`). +- its access is not revoked and its trust tier is not in + `hub.disabled_tiers`. The hub refuses such a relay work (`tier_disabled`), + using the same check as coverage, and covers again as soon as you re-enable + the tier; +- it is not idle on an empty queue: a relay holding no task whose every + request for work has been answered `no_matching_work` (or `repo_unmintable`, + the same answer when the hub had to skip a repository it cannot get a token + for) for two minutes is connected but working nothing, so it stops covering. + An idle relay asks again about every 30 seconds, so one empty answer between + two tasks never drops it. A task or a fix-pr task handed to the relay ends + the idle stretch at once. A relay that is busy on a task still counts, including one holding its tier's `max_concurrent` tasks. diff --git a/src/docs/design/agent-state-inventory.md b/src/docs/design/agent-state-inventory.md index f228d928e2..75819f97a8 100644 --- a/src/docs/design/agent-state-inventory.md +++ b/src/docs/design/agent-state-inventory.md @@ -180,7 +180,7 @@ key generations (`/data/saas/hub-generations.json`). | # | State | Where (v5) | On a spoke (dashboard) pod roll | Verdict | |---|-------|------------|--------------------------------|---------| -| 27 | Completed-task ledger + per-task cooldowns + PR URLs | `completedTasks` / `completedTaskCooldown` / `completedTaskPRURL` (`contribute_ws.go:965`), persisted to `/data/contributors/completed-tasks.json` | Survives; was moved to the PVC ledger through the #3792 → #3989 lineage after re-offer loops | **fixed** | +| 27 | Completed-task ledger + per-task cooldowns + PR URLs | `completedTasks` / `completedTaskCooldown` / `completedTaskPRURL` (`contribute_ws.go:975`), persisted to `/data/contributors/completed-tasks.json` | Survives; was moved to the PVC ledger through the #3792 → #3989 lineage after re-offer loops | **fixed** | | 28 | Failure cooldowns + quarantine counters | `failedTasks` / `consecutiveFailures` (`:373`, `:381`), persisted to `/data/contributors/failed-tasks.json` | Survives (#2435 livelock fix, made durable with the ledger) | **fixed** | | 29 | No-PR completion streaks | `noPRStreaks` (`:394`), `/data/contributors/no-pr-streaks.json` | Survives (#3980; geometric backoff must outlive the completion entry it escalates) | **fixed** | | 30 | `no_work_needed` verdicts | `noWorkVerdicts` (`:416`), `/data/contributors/no-work-verdicts.json` | Survives (#3987/#3997) — "persisted in the same PVC-backed ledger dir as the cooldowns so a pod restart does not forget the verdict" | **fixed** | diff --git a/src/pkg/dashboard/contribute_coverage.go b/src/pkg/dashboard/contribute_coverage.go index a4e7d037d4..ee19abdf91 100644 --- a/src/pkg/dashboard/contribute_coverage.go +++ b/src/pkg/dashboard/contribute_coverage.go @@ -12,7 +12,7 @@ import ( // agent role right now, keyed by role, for the governor's contributor-first // gate (`agents..contributor_first`; see pkg/governor/contributor_first.go). // -// A relay counts as covering a role when: +// A relay counts as covering a role when ALL of these hold: // // - its EFFECTIVE role is that role — the owner's "Acting as" assignment // after it has overridden whatever the relay requested, which is what @@ -20,8 +20,11 @@ import ( // - its heartbeat is fresh, by the same wsHeartbeatTimeout rule // FleetSnapshot uses to mark a relay stale, so a relay whose socket died // silently cannot keep the resident agent idle; -// - it is not in a failure-streak pause: a relay whose runtime keeps dying -// is connected but is not doing the lane's work; +// - it is signed in as a contributor with a GitHub login; +// - the operator has not suspended contributions (hub.contribute_suspended). +// The suspend switch refuses work to every relay while their sockets stay +// up, so a connected relay is then covering nothing; reporting it as +// coverage would turn "pause contributors" into "pause the lane"; // - it has not declared standby (standby_declare, accepted or not). A // standby relay stays connected but never asks for ordinary work; it only // takes what an owner dispatches to it by hand, so it drains nothing; @@ -31,24 +34,35 @@ import ( // Observed on 2026-11-08: a relay held at 15% of its weekly window kept // the scanner lane "covered" while nobody worked it. A relay too old to // send quota_hold is still counted, as before; +// - it is not in a failure-streak pause: a relay whose runtime keeps dying +// is connected but is not doing the lane's work; // - its identity has not used up its hourly or daily task allowance // (hub.tier_limits max_per_hour / max_per_day for its trust tier). Task // selection refuses such a relay with hourly_limit / daily_limit until the // window rolls, so it takes nothing in the meantime. The check is // tierRateLimitReached, the same function selection refuses with, so the // two can never disagree about which limits apply or how they are counted; -// - the operator has not suspended contributions (hub.contribute_suspended). -// The suspend switch refuses work to every relay while their sockets stay -// up, so a connected relay is then covering nothing; reporting it as -// coverage would turn "pause contributors" into "pause the lane". +// - its profile is not revoked and its trust tier is not in +// hub.disabled_tiers. The check is tierAccessGateReason, the rule a resume +// is refused with and whose disabled-tier half (tierDisabled) refuses a +// fresh assignment, so a relay the hub will not hand work to is never +// reported as working the lane; +// - it has not been answered no_matching_work (or repo_unmintable, its +// re-scan form) continuously for coverageNoWorkWindow while holding no +// task. Such a relay is connected and idle: the queue has nothing it may +// take. Observed on 2026-11-08: a relay covered the scanner lane while +// every ready it sent was answered no_matching_work. One empty answer +// between two tasks does not drop it (see coverageNoWorkWindow). // // A relay that is busy on a task still counts — it is draining the lane, -// which is the point. Standby, quota hold and a used-up allowance do not wait -// for the task to end: each means the relay will not take the lane's NEXT +// which is the point. That includes a relay working only a fix-pr task, and a +// relay whose last answer was no_matching_work but which has been handed work +// since. Standby, quota hold, a used-up allowance and a closed tier do not +// wait for the task to end: each means the relay will not take the lane's NEXT // item, which is what the governor is deciding whether to withhold. (Holding -// max_concurrent tasks is different: that relay is busy, and counts.) Whether the relay is ALLOWED the role (trust tier, -// grants, the delegatable list) was settled when the hub set c.role; nothing -// is re-derived here. +// max_concurrent tasks is different: that relay is busy, and counts.) Whether +// the relay is ALLOWED the role (trust tier, grants, the delegatable list) was +// settled when the hub set c.role; nothing is re-derived here. func (h *ContributeWSHub) RoleCoverage(now time.Time) map[string]governor.ContributorCoverage { if h == nil || h.contributionsSuspended() { return nil @@ -71,8 +85,9 @@ func (h *ContributeWSHub) RoleCoverage(now time.Time) map[string]governor.Contri tier = c.profile.TrustTier } notTakingWork := c.standbyDeclared || len(c.standby) > 0 || c.quotaHeld + queueEmpty := c.noWorkForWindowLocked(now) c.mu.Unlock() - if role == "" || !fresh || login == "" || notTakingWork { + if role == "" || !fresh || login == "" || notTakingWork || queueEmpty { continue } candidates = append(candidates, candidate{role: role, login: login, identity: identityOf(c), tier: tier}) @@ -83,9 +98,13 @@ func (h *ContributeWSHub) RoleCoverage(now time.Time) map[string]governor.Contri // their own lock (failureStreakMu, rateMu); consult them with h.mu // released so neither is ever nested inside it. The rate check is a scan // of at most a day of assignment stamps for one identity, once per - // candidate per governor evaluation, and never takes selectMu. + // candidate per governor evaluation, and never takes selectMu. The tier + // gate reads only the config, from the tier copied out under c.mu above. byRole := make(map[string]map[string]struct{}) for _, cand := range candidates { + if h.tierAccessGateReason(cand.tier) != "" { + continue + } if paused, _, _ := h.contributorFailureStreakActive(cand.identity, now); paused { continue } @@ -112,6 +131,65 @@ func (h *ContributeWSHub) RoleCoverage(now time.Time) map[string]governor.Contri return out } +// coverageNoWorkWindow is how long a relay holding no task must have been +// answered nothing but no_matching_work / repo_unmintable before RoleCoverage +// stops counting it. +// +// Two minutes, measured from the FIRST answer of the run. An idle relay asks +// again 30 s after each task_unavailable (TASK_UNAVAILABLE_RETRY_MS in +// bin/contributor-relay.js), so that is five empty answers in a row: one +// empty answer between two tasks, with work arriving on the next ask, never +// reaches it. It is a time, not an answer count, so a relay that asks less +// often (or stops asking) is judged on how long it has been idle, not on how +// many times it happened to ask. It does not need to be longer to stop the +// resident agent flapping: RoleCoverage is sampled once per governor +// evaluation (governor.eval_interval_s, 300 s by default, sooner on webhook +// triggers), and a role that drops out of the report is still withheld for +// contributor_first_grace (10 min by default) before the resident agent is +// kicked, so a relay that finds work again inside the grace keeps the lane. +const coverageNoWorkWindow = 2 * time.Minute + +// idleRunKey folds the task_unavailable reasons that mean "nothing in the queue +// for this relay" into one run: the re-scan wrapper in selectTask renames +// no_matching_work to repo_unmintable when it dropped an unmintable repo on the +// way, so the two can alternate on consecutive readies with nothing changing. +func idleRunKey(reason string) string { + if reason == taskUnavailableRepoUnmintable { + return taskUnavailableNoMatchingWork + } + return reason +} + +// noteIdleReasonLocked records reason as the task_unavailable answer just sent +// to c, starting a new run unless it continues the current one. The caller +// holds c.mu. +func (c *ContributorConnection) noteIdleReasonLocked(reason string, now time.Time) { + if c.lastIdleReasonSince.IsZero() || idleRunKey(reason) != idleRunKey(c.lastIdleReason) { + c.lastIdleReasonSince = now + } + c.lastIdleReason = reason +} + +// clearIdleReasonLocked ends the idle run: c has been handed work. The caller +// holds c.mu. +func (c *ContributorConnection) clearIdleReasonLocked() { + c.lastIdleReason = "" + c.lastIdleReasonSince = time.Time{} +} + +// noWorkForWindowLocked reports whether c holds no task (primary or fix-pr) +// and has been answered no_matching_work / repo_unmintable continuously for at +// least coverageNoWorkWindow at now. The caller holds c.mu. +func (c *ContributorConnection) noWorkForWindowLocked(now time.Time) bool { + if c.currentTask != nil || len(c.fixTasks) > 0 { + return false + } + if c.lastIdleReasonSince.IsZero() || idleRunKey(c.lastIdleReason) != taskUnavailableNoMatchingWork { + return false + } + return now.Sub(c.lastIdleReasonSince) >= coverageNoWorkWindow +} + // contributionsSuspended mirrors the task-selection gate in // contribute_select.go: with hub.contribute_suspended set, no relay is handed // work, whatever role it holds. diff --git a/src/pkg/dashboard/contribute_coverage_test.go b/src/pkg/dashboard/contribute_coverage_test.go index 4d89edf844..cc5b53ea64 100644 --- a/src/pkg/dashboard/contribute_coverage_test.go +++ b/src/pkg/dashboard/contribute_coverage_test.go @@ -7,6 +7,7 @@ import ( "time" "github.com/hivecommons/hive/pkg/config" + "github.com/hivecommons/hive/pkg/fixqueue" "github.com/hivecommons/hive/pkg/governor" standbypkg "github.com/hivecommons/hive/pkg/standby" ) @@ -354,3 +355,262 @@ func TestTierRateLimitReached(t *testing.T) { t.Fatalf("nil hub: tierRateLimitReached = %+v, want zero", got) } } + +// A relay whose trust tier the operator listed in hub.disabled_tiers is refused +// every fresh assignment with tier_disabled and every resume too, while its +// socket stays up. It must not keep the resident agent withheld; once the tier +// is re-enabled, the same connection covers again. +func TestRoleCoverage_DisabledTierDoesNotCover(t *testing.T) { + hub, s := covK2Hub(t) + now := time.Now() + relay := coverageConn("alice", "scanner", now) + hub.mu.Lock() + hub.connections["alice"] = relay + hub.mu.Unlock() + + s.deps.Config.Hub.DisabledTiers = []string{"newcomer", "contributor"} + if got := selectionRefusal(t, hub, s, relay); got != taskUnavailableTierDisabled { + t.Fatalf("setup: selection refused with %q, want %q", got, taskUnavailableTierDisabled) + } + if got := hub.resumeGateReason(relay); got != taskUnavailableTierDisabled { + t.Fatalf("setup: resumeGateReason = %q, want %q", got, taskUnavailableTierDisabled) + } + if got := hub.RoleCoverage(now); got != nil { + t.Fatalf("relay in a disabled tier counted as cover: RoleCoverage = %+v, want nil", got) + } + + s.deps.Config.Hub.DisabledTiers = []string{"newcomer"} + want := map[string]governor.ContributorCoverage{"scanner": {Contributors: []string{"alice"}}} + if got := hub.RoleCoverage(now); !reflect.DeepEqual(got, want) { + t.Fatalf("tier re-enabled: RoleCoverage = %+v, want %+v", got, want) + } +} + +// A revoked profile is refused a resume by the same gate. A connection that +// still carries one covers nothing. +func TestRoleCoverage_RevokedDoesNotCover(t *testing.T) { + hub, _ := covK2Hub(t) + now := time.Now() + relay := coverageConn("alice", "scanner", now) + relay.profile.TrustTier = contributorTierRevoked + hub.mu.Lock() + hub.connections["alice"] = relay + hub.connections["bob"] = coverageConn("bob", "scanner", now) + hub.mu.Unlock() + + if got := hub.resumeGateReason(relay); got != resumeRefusedAccessRevoked { + t.Fatalf("setup: resumeGateReason = %q, want %q", got, resumeRefusedAccessRevoked) + } + want := map[string]governor.ContributorCoverage{"scanner": {Contributors: []string{"bob"}}} + if got := hub.RoleCoverage(now); !reflect.DeepEqual(got, want) { + t.Fatalf("revoked relay counted as cover: RoleCoverage = %+v, want %+v", got, want) + } +} + +// tierAccessGateReason is the rule resumeGateReason, selection's disabled-tier +// refusal and RoleCoverage share. +func TestTierAccessGateReason(t *testing.T) { + hub, s := covK2Hub(t) + s.deps.Config.Hub.DisabledTiers = []string{"newcomer"} + for tier, want := range map[string]string{ + "contributor": "", + "newcomer": taskUnavailableTierDisabled, + contributorTierRevoked: resumeRefusedAccessRevoked, + } { + if got := hub.tierAccessGateReason(tier); got != want { + t.Errorf("tierAccessGateReason(%q) = %q, want %q", tier, got, want) + } + } + var none *ContributeWSHub + if got := none.tierAccessGateReason("newcomer"); got != "" { + t.Errorf("nil hub: tierAccessGateReason = %q, want empty", got) + } + if got := none.tierAccessGateReason(contributorTierRevoked); got != resumeRefusedAccessRevoked { + t.Errorf("nil hub: a revoked tier is still refused, got %q", got) + } +} + +// noWorkRelay connects a scanner relay over a real socket to a hub whose queue +// is empty, so every ready it sends is answered by real task selection. +func noWorkRelay(t *testing.T) (*ContributeWSHub, *Server, *ContributorConnection, *wsSession, func() map[string]any) { + t.Helper() + hub, s := covK2Hub(t) + s.statusMu.Lock() + s.status = &StatusPayload{} + s.statusMu.Unlock() + server, client := wsPipe(t) + relay := coverageConn("alice", "scanner", time.Now()) + relay.ws = server + hub.mu.Lock() + hub.connections["alice"] = relay + hub.mu.Unlock() + sess := &wsSession{h: hub, conn: server, contributor: relay} + return hub, s, relay, sess, func() map[string]any { return readWire(t, client) } +} + +// sendReady drives one ready through the real handler and returns the frame +// the relay received. +func sendReady(t *testing.T, sess *wsSession, read func() map[string]any) map[string]any { + t.Helper() + if stop := sess.handleReady(WSMessage{Type: "ready"}); stop { + t.Fatal("ready must not stop the session") + } + return read() +} + +// coverageAt evaluates RoleCoverage at a moment the relay's heartbeat is +// fresh, so only the idle run decides. +func coverageAt(hub *ContributeWSHub, relay *ContributorConnection, at time.Time) map[string]governor.ContributorCoverage { + relay.mu.Lock() + relay.lastPong = at + relay.mu.Unlock() + return hub.RoleCoverage(at) +} + +func idleSince(relay *ContributorConnection) time.Time { + relay.mu.Lock() + defer relay.mu.Unlock() + return relay.lastIdleReasonSince +} + +var scannerAlice = map[string]governor.ContributorCoverage{"scanner": {Contributors: []string{"alice"}}} + +// One empty answer is what a relay gets between two tasks when the queue is +// momentarily dry. It must not drop the relay's cover. +func TestRoleCoverage_OneNoMatchingWorkAnswerStillCovers(t *testing.T) { + hub, _, relay, sess, read := noWorkRelay(t) + if f := sendReady(t, sess, read); f["type"] != "task_unavailable" || f["reason"] != taskUnavailableNoMatchingWork { + t.Fatalf("setup: expected task_unavailable/%s, got %v", taskUnavailableNoMatchingWork, f) + } + since := idleSince(relay) + if since.IsZero() { + t.Fatal("the no_matching_work answer must start an idle run on the connection") + } + // The next ready, ~30 s on, is still inside the window. + if got := coverageAt(hub, relay, since.Add(30*time.Second)); !reflect.DeepEqual(got, scannerAlice) { + t.Fatalf("one empty answer dropped cover: RoleCoverage = %+v, want %+v", got, scannerAlice) + } +} + +// A relay answered no_matching_work on every ready for the whole window is +// connected and idle: it covers nothing. Observed on 2026-11-08 on +// hive-wild-mole, where such a relay kept the scanner lane "covered". A task +// assigned afterwards ends the run, so it covers again — while it works the +// task and after it finishes it. +func TestRoleCoverage_NoMatchingWorkForWindowDoesNotCover(t *testing.T) { + hub, s, relay, sess, read := noWorkRelay(t) + sendReady(t, sess, read) + since := idleSince(relay) + for i := 0; i < 3; i++ { + if f := sendReady(t, sess, read); f["reason"] != taskUnavailableNoMatchingWork { + t.Fatalf("setup: ready %d answered %v, want %s", i+2, f, taskUnavailableNoMatchingWork) + } + } + if got := idleSince(relay); !got.Equal(since) { + t.Fatalf("consecutive no_matching_work answers must extend one run: since moved %v -> %v", since, got) + } + if got := coverageAt(hub, relay, since.Add(coverageNoWorkWindow-time.Second)); !reflect.DeepEqual(got, scannerAlice) { + t.Fatalf("inside the window: RoleCoverage = %+v, want %+v", got, scannerAlice) + } + later := since.Add(coverageNoWorkWindow) + if got := coverageAt(hub, relay, later); got != nil { + t.Fatalf("relay answered no_matching_work for the whole window counted as cover: RoleCoverage = %+v, want nil", got) + } + + // Work arrives: the next ready is handed a task. + oneActionableIssue(s) + if f := sendReady(t, sess, read); f["type"] != "task_assign" { + t.Fatalf("setup: expected task_assign once work exists, got %v", f) + } + if got := coverageAt(hub, relay, later); !reflect.DeepEqual(got, scannerAlice) { + t.Fatalf("relay working an assigned task: RoleCoverage = %+v, want %+v", got, scannerAlice) + } + if got := idleSince(relay); !got.IsZero() { + t.Fatalf("an assignment must end the idle run, since = %v", got) + } + relay.mu.Lock() + relay.currentTask = nil // the task finished; no new ready yet + relay.mu.Unlock() + if got := coverageAt(hub, relay, later.Add(time.Hour)); !reflect.DeepEqual(got, scannerAlice) { + t.Fatalf("relay between tasks after an assignment: RoleCoverage = %+v, want %+v", got, scannerAlice) + } +} + +// repo_unmintable is the name selectTask's re-scan gives no_matching_work when +// it dropped an unmintable repo on the way: nothing in the queue may be handed +// to this relay either way. It drops cover like no_matching_work does, and the +// two continue one run. +func TestRoleCoverage_RepoUnmintableBehavesLikeNoMatchingWork(t *testing.T) { + hub, s, relay, sess, read := noWorkRelay(t) + sendReady(t, sess, read) // no_matching_work: the queue is empty + since := idleSince(relay) + + oneActionableIssue(s) + hub.markRepoUnmintable("myorg/repo1", time.Now()) + for i := 0; i < 2; i++ { + if f := sendReady(t, sess, read); f["type"] != "task_unavailable" || f["reason"] != taskUnavailableRepoUnmintable { + t.Fatalf("setup: expected task_unavailable/%s, got %v", taskUnavailableRepoUnmintable, f) + } + } + if got := idleSince(relay); !got.Equal(since) { + t.Fatalf("repo_unmintable after no_matching_work must continue the run: since moved %v -> %v", since, got) + } + if got := coverageAt(hub, relay, since.Add(30*time.Second)); !reflect.DeepEqual(got, scannerAlice) { + t.Fatalf("inside the window: RoleCoverage = %+v, want %+v", got, scannerAlice) + } + if got := coverageAt(hub, relay, since.Add(coverageNoWorkWindow)); got != nil { + t.Fatalf("relay answered repo_unmintable for the whole window counted as cover: RoleCoverage = %+v, want nil", got) + } + + // A run of repo_unmintable alone drops cover too. + relay.mu.Lock() + relay.clearIdleReasonLocked() + relay.mu.Unlock() + sendReady(t, sess, read) + if got := coverageAt(hub, relay, idleSince(relay).Add(coverageNoWorkWindow)); got != nil { + t.Fatalf("repo_unmintable-only run counted as cover: RoleCoverage = %+v, want nil", got) + } +} + +// A different answer ends the no-work run: a relay refused for another reason +// is judged by that reason's own rule, not by this window. +func TestRoleCoverage_OtherIdleReasonEndsNoWorkRun(t *testing.T) { + _, _, relay, _, _ := noWorkRelay(t) + start := time.Now() + relay.mu.Lock() + relay.noteIdleReasonLocked(taskUnavailableNoMatchingWork, start) + relay.noteIdleReasonLocked(taskUnavailableHubNotReady, start.Add(time.Minute)) + relay.noteIdleReasonLocked(taskUnavailableNoMatchingWork, start.Add(90*time.Second)) + got := relay.lastIdleReasonSince + relay.mu.Unlock() + if !got.Equal(start.Add(90 * time.Second)) { + t.Fatalf("a run broken by another reason must restart, since = %v, want %v", got, start.Add(90*time.Second)) + } +} + +// A dispatched fix-pr task is work too: it ends the idle run, and the relay +// covers while the fix runs and after it settles. +func TestRoleCoverage_FixTaskDispatchEndsNoWorkRun(t *testing.T) { + hub, _, relay, sess, read := noWorkRelay(t) + sendReady(t, sess, read) + later := idleSince(relay).Add(coverageNoWorkWindow) + if got := coverageAt(hub, relay, later); got != nil { + t.Fatalf("setup: idle relay must not cover, got %+v", got) + } + + relay.mu.Lock() + relay.ws = nil // sendFixTask pushes only to a live socket; the frame is not under test + relay.mu.Unlock() + hub.sendFixTask(relay, "alice", &fixqueue.Dispatch{ID: "fix-1", Mode: fixqueue.ModeBoundary, Items: []fixqueue.Item{ + {Repo: "myorg/repo1", Number: 7, URL: "https://github.com/myorg/repo1/pull/7", Title: "red PR"}, + }}) + if got := coverageAt(hub, relay, later); !reflect.DeepEqual(got, scannerAlice) { + t.Fatalf("relay working a fix task: RoleCoverage = %+v, want %+v", got, scannerAlice) + } + relay.mu.Lock() + delete(relay.fixTasks, "fix-1") + relay.mu.Unlock() + if got := coverageAt(hub, relay, later.Add(time.Hour)); !reflect.DeepEqual(got, scannerAlice) { + t.Fatalf("relay after its fix settled: RoleCoverage = %+v, want %+v", got, scannerAlice) + } +} diff --git a/src/pkg/dashboard/contribute_fixqueue.go b/src/pkg/dashboard/contribute_fixqueue.go index 787f819da1..673aca1627 100644 --- a/src/pkg/dashboard/contribute_fixqueue.go +++ b/src/pkg/dashboard/contribute_fixqueue.go @@ -448,6 +448,8 @@ func (h *ContributeWSHub) sendFixTask(c *ContributorConnection, user string, d * c.fixTasksAt = map[string]time.Time{} } c.fixTasksAt[d.ID] = time.Now() + // A dispatched fix is work: the relay is no longer idle on an empty queue. + c.clearIdleReasonLocked() c.mu.Unlock() msg := WSMessage{ Type: "task_assign", Seq: h.nextSeq(), TaskID: d.ID, TaskGen: h.nextTaskGen(), diff --git a/src/pkg/dashboard/contribute_select.go b/src/pkg/dashboard/contribute_select.go index b0c804acd3..f76f593793 100644 --- a/src/pkg/dashboard/contribute_select.go +++ b/src/pkg/dashboard/contribute_select.go @@ -377,11 +377,15 @@ func (h *ContributeWSHub) canonicalRepoKey(repo string) string { func (h *ContributeWSHub) resumeGateReason(c *ContributorConnection) string { tier := "" if c != nil && c.profile != nil { - if c.profile.TrustTier == "revoked" { - return "contribution access revoked" - } tier = c.profile.TrustTier } + // The per-contributor half (revoked, disabled tier) is tierAccessGateReason, + // which RoleCoverage shares. A revoked profile outranks every other reason, + // as it always has; a disabled tier is reported after the hub checks. + tierReason := h.tierAccessGateReason(tier) + if tierReason == resumeRefusedAccessRevoked { + return tierReason + } if h.server == nil { return taskUnavailableHubNotReady } @@ -389,15 +393,51 @@ func (h *ContributeWSHub) resumeGateReason(c *ContributorConnection) string { if h.server.deps.Config.Hub.ContributeSuspended { return taskUnavailableContributionSuspended } - for _, dt := range h.server.deps.Config.Hub.DisabledTiers { - if dt == tier { - return taskUnavailableTierDisabled - } - } + } + return tierReason +} + +// resumeRefusedAccessRevoked is the reason resumeGateReason gives for a revoked +// contributor profile. +const resumeRefusedAccessRevoked = "contribution access revoked" + +// tierAccessGateReason is the one rule for "this trust tier may not be handed +// work": a revoked profile (resumeRefusedAccessRevoked) or a tier the operator +// listed in hub.disabled_tiers (taskUnavailableTierDisabled). "" means the tier +// is open. resumeGateReason refuses a resume with it, selectTaskPass refuses a +// fresh assignment with its disabled-tier half (tierDisabled; selection never +// checked revoked, which is refused at auth), and RoleCoverage uses it so a +// relay that would be refused for either reason does not count as +// contributor-first cover. +// +// It takes a tier rather than a connection so callers read the tier under c.mu +// and evaluate with every lock released. It reads the config without a lock, +// as selection does. +func (h *ContributeWSHub) tierAccessGateReason(tier string) string { + if tier == contributorTierRevoked { + return resumeRefusedAccessRevoked + } + if h.tierDisabled(tier) { + return taskUnavailableTierDisabled } return "" } +// tierDisabled reports whether the operator listed tier in hub.disabled_tiers +// (#2436 finding 2). It is the disabled-tier half of tierAccessGateReason, and +// what selectTaskPass refuses a fresh assignment with. +func (h *ContributeWSHub) tierDisabled(tier string) bool { + if h == nil || h.server == nil || h.server.deps == nil || h.server.deps.Config == nil { + return false + } + for _, dt := range h.server.deps.Config.Hub.DisabledTiers { + if dt == tier { + return true + } + } + return false +} + // mintFailureRepoCooldown is how long a repository stays excluded from selection // after the App refused to mint a token scoped to it (#7869). A renamed repo // stays broken until the operator fixes the repo list, so the block is long @@ -551,16 +591,15 @@ func (h *ContributeWSHub) selectTaskPass(c *ContributorConnection, skippedUnmint if c.profile != nil { tier = c.profile.TrustTier } - if h.server.deps != nil && h.server.deps.Config != nil { - for _, dt := range h.server.deps.Config.Hub.DisabledTiers { - if dt == tier { - h.logger.Warn("[contribute-ws] refusing task: tier disabled", - "username", identityOf(c), "tier", tier) - h.recordDecision(decisionUsername(c), decisionRefused, "", "", 0, - "tier "+tier+" is disabled on this hive") - return h.taskUnavailable(taskUnavailableTierDisabled) - } - } + // The disabled-tier rule is tierDisabled, which RoleCoverage shares through + // tierAccessGateReason, so a relay refused here is never reported as + // covering its role. + if h.tierDisabled(tier) { + h.logger.Warn("[contribute-ws] refusing task: tier disabled", + "username", identityOf(c), "tier", tier) + h.recordDecision(decisionUsername(c), decisionRefused, "", "", 0, + "tier "+tier+" is disabled on this hive") + return h.taskUnavailable(taskUnavailableTierDisabled) } // activeIssues tracks issues already held by SOME live connection so we do @@ -1233,7 +1272,7 @@ func (h *ContributeWSHub) selectTaskPass(c *ContributorConnection, skippedUnmint // rollbackAssignment hands back to unrecordAssignment. c.taskAssignedAt = assignedAt c.currentLabels = chosen.labels - c.lastIdleReason = "" + c.clearIdleReasonLocked() // The prompt and the credential are not known yet — see below. A stale // pendingToken from a previous task must not survive into this claim. c.currentPrompt = "" diff --git a/src/pkg/dashboard/contribute_standby_dispatch.go b/src/pkg/dashboard/contribute_standby_dispatch.go index d3c82a14ac..a68a32459d 100644 --- a/src/pkg/dashboard/contribute_standby_dispatch.go +++ b/src/pkg/dashboard/contribute_standby_dispatch.go @@ -301,7 +301,7 @@ func (h *ContributeWSHub) assignStandbyTask(c *ContributorConnection, item Ready c.lastLeaseRenew = assignedAt c.taskAssignedAt = assignedAt c.currentLabels = item.Labels - c.lastIdleReason = "" + c.clearIdleReasonLocked() c.currentPrompt = "" c.pendingToken = "" c.credentialDelivered = false diff --git a/src/pkg/dashboard/contribute_ws.go b/src/pkg/dashboard/contribute_ws.go index b7e908a2ad..658f544127 100644 --- a/src/pkg/dashboard/contribute_ws.go +++ b/src/pkg/dashboard/contribute_ws.go @@ -202,8 +202,18 @@ type ContributorConnection struct { // connection (#2546): one of the taskUnavailable* reasons. It lets the ops tab // show WHY a connected clanker is idle (suspended vs hub-not-ready vs // no-matching-work vs an enforced refusal) instead of an indistinguishable - // silence. Cleared when a task is actually assigned. Purely diagnostic. + // silence. Cleared with lastIdleReasonSince below. It never routes or gates + // a work item; its one gating reader is RoleCoverage, which uses the pair to + // tell the governor a relay is not working its lane. lastIdleReason string + // lastIdleReasonSince is when the current run of lastIdleReason began: the + // first of the consecutive task_unavailable answers that carried it, with + // no_matching_work and repo_unmintable counted as one run (idleRunKey). + // Set by noteIdleReasonLocked; zeroed with lastIdleReason by + // clearIdleReasonLocked when a task is assigned or adopted on resume, or a + // fix-pr task is dispatched. RoleCoverage stops counting a relay that has + // had nothing but "no work for you" for coverageNoWorkWindow. + lastIdleReasonSince time.Time // capabilities is the client-declared runtime posture from auth_response // (#2547 declare half): container runtime, OS/arch, agent/relay versions, // credential type. Nil when the client declared nothing (an unversioned @@ -2245,7 +2255,7 @@ func (h *ContributeWSHub) yankAndReassign(targets []releaseTarget, reason, revok // the ready handler does. if msg != nil { tgt.conn.mu.Lock() - tgt.conn.lastIdleReason = msg.Reason + tgt.conn.noteIdleReasonLocked(msg.Reason, time.Now()) tgt.conn.mu.Unlock() } continue @@ -3765,7 +3775,7 @@ func (s *wsSession) handleReady(msg WSMessage) (stop bool) { // no_matching_work). Record the reason on the connection so the ops // tab can show WHY this clanker is idle, then send it. s.contributor.mu.Lock() - s.contributor.lastIdleReason = task.Reason + s.contributor.noteIdleReasonLocked(task.Reason, time.Now()) s.contributor.mu.Unlock() if err := s.contributor.send(*task); err != nil { h.logger.Warn("[contribute-ws] failed to send task_unavailable", "error", err) @@ -4182,6 +4192,7 @@ func (s *wsSession) handleTaskProgress(msg WSMessage) { s.contributor.mu.Lock() s.contributor.currentTask = rebuilt s.contributor.currentTaskGen = lease.gen + s.contributor.clearIdleReasonLocked() s.contributor.lastLeaseRenew = time.Now() s.contributor.tmuxOutput = msg.TmuxOutput s.contributor.tmuxOutputTask = lease.taskID diff --git a/src/pkg/governor/contributor_first.go b/src/pkg/governor/contributor_first.go index 9013c30a93..0d815f3905 100644 --- a/src/pkg/governor/contributor_first.go +++ b/src/pkg/governor/contributor_first.go @@ -22,7 +22,8 @@ import ( // contributor_first inverts the order for one lane. While at least one // connected, non-stale, non-failure-paused relay is acting as the role and // taking work (not in standby, not quota-held, not at its hourly or daily -// task limit; see RoleCoverage in pkg/dashboard), and the operator has not suspended contributions, the +// task limit, not revoked or in a disabled tier, not idle on an empty queue; +// see RoleCoverage in pkg/dashboard), and the operator has not suspended contributions, the // governor WITHHOLDS the resident agent's kicks — at exactly the gates where it // withholds them for an exhausted budget. The relay is the primary worker; the // resident agent is the fallback, and it resumes on its normal cadence once no