From e74d45e2e420853f3996e03a1290fbcf19c47102 Mon Sep 17 00:00:00 2001 From: Danathar Date: Thu, 17 Sep 2026 22:39:18 -0400 Subject: [PATCH 01/17] =?UTF-8?q?=F0=9F=90=9B=20fix(dashboard):=20keep=20t?= =?UTF-8?q?he=20sweep's=20own=20reason=20on=20a=20blocked=20PR=20pill=20(#?= =?UTF-8?q?7516)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A PR GitHub reports as not mergeable hovered as the bare GitHub enum — "not mergeable on GitHub (blocked)" — even when classifyMergeEligibility had just been handed the exact gate that made GitHub say blocked ("CI failing: build, lint", "awaiting review approval", "held: …") and threw it away in blockedOrOutstanding. On a protected branch that discarded reason is the one that mattered, so the operator had to open the PR on GitHub — the trip the pill was meant to save. Step 1 of hivecommons/hive#7515, no new API calls: - blockedOrOutstanding keeps the sweep's reason for MergeableNo: "blocked — CI failing: build, lint", "blocked — awaiting review approval", "blocked — held: …", "blocked — intent verification: …", "blocked — CI pending". - The review gate now runs before the GitHub-says-no return, so a PR blocked for want of a review carries that reason. Both paths file a MergeableNo PR in the skip bucket; only the wording changes. - When every sweep gate passes and GitHub still says blocked, say "blocked — all sweep gates pass; a branch-protection rule is unsatisfied" until step 2 can name the rule. - Plain-language wording for the rest: "has merge conflicts with v4 — needs a rebase", "behind v4 — needs an update from the base branch", "draft — mark ready for review to enter the sweep", "mergeability not yet computed by GitHub — re-checked next tick". - PullRequest gains BaseRef (base_ref), read from the list payload the enumeration already has, so the wording can name the base branch. The frontend needs no change: prMergeNote already shows a blocked verdict's reason verbatim. Fixtures in the pill test move to the new wording and pin that "blocked — CI failing: build, lint" reaches the tooltip without GitHub's raw state appended. Tests: merge_verdict_7515_test.go pins every row of the issue's table as an exact string and fails on the parent commit (all 14 cases); the 7478 table is updated for the new wording; fetchPRs is checked to populate BaseRef. Refs #7515 Signed-off-by: Douglas Baggett Co-authored-by: Claude Opus 5 (1M context) --- .../fixed-7515-blocked-pill-sweep-reason.md | 1 + src/cmd/hive/main.go | 90 +++++++-- src/cmd/hive/merge_verdict_7478_test.go | 29 +-- src/cmd/hive/merge_verdict_7515_test.go | 189 ++++++++++++++++++ .../pr_pill_merge_verdict_7478_test.go | 19 +- src/pkg/github/client.go | 17 ++ src/pkg/github/pr_head_origin_test.go | 16 ++ 7 files changed, 322 insertions(+), 39 deletions(-) create mode 100644 changelog.d/fixed-7515-blocked-pill-sweep-reason.md create mode 100644 src/cmd/hive/merge_verdict_7515_test.go diff --git a/changelog.d/fixed-7515-blocked-pill-sweep-reason.md b/changelog.d/fixed-7515-blocked-pill-sweep-reason.md new file mode 100644 index 0000000000..188fc4f408 --- /dev/null +++ b/changelog.d/fixed-7515-blocked-pill-sweep-reason.md @@ -0,0 +1 @@ +- Dashboard: hovering a PR pill that is not green now says, in plain words, why the sweep will not merge it and what would unblock it ([#7515](https://github.com/hivecommons/hive/issues/7515)). A PR GitHub reported as not mergeable used to hover as the bare GitHub enum — `not mergeable on GitHub (blocked)` — even though the merge-eligible classifier had just been handed the exact gate that made GitHub say blocked and then dropped it. `blocked` now keeps the sweep's reason (`blocked — CI failing: build, lint`, `blocked — awaiting review approval`, `blocked — held: …`, `blocked — intent verification: …`, `blocked — CI pending`), and when every sweep gate passes it says `blocked — all sweep gates pass; a branch-protection rule is unsatisfied` instead of nothing. The other states read as what to do rather than the API name: `has merge conflicts with v4 — needs a rebase`, `behind v4 — needs an update from the base branch`, `draft — mark ready for review to enter the sweep`, `mergeability not yet computed by GitHub — re-checked next tick`. The base branch comes from the PR list payload (`base_ref`, new on the status snapshot); no extra GitHub calls. Naming the exact branch-protection rule when the sweep itself has no reason (a review GitHub requires, a required check that never reported) is the issue's step 2 and still open. diff --git a/src/cmd/hive/main.go b/src/cmd/hive/main.go index c673ca2e34..ef95c90503 100644 --- a/src/cmd/hive/main.go +++ b/src/cmd/hive/main.go @@ -9076,19 +9076,23 @@ func classifyMergeEligibility(pr github.PullRequest, held bool, fullRepo string, // blockedOrOutstanding is the verdict for a PR the sweep will not take // for a reason of its own: the state still depends on what GitHub says, // because a conflicting PR is blocked whatever else is outstanding, and - // one whose mergeability was never fetched is unknown, not amber. + // one whose mergeability was never fetched is unknown, not amber. The + // sweep's reason is kept in every state (hivecommons/hive#7515): on a + // protected branch a red required check or a missing review is exactly + // what makes GitHub say "blocked", so dropping it for the bare enum sent + // the operator to GitHub to learn what this function already knew. blockedOrOutstanding := func(reason string) github.MergeVerdict { switch pr.Mergeable { case github.MergeableNo: - return github.MergeVerdict{State: github.MergeVerdictBlocked, Reason: notMergeableReason(pr)} + return github.MergeVerdict{State: github.MergeVerdictBlocked, Reason: notMergeableReason(pr, reason)} case github.MergeableUnknown: - return github.MergeVerdict{State: github.MergeVerdictUnknown, Reason: "mergeability not yet known; " + reason} + return github.MergeVerdict{State: github.MergeVerdictUnknown, Reason: mergeabilityUnknownReason + "; " + reason} } return github.MergeVerdict{State: github.MergeVerdictOutstanding, Reason: reason} } if pr.Draft { - return mergeBucketSkip, github.MergeVerdict{State: github.MergeVerdictBlocked, Reason: "draft"}, "" + return mergeBucketSkip, github.MergeVerdict{State: github.MergeVerdictBlocked, Reason: "draft — mark ready for review to enter the sweep"}, "" } if g.enforceIntent { if v, ok := g.intentVerdicts[fmt.Sprintf("%s/%d", fullRepo, pr.Number)]; ok && v.AgentPR && !v.MergeAllowed() { @@ -9160,17 +9164,10 @@ func classifyMergeEligibility(pr github.PullRequest, held bool, fullRepo string, return mergeBucketSkip, blockedOrOutstanding("CI pending"), "" } - if pr.Mergeable == github.MergeableNo { - // A conflicting PR cannot merge no matter how green its checks - // are. Listing it as merge-eligible left the eligible count stuck - // at N forever while nothing could actually merge (console - // #23002/#23003, 2026-08-31: the only two build-gate-green PRs - // were DIRTY go.mod dependabot bumps). Conflicts are the - // rebase/needs-human path's job, not the sweep's — keep them out - // of the eligible bucket. - return mergeBucketSkip, github.MergeVerdict{State: github.MergeVerdictBlocked, Reason: notMergeableReason(pr)}, "" - } - + // The review gate runs BEFORE the GitHub-says-no return below so that a + // PR GitHub calls "blocked" for want of a review carries that reason + // (hivecommons/hive#7515). Both paths file a MergeableNo PR in the skip + // bucket, so the order changes only the verdict's wording. if g.requireReviewApproval { if !g.reviewLoaded { return mergeBucketSkip, blockedOrOutstanding("review approval required, but review-verdicts.json is unavailable"), "" @@ -9180,6 +9177,20 @@ func classifyMergeEligibility(pr github.PullRequest, held bool, fullRepo string, } } + if pr.Mergeable == github.MergeableNo { + // A conflicting PR cannot merge no matter how green its checks + // are. Listing it as merge-eligible left the eligible count stuck + // at N forever while nothing could actually merge (console + // #23002/#23003, 2026-08-31: the only two build-gate-green PRs + // were DIRTY go.mod dependabot bumps). Conflicts are the + // rebase/needs-human path's job, not the sweep's — keep them out + // of the eligible bucket. No sweep gate explains this one: for + // "blocked" that means a branch-protection rule we do not read yet + // (a review GitHub requires, a required check that never reported); + // notMergeableReason says so rather than the bare enum. + return mergeBucketSkip, github.MergeVerdict{State: github.MergeVerdictBlocked, Reason: notMergeableReason(pr, "")}, "" + } + // Eligible. The reason names what GitHub still shows outstanding that // the sweep chooses to ignore, so a green pill beside a red optional // check does not read as "all green". @@ -9197,14 +9208,49 @@ func classifyMergeEligibility(pr github.PullRequest, held bool, fullRepo string, return mergeBucketEligible, github.MergeVerdict{State: github.MergeVerdictEligible, Reason: reason}, "" } -// notMergeableReason names GitHub's own state (dirty, blocked, behind, ...) -// for a PR it reports as not mergeable; the state is what the operator has -// to resolve. -func notMergeableReason(pr github.PullRequest) string { - if pr.MergeableState != "" { - return "not mergeable on GitHub (" + pr.MergeableState + ")" +// mergeabilityUnknownReason is the verdict prefix for a PR whose +// mergeability GitHub has not computed yet (or the fetch failed); the +// classifier appends the sweep's own reason after it. +const mergeabilityUnknownReason = "mergeability not yet computed by GitHub — re-checked next tick" + +// notMergeableReason explains, in words an operator can act on, why GitHub +// reports a PR as not mergeable — what to do, not the API enum +// (hivecommons/hive#7515). sweepReason is the gate the sweep itself failed +// the PR on, or "" when every sweep gate passed: +// +// - "blocked" folds every unsatisfied branch-protection rule into one +// word. When the sweep has a reason it is almost always the rule +// ("blocked — CI failing: build"); without one, say that a rule we do +// not read is unsatisfied rather than nothing at all. +// - "dirty" and "behind" name the base branch and the fix (rebase / +// update); a sweep reason is appended, since it still stands once the +// branch is fixed. +// - Any other state falls back to naming it. +func notMergeableReason(pr github.PullRequest, sweepReason string) string { + base, from := pr.BaseRef, "the base branch" + if base == "" { + base, from = "the base branch", "it" + } + var msg string + switch pr.MergeableState { + case "blocked": + if sweepReason == "" { + return "blocked — all sweep gates pass; a branch-protection rule is unsatisfied" + } + return "blocked — " + sweepReason + case "dirty": + msg = "has merge conflicts with " + base + " — needs a rebase" + case "behind": + msg = "behind " + base + " — needs an update from " + from + case "": + msg = "not mergeable on GitHub" + default: + msg = "not mergeable on GitHub (" + pr.MergeableState + ")" + } + if sweepReason != "" { + msg += "; also " + sweepReason } - return "not mergeable on GitHub" + return msg } func writeMergeEligible(actionable *github.ActionableResult, hold github.HoldResult, org string, escalatedPRs map[string]bool, enforceIntent bool, intentVerdicts map[string]intent.Verdict, requireReviewApproval bool, requiredChecks map[string]bool, logger *slog.Logger) map[string]github.MergeVerdict { diff --git a/src/cmd/hive/merge_verdict_7478_test.go b/src/cmd/hive/merge_verdict_7478_test.go index 364d0b8d81..b5b5099b58 100644 --- a/src/cmd/hive/merge_verdict_7478_test.go +++ b/src/cmd/hive/merge_verdict_7478_test.go @@ -88,32 +88,36 @@ func TestClassifyMergeEligibility_VerdictTracksBucket(t *testing.T) { wantReason: []string{"pending", "unstable"}, }, { - name: "pending with mergeability unknown is unknown, not amber", + name: "pending with mergeability unknown is unknown, not amber, and keeps the sweep reason", pr: github.PullRequest{Number: 6, CIStatus: "pending"}, wantBucket: mergeBucketSkip, wantState: github.MergeVerdictUnknown, - wantReason: []string{"not yet known"}, + wantReason: []string{"not yet computed by GitHub", "re-checked next tick", "CI pending"}, }, { - name: "dirty is blocked and names the GitHub state", - pr: github.PullRequest{Number: 1259, Mergeable: github.MergeableNo, MergeableState: "dirty", CIStatus: "failure", FailingChecks: []string{"build"}}, + // Conflicts read as what to do, name the base branch, and keep + // the sweep's own reason — it still stands after the rebase. + name: "dirty is blocked, names the base branch and the fix, and keeps the sweep reason", + pr: github.PullRequest{Number: 1259, Mergeable: github.MergeableNo, MergeableState: "dirty", BaseRef: "v4", CIStatus: "failure", FailingChecks: []string{"build"}}, wantBucket: mergeBucketFailing, wantState: github.MergeVerdictBlocked, - wantReason: []string{"not mergeable", "dirty"}, + wantReason: []string{"merge conflicts with v4", "needs a rebase", "CI failing: build"}, }, { - name: "blocked with green CI is blocked", + // GitHub "blocked" with every sweep gate green: the sweep cannot + // name the rule yet, but must say that, not just "blocked". + name: "blocked with green CI is blocked and says a branch-protection rule is unsatisfied", pr: github.PullRequest{Number: 603, Mergeable: github.MergeableNo, MergeableState: "blocked", CIStatus: "success"}, wantBucket: mergeBucketSkip, wantState: github.MergeVerdictBlocked, - wantReason: []string{"blocked"}, + wantReason: []string{"blocked — all sweep gates pass", "branch-protection rule"}, }, { - name: "a draft is blocked", + name: "a draft is blocked and says how to enter the sweep", pr: github.PullRequest{Number: 7, Draft: true, Mergeable: yes, CIStatus: "success"}, wantBucket: mergeBucketSkip, wantState: github.MergeVerdictBlocked, - wantReason: []string{"draft"}, + wantReason: []string{"draft — mark ready for review"}, }, { name: "review approval required and missing is outstanding", @@ -143,13 +147,14 @@ func TestClassifyMergeEligibility_VerdictTracksBucket(t *testing.T) { }, { // A conflicting PR the sweep also refuses for another reason is - // still BLOCKED: the conflict is what the operator resolves first. - name: "held AND dirty is blocked, not amber", + // still BLOCKED: the conflict is what the operator resolves first + // — and the hold is still named, since it outlives the rebase. + name: "held AND dirty is blocked, not amber, and still names the hold", pr: github.PullRequest{Number: 11, Mergeable: github.MergeableNo, MergeableState: "dirty", CIStatus: "success"}, held: true, wantBucket: mergeBucketSkip, wantState: github.MergeVerdictBlocked, - wantReason: []string{"dirty"}, + wantReason: []string{"merge conflicts with the base branch", "held"}, }, } for _, tc := range cases { diff --git a/src/cmd/hive/merge_verdict_7515_test.go b/src/cmd/hive/merge_verdict_7515_test.go new file mode 100644 index 0000000000..6f4047a5b3 --- /dev/null +++ b/src/cmd/hive/merge_verdict_7515_test.go @@ -0,0 +1,189 @@ +package main + +import ( + "strings" + "testing" + + "github.com/hivecommons/hive/pkg/github" + "github.com/hivecommons/hive/pkg/intent" +) + +// hivecommons/hive#7515: hovering a non-green PR pill must say, in plain +// words, why the sweep will not merge it and what would unblock it. Before +// this, a PR GitHub reported as not mergeable got the bare GitHub enum +// ("not mergeable on GitHub (blocked)") even when the classifier had just +// been handed the exact gate — "CI failing: build, lint", "awaiting review +// approval", "held: …" — that made GitHub say blocked. These cases pin the +// wording for every row of the issue's table so the reason cannot be +// dropped again. +func TestClassifyMergeEligibility_BlockedKeepsSweepReason(t *testing.T) { + no := github.MergeableNo + blockedPR := func(n int, ci string, failing ...string) github.PullRequest { + return github.PullRequest{Number: n, Mergeable: no, MergeableState: "blocked", BaseRef: "v4", CIStatus: ci, FailingChecks: failing} + } + cases := []struct { + name string + pr github.PullRequest + held bool + gates mergeGates + wantBucket mergeBucket + wantState github.MergeVerdictState + wantReason string // exact + }{ + { + name: "required check red", + pr: blockedPR(1, "failure", "build", "lint"), + gates: mergeGates{requiredChecks: map[string]bool{"build": true, "lint": true}}, + wantBucket: mergeBucketFailing, + wantState: github.MergeVerdictBlocked, + wantReason: "blocked — CI failing: build, lint", + }, + { + // The review gate used to sit behind the GitHub-says-no return, + // so a PR blocked for want of a review never reached it. + name: "review missing", + pr: blockedPR(2, "success"), + gates: mergeGates{requireReviewApproval: true, reviewLoaded: true}, + wantBucket: mergeBucketSkip, + wantState: github.MergeVerdictBlocked, + wantReason: "blocked — awaiting review approval", + }, + { + name: "review artifact unavailable", + pr: blockedPR(3, "success"), + gates: mergeGates{requireReviewApproval: true, reviewLoaded: false}, + wantBucket: mergeBucketSkip, + wantState: github.MergeVerdictBlocked, + wantReason: "blocked — review approval required, but review-verdicts.json is unavailable", + }, + { + name: "hold label", + pr: blockedPR(4, "success"), + held: true, + wantBucket: mergeBucketSkip, + wantState: github.MergeVerdictBlocked, + wantReason: "blocked — held: a hold label keeps it out of the sweep", + }, + { + name: "intent verdict", + pr: blockedPR(5, "success"), + gates: mergeGates{enforceIntent: true, intentVerdicts: map[string]intent.Verdict{ + "org/repo/5": {AgentPR: true, Authorized: false, Reason: "no authorizing issue"}, + }}, + wantBucket: mergeBucketSkip, + wantState: github.MergeVerdictBlocked, + wantReason: "blocked — intent verification: no authorizing issue", + }, + { + // A required check still running is why GitHub says blocked. + name: "required check pending", + pr: blockedPR(6, "pending"), + wantBucket: mergeBucketSkip, + wantState: github.MergeVerdictBlocked, + wantReason: "blocked — CI pending", + }, + { + // Every sweep gate passes and GitHub still says blocked: a + // branch-protection rule the sweep does not read yet. Say so. + name: "no sweep gate explains it", + pr: blockedPR(7, "success"), + wantBucket: mergeBucketSkip, + wantState: github.MergeVerdictBlocked, + wantReason: "blocked — all sweep gates pass; a branch-protection rule is unsatisfied", + }, + { + name: "conflicts", + pr: github.PullRequest{Number: 8, Mergeable: no, MergeableState: "dirty", BaseRef: "v4", CIStatus: "success"}, + wantBucket: mergeBucketSkip, + wantState: github.MergeVerdictBlocked, + wantReason: "has merge conflicts with v4 — needs a rebase", + }, + { + name: "conflicts with a sweep reason on top", + pr: github.PullRequest{Number: 9, Mergeable: no, MergeableState: "dirty", BaseRef: "v4", CIStatus: "failure", FailingChecks: []string{"build"}}, + wantBucket: mergeBucketFailing, + wantState: github.MergeVerdictBlocked, + wantReason: "has merge conflicts with v4 — needs a rebase; also CI failing: build", + }, + { + name: "behind", + pr: github.PullRequest{Number: 10, Mergeable: no, MergeableState: "behind", BaseRef: "v4", CIStatus: "success"}, + wantBucket: mergeBucketSkip, + wantState: github.MergeVerdictBlocked, + wantReason: "behind v4 — needs an update from the base branch", + }, + { + // An abbreviated payload with no base ref still reads sensibly. + name: "behind with the base branch unknown", + pr: github.PullRequest{Number: 11, Mergeable: no, MergeableState: "behind", CIStatus: "success"}, + wantBucket: mergeBucketSkip, + wantState: github.MergeVerdictBlocked, + wantReason: "behind the base branch — needs an update from it", + }, + { + name: "draft", + pr: github.PullRequest{Number: 12, Draft: true, Mergeable: github.MergeableYes, CIStatus: "success"}, + wantBucket: mergeBucketSkip, + wantState: github.MergeVerdictBlocked, + wantReason: "draft — mark ready for review to enter the sweep", + }, + { + name: "unknown", + pr: github.PullRequest{Number: 13, CIStatus: "pending"}, + wantBucket: mergeBucketSkip, + wantState: github.MergeVerdictUnknown, + wantReason: "mergeability not yet computed by GitHub — re-checked next tick; CI pending", + }, + { + // A GitHub state this function has no wording for still names + // the state and keeps the sweep reason. + name: "an unrecognised GitHub state falls back to naming it", + pr: github.PullRequest{Number: 14, Mergeable: no, MergeableState: "draft", CIStatus: "success"}, + held: true, + wantBucket: mergeBucketSkip, + wantState: github.MergeVerdictBlocked, + wantReason: "not mergeable on GitHub (draft); also held: a hold label keeps it out of the sweep", + }, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + bucket, verdict := mergeVerdictOf(t, tc.pr, tc.held, tc.gates) + if bucket != tc.wantBucket { + t.Errorf("bucket = %v, want %v", bucket, tc.wantBucket) + } + if verdict.State != tc.wantState { + t.Errorf("state = %q, want %q", verdict.State, tc.wantState) + } + if verdict.Reason != tc.wantReason { + t.Errorf("reason = %q\n want %q", verdict.Reason, tc.wantReason) + } + // The bare enum is never the whole story any more. + for _, enum := range []string{"(blocked)", "(dirty)", "(behind)"} { + if strings.Contains(verdict.Reason, enum) { + t.Errorf("reason %q still shows GitHub's raw state %s", verdict.Reason, enum) + } + } + }) + } +} + +// Moving the review gate ahead of the GitHub-says-no return must not change +// which bucket anything lands in: a MergeableNo PR skips either way, and a +// mergeable one never reached that return. The verdict is the only thing +// the order affects. +func TestClassifyMergeEligibility_ReviewGateOrderKeepsBuckets(t *testing.T) { + gates := mergeGates{requireReviewApproval: true, reviewLoaded: true} + for _, pr := range []github.PullRequest{ + {Number: 1, Mergeable: github.MergeableNo, MergeableState: "blocked", CIStatus: "success"}, + {Number: 2, Mergeable: github.MergeableNo, MergeableState: "dirty", CIStatus: "success"}, + {Number: 3, Mergeable: github.MergeableYes, MergeableState: "clean", CIStatus: "success"}, + } { + bucket, verdict := mergeVerdictOf(t, pr, false, gates) + if bucket != mergeBucketSkip { + t.Errorf("#%d: bucket = %v, want skip (reason %q)", pr.Number, bucket, verdict.Reason) + } + if !strings.Contains(verdict.Reason, "awaiting review approval") { + t.Errorf("#%d: reason %q does not name the missing review", pr.Number, verdict.Reason) + } + } +} diff --git a/src/pkg/dashboard/pr_pill_merge_verdict_7478_test.go b/src/pkg/dashboard/pr_pill_merge_verdict_7478_test.go index ba6f0c295d..567b50bbd4 100644 --- a/src/pkg/dashboard/pr_pill_merge_verdict_7478_test.go +++ b/src/pkg/dashboard/pr_pill_merge_verdict_7478_test.go @@ -156,7 +156,7 @@ function check(name, cond) { // Green only on the sweep's verdict. check('eligible is green', prMergeState({ mergeable: 'yes', merge_verdict: { state: 'eligible' } }) === 'eligible'); check('outstanding is amber', prMergeState({ mergeable: 'yes', merge_verdict: { state: 'outstanding', reason: 'CI failing: build' } }) === 'outstanding'); -check('blocked lights nothing', prMergeState({ mergeable: 'no', merge_verdict: { state: 'blocked', reason: 'not mergeable on GitHub (dirty)' } }) === ''); +check('blocked lights nothing', prMergeState({ mergeable: 'no', merge_verdict: { state: 'blocked', reason: 'has merge conflicts with v4 — needs a rebase' } }) === ''); check('unknown lights nothing', prMergeState({ mergeable: '', merge_verdict: { state: 'unknown' } }) === ''); check('an unrecognised state lights nothing', prMergeState({ mergeable: 'yes', merge_verdict: { state: 'someday' } }) === ''); @@ -186,11 +186,20 @@ check('eligible tooltip carries the reason', prMergeNote(green).includes('would const greenOptionalRed = { mergeable: 'yes', mergeable_state: 'unstable', merge_verdict: { state: 'eligible', reason: 'the sweep would merge this now — only non-required checks are red (playwright)' } }; check('eligible beside a red optional check names it', prMergeNote(greenOptionalRed).includes('playwright')); check('eligible without a reason still reads sensibly', prMergeNote({ mergeable: 'yes', merge_verdict: { state: 'eligible' } }).includes('would merge this now')); -const dirty = { mergeable: 'no', mergeable_state: 'dirty', merge_verdict: { state: 'blocked', reason: 'not mergeable on GitHub (dirty)' } }; -check('blocked tooltip carries the reason', prMergeNote(dirty).includes('dirty')); +const dirty = { mergeable: 'no', mergeable_state: 'dirty', merge_verdict: { state: 'blocked', reason: 'has merge conflicts with v4 — needs a rebase' } }; +check('blocked tooltip carries the reason', prMergeNote(dirty).includes('merge conflicts with v4')); check('blocked tooltip does not say eligible', !prMergeNote(dirty).includes('eligible')); -check('unknown tooltip says not yet known', prMergeNote({ mergeable: '', merge_verdict: { state: 'unknown', reason: 'mergeability not yet known; CI pending' } }).includes('not yet known')); -check('no undefined anywhere', ![green, bluefin1253, dirty, { mergeable: 'yes' }, {}, { merge_verdict: {} }].some(p => prMergeNote(p).includes('undefined'))); +// hivecommons/hive#7515: a "blocked" verdict now carries the sweep's own +// reason (the red required check, the missing review) and the tooltip shows +// it verbatim — GitHub's raw state is not appended on top. +const blocked = { mergeable: 'no', mergeable_state: 'blocked', merge_verdict: { state: 'blocked', reason: 'blocked — CI failing: build, lint' } }; +check('blocked tooltip names the failing required checks', prMergeNote(blocked).includes('blocked — CI failing: build, lint')); +check('blocked tooltip does not repeat the raw GitHub state', !prMergeNote(blocked).includes('GitHub state')); +const blockedReview = { mergeable: 'no', mergeable_state: 'blocked', merge_verdict: { state: 'blocked', reason: 'blocked — awaiting review approval' } }; +check('blocked tooltip names the missing review', prMergeNote(blockedReview).includes('awaiting review approval')); +check('draft tooltip says what to do', prMergeNote({ mergeable: 'yes', merge_verdict: { state: 'blocked', reason: 'draft — mark ready for review to enter the sweep' } }).includes('mark ready for review')); +check('unknown tooltip says not yet computed', prMergeNote({ mergeable: '', merge_verdict: { state: 'unknown', reason: 'mergeability not yet computed by GitHub — re-checked next tick; CI pending' } }).includes('not yet computed')); +check('no undefined anywhere', ![green, bluefin1253, dirty, blocked, { mergeable: 'yes' }, {}, { merge_verdict: {} }].some(p => prMergeNote(p).includes('undefined'))); if (fails) { console.log(fails + ' check(s) failed'); process.exit(1); } ` diff --git a/src/pkg/github/client.go b/src/pkg/github/client.go index ffa22c9a50..6305e7327b 100644 --- a/src/pkg/github/client.go +++ b/src/pkg/github/client.go @@ -355,6 +355,13 @@ type PullRequest struct { HeadRef string `json:"head_ref,omitempty"` HeadRepo string `json:"head_repo,omitempty"` FromFork bool `json:"from_fork,omitempty"` + // BaseRef is the branch the PR targets. Display only: the merge-verdict + // reason names it ("has merge conflicts with v4 — needs a rebase") so a + // blocked pill says what to do rather than GitHub's enum + // (hivecommons/hive#7515). It comes from the list payload — no extra + // fetch — and stays empty on an abbreviated payload, where the wording + // falls back to "the base branch". + BaseRef string `json:"base_ref,omitempty"` // FailingChecks names the completed check runs whose conclusion was // failure/action_required. CIFailureExcerpt carries the raw error lines // pulled from those runs' annotations — the evidence a fix agent (or an @@ -805,6 +812,14 @@ func prHeadSHA(pr *gh.PullRequest) string { return pr.GetHead().GetSHA() } +// prBaseRef is the branch a PR targets, or "" on an abbreviated payload. +func prBaseRef(pr *gh.PullRequest) string { + if pr.GetBase() == nil { + return "" + } + return pr.GetBase().GetRef() +} + func (c *Client) fetchPRs(ctx context.Context, repo string) (actionable []PullRequest, held []HoldItem, heldPRs []PullRequest, staleDrafts []PullRequest, totalPRs int, err error) { now := time.Now() owner, repoName := c.splitRepo(repo) @@ -857,6 +872,7 @@ func (c *Client) fetchPRs(ctx context.Context, repo string) (actionable []PullRe HeadRef: headRef, HeadRepo: headRepo, FromFork: fromFork, + BaseRef: prBaseRef(pr), }) } continue @@ -907,6 +923,7 @@ func (c *Client) fetchPRs(ctx context.Context, repo string) (actionable []PullRe HeadRef: headRef, HeadRepo: headRepo, FromFork: fromFork, + BaseRef: prBaseRef(pr), }) } diff --git a/src/pkg/github/pr_head_origin_test.go b/src/pkg/github/pr_head_origin_test.go index 5573d5673b..d2c5d3b4cc 100644 --- a/src/pkg/github/pr_head_origin_test.go +++ b/src/pkg/github/pr_head_origin_test.go @@ -89,4 +89,20 @@ func TestFetchPRs_PopulatesForkOrigin(t *testing.T) { if pr := byNum[841]; !pr.FromFork || pr.HeadRepo != "" { t.Errorf("deleted-fork PR must be unpushable: %+v", pr) } + // The base branch rides along from the same payload so the merge + // verdict can say "has merge conflicts with main" (hivecommons/hive#7515). + for _, n := range []int{839, 840, 841} { + if pr := byNum[n]; pr.BaseRef != "main" { + t.Errorf("#%d: BaseRef = %q, want %q", n, pr.BaseRef, "main") + } + } +} + +func TestPRBaseRef(t *testing.T) { + if got := prBaseRef(&gh.PullRequest{Base: &gh.PullRequestBranch{Ref: gh.Ptr("v4")}}); got != "v4" { + t.Errorf("prBaseRef = %q, want v4", got) + } + if got := prBaseRef(&gh.PullRequest{}); got != "" { + t.Errorf("prBaseRef with no base = %q, want empty", got) + } } From 9fb8f47d1326bc3d320b09fa63a7e6b0a43c128d Mon Sep 17 00:00:00 2001 From: hive-release-bot Date: Fri, 18 Sep 2026 02:56:55 +0000 Subject: [PATCH 02/17] =?UTF-8?q?=F0=9F=94=96=20release:=20v4.54.2?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Automated release commit. Compiles changelog.d/ fragments and moves the CHANGELOG.md Unreleased section into a dated v4.54.2 entry. See src/docs/releases.md. Signed-off-by: hive-release-bot --- CHANGELOG.md | 6 ++++++ changelog.d/fixed-7515-blocked-pill-sweep-reason.md | 1 - 2 files changed, 6 insertions(+), 1 deletion(-) delete mode 100644 changelog.d/fixed-7515-blocked-pill-sweep-reason.md diff --git a/CHANGELOG.md b/CHANGELOG.md index 7f77900a2d..f82b35005b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -11,6 +11,12 @@ Hive did not historically maintain a complete changelog. This file starts a prag ## Unreleased +## 2026-09-18 (v4.54.2) + +### Fixed + +- Dashboard: hovering a PR pill that is not green now says, in plain words, why the sweep will not merge it and what would unblock it ([#7515](https://github.com/hivecommons/hive/issues/7515)). A PR GitHub reported as not mergeable used to hover as the bare GitHub enum — `not mergeable on GitHub (blocked)` — even though the merge-eligible classifier had just been handed the exact gate that made GitHub say blocked and then dropped it. `blocked` now keeps the sweep's reason (`blocked — CI failing: build, lint`, `blocked — awaiting review approval`, `blocked — held: …`, `blocked — intent verification: …`, `blocked — CI pending`), and when every sweep gate passes it says `blocked — all sweep gates pass; a branch-protection rule is unsatisfied` instead of nothing. The other states read as what to do rather than the API name: `has merge conflicts with v4 — needs a rebase`, `behind v4 — needs an update from the base branch`, `draft — mark ready for review to enter the sweep`, `mergeability not yet computed by GitHub — re-checked next tick`. The base branch comes from the PR list payload (`base_ref`, new on the status snapshot); no extra GitHub calls. Naming the exact branch-protection rule when the sweep itself has no reason (a review GitHub requires, a required check that never reported) is the issue's step 2 and still open. + ## 2026-09-18 (v4.54.1) ### Changed diff --git a/changelog.d/fixed-7515-blocked-pill-sweep-reason.md b/changelog.d/fixed-7515-blocked-pill-sweep-reason.md deleted file mode 100644 index 188fc4f408..0000000000 --- a/changelog.d/fixed-7515-blocked-pill-sweep-reason.md +++ /dev/null @@ -1 +0,0 @@ -- Dashboard: hovering a PR pill that is not green now says, in plain words, why the sweep will not merge it and what would unblock it ([#7515](https://github.com/hivecommons/hive/issues/7515)). A PR GitHub reported as not mergeable used to hover as the bare GitHub enum — `not mergeable on GitHub (blocked)` — even though the merge-eligible classifier had just been handed the exact gate that made GitHub say blocked and then dropped it. `blocked` now keeps the sweep's reason (`blocked — CI failing: build, lint`, `blocked — awaiting review approval`, `blocked — held: …`, `blocked — intent verification: …`, `blocked — CI pending`), and when every sweep gate passes it says `blocked — all sweep gates pass; a branch-protection rule is unsatisfied` instead of nothing. The other states read as what to do rather than the API name: `has merge conflicts with v4 — needs a rebase`, `behind v4 — needs an update from the base branch`, `draft — mark ready for review to enter the sweep`, `mergeability not yet computed by GitHub — re-checked next tick`. The base branch comes from the PR list payload (`base_ref`, new on the status snapshot); no extra GitHub calls. Naming the exact branch-protection rule when the sweep itself has no reason (a review GitHub requires, a required check that never reported) is the issue's step 2 and still open. From bf298409c67f2c44527f5cc41e228206055a5730 Mon Sep 17 00:00:00 2001 From: "kubestellar-hive[bot]" <280983584+kubestellar-hive[bot]@users.noreply.github.com> Date: Thu, 17 Sep 2026 23:09:59 -0400 Subject: [PATCH 03/17] [strategist] planning: define the v6 designation before it defines itself (#7520) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ROADMAP.md gains a 'v6 — Future Line (not open)' section and src/docs/roadmap.md gains a matching Later row: no v6 branch exists, v6-designated work is design-only, the line opens after the v5 GA bar (#6016) via the same RFC gate that governs v5, and the one existing v6-designated design (github-mention-triggers.md, #7483) is indexed. Closes #7519 Signed-off-by: sec-check Co-authored-by: sec-check Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- ROADMAP.md | 31 +++++++++++++++++++++++++++++++ src/docs/roadmap.md | 1 + 2 files changed, 32 insertions(+) diff --git a/ROADMAP.md b/ROADMAP.md index a10e21e347..c195400985 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -122,6 +122,37 @@ operation during the transition, is part of the [v5 GA readiness bar](https://github.com/hivecommons/hive/blob/v5/src/docs/v5-ga.md) (live tracker: [#6016](https://github.com/hivecommons/hive/issues/6016)). +## v6 — Future Line (not open) + +There is no v6 branch, milestone, or release channel, and none is planned +until v5 reaches GA. "v6" is a **designation, not a line**: a label for +proposals that deliberately target the horizon after the current one, so +they can be written down without pretending they are scheduled. + +Policy for v6-designated work: + +- **Design-only.** A v6-designated proposal may land as a design document + under [`src/docs/design/`](src/docs/design/) after normal review. + Implementation PRs for v6-designated work are out of scope on every + current branch and should be closed with a pointer to this section. +- **The line opens after v5 GA.** Opening a v6 line is blocked on the + [v5 GA readiness bar](https://github.com/hivecommons/hive/blob/v5/src/docs/v5-ga.md) + closing (live tracker: + [#6016](https://github.com/hivecommons/hive/issues/6016)). When it opens, + it opens the same way v5 did — public RFC issues gate each workstream, + per [GOVERNANCE.md](GOVERNANCE.md) — and designs parked here re-enter + through that gate rather than being grandfathered in. +- **Designation is cheap and non-binding.** Marking a design "v6" records + intent and preserves the work; it is not acceptance, priority, or a + commitment that a v6 line will include it. + +v6-designated designs to date: + +- **GitHub @-mention triggers** — a human summons an agent by mentioning + the App on an issue or PR, mirroring the existing Linear inbound-mention + path ([#7483](https://github.com/hivecommons/hive/issues/7483), + [design doc](src/docs/design/github-mention-triggers.md)). + ## Hosted Hive Hub The hosted hub at [hive.hivecommons.dev](https://hive.hivecommons.dev) is the diff --git a/src/docs/roadmap.md b/src/docs/roadmap.md index 421fdb68d0..66975d518c 100644 --- a/src/docs/roadmap.md +++ b/src/docs/roadmap.md @@ -52,6 +52,7 @@ order, not priority rank. | Cross-forge orchestration | Coordinate issues, merge requests, policy, and evidence across GitHub, GitLab, and Forgejo/Gitea-style forges. | [ADR-0005](adr/0005-forge-abstraction.md) | | Memory and learning maturation | Turn retro findings and curated knowledge into durable, testable priming without hidden or unauditable agent memory. | [knowledge design](design/knowledge-system.md), [retro lane](retro-lane.md) | | Kubernetes-native agent sandboxes | Graduate from tmux/container execution toward k8s-native, policy-isolated agent workloads where that complexity is justified. | [architecture](architecture.md), [security threat model](security-threat-model.md) | +| v6-designated designs (line not open) | "v6" is a designation, not a line: no v6 branch, milestone, or channel exists, and opening one is blocked on the v5 GA bar closing. v6-designated proposals land as design docs only (no implementation PRs on current branches) and re-enter through the RFC gate when the line opens. First entry: GitHub @-mention triggers — the first inbound GitHub trigger, mirroring the Linear mention path. Full policy in [ROADMAP.md](../../ROADMAP.md#v6--future-line-not-open). | [#7519](https://github.com/hivecommons/hive/issues/7519), [#7483](https://github.com/hivecommons/hive/issues/7483), [mention-triggers design](design/github-mention-triggers.md) | ## Reading this roadmap From ffb58e66cffa00afc8762a510060d8cb1383209c Mon Sep 17 00:00:00 2001 From: Andy Anderson Date: Thu, 17 Sep 2026 23:11:18 -0400 Subject: [PATCH 04/17] =?UTF-8?q?=E2=9C=A8=20feat(review):=20route=20block?= =?UTF-8?q?ing=20verdicts=20to=20a=20human=20(#7527)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A hive at ACMM L5 cannot merge, so the only possible outcome of a review is a human acting on it. Reviewers already produce well-cited comments with requires_human and reject verdicts, but nothing carried those to a person: a blocking finding landed in a queue of hundreds looking exactly like routine review noise, and the two reviews posted on the projectbluefin spoke contained zero mentions and no way to filter for "a human must decide this". Add a routing block to the publish half of the kick. A requires_human or reject verdict must open the comment with a pinned `**HUMAN DECISION NEEDED**` marker, which is the string a maintainer triaging a long queue can actually search on; its whole value is being identical everywhere, so it is pinned by test. Mention the PR author only when the author is a person. On a hive fleet most PRs are agent-authored, so the author is an App ("app/") or a bot ("[bot]"), and @-mentioning either notifies no one while still reading as though the review had been routed somewhere. That false signal is worse than no mention, so for those authors the prompt explicitly suppresses the mention and leans on the marker instead. Also ask the reviewer to report the limits of its own review. An honest "I could not verify X" is more useful than a confident guess, and leaving it out is how an unreviewed change gets waved through on the strength of an automated approval. Routing rides the existing review.post_comments switch, so a hive that has not opted into commenting sees no prompt change. Signed-off-by: Andrew Anderson Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../added-reviewer-routes-to-humans.md | 1 + src/pkg/review/prompts.go | 39 ++++++++++ src/pkg/review/publish_prompt_test.go | 78 +++++++++++++++++++ 3 files changed, 118 insertions(+) create mode 100644 changelog.d/added-reviewer-routes-to-humans.md diff --git a/changelog.d/added-reviewer-routes-to-humans.md b/changelog.d/added-reviewer-routes-to-humans.md new file mode 100644 index 0000000000..f1b1a9981a --- /dev/null +++ b/changelog.d/added-reviewer-routes-to-humans.md @@ -0,0 +1 @@ +- Review comments now route: a `requires_human` or `reject` verdict opens with a pinned `**HUMAN DECISION NEEDED**` marker maintainers can filter on, mentions the PR author when that author is a person (never an app/bot account, which notifies nobody), and asks the reviewer to state plainly what it could not judge instead of approving around it. diff --git a/src/pkg/review/prompts.go b/src/pkg/review/prompts.go index c49691ab20..f37560ec34 100644 --- a/src/pkg/review/prompts.go +++ b/src/pkg/review/prompts.go @@ -86,9 +86,48 @@ func buildPublishInstruction(pr PullRequest) string { b.WriteString("Say nothing rather than pad. Do NOT post a comment that is only nits, only praise, or a restatement of the diff. If this perspective found nothing a human needs, skip the comment entirely and just return the JSON.\n") b.WriteString("If the PR body claims behavior the diff does not implement, say so with file:line — that gap is one of the most useful things you can report.\n") b.WriteString("Be brief and specific. One comment, at most a few findings, worst first.\n") + b.WriteString(buildRoutingInstruction(pr)) return b.String() } +// buildRoutingInstruction is what turns a verdict into a decision. +// +// A hive that cannot merge produces reviews whose only possible outcome is a +// human acting on them. But a correct, well-cited comment buried in a queue of +// hundreds is not actionable: nothing distinguishes "a human must decide this" +// from routine review noise. Routing is therefore not a nicety on top of the +// review — it is the step that makes the review reach anyone. +func buildRoutingInstruction(pr PullRequest) string { + var b strings.Builder + b.WriteString("\nROUTING — make a needed human decision findable.\n") + b.WriteString("If your verdict is requires_human or reject, the FIRST line of the comment must be exactly:\n") + b.WriteString(" **HUMAN DECISION NEEDED** — \n") + b.WriteString("A maintainer triaging a long queue filters on that marker; without it a blocking finding reads as one more comment and is skipped.\n") + if handle := mentionableAuthor(pr.Author); handle != "" { + fmt.Fprintf(&b, "On that same line, mention @%s (the PR author) so the person who can act is notified.\n", handle) + } else { + b.WriteString("Do NOT @-mention the PR author: this PR was opened by an app or bot account, so a mention notifies nobody. The marker line is the routing.\n") + } + b.WriteString("Report the limits of your own review. If you could not judge part of this PR — missing context, an unfamiliar subsystem, an ambiguous requirement, a change you cannot test — say so plainly and use requires_human. Naming what you could not verify is more useful than a confident guess, and omitting it is how an unreviewed change gets waved through on the strength of an automated approval.\n") + return b.String() +} + +// mentionableAuthor returns the bare @-handle for a PR author when mentioning +// it would reach a person, and "" when it would not. +// +// On a hive fleet most PRs are agent-authored, so the author is an App +// ("app/") or a bot ("[bot]"). @-mentioning either notifies no one +// — it renders as a link and nothing else — while still looking to a reader +// like the review was routed somewhere. That false signal is worse than no +// mention at all, because it suggests a human is already on it. +func mentionableAuthor(author string) string { + a := strings.TrimSpace(strings.TrimPrefix(strings.TrimSpace(author), "@")) + if a == "" || strings.HasSuffix(a, "[bot]") || strings.Contains(a, "/") { + return "" + } + return a +} + func BuildPerspectivePrompts(pr PullRequest, perspectives []Perspective) map[Perspective]string { if len(perspectives) == 0 { perspectives = DefaultPerspectives diff --git a/src/pkg/review/publish_prompt_test.go b/src/pkg/review/publish_prompt_test.go index 2fac0685cb..3cada0aa68 100644 --- a/src/pkg/review/publish_prompt_test.go +++ b/src/pkg/review/publish_prompt_test.go @@ -111,3 +111,81 @@ func TestPromptPublishAppliesToEveryPerspective(t *testing.T) { } } } + +// TestRoutingMarkerIsPinned pins the exact marker a maintainer filters on. The +// value of the marker is entirely in its being identical across every repo and +// every reviewer; a reworded variant is unsearchable and therefore useless. +func TestRoutingMarkerIsPinned(t *testing.T) { + got := BuildPerspectivePromptOpts(PerspectiveCorrectness, testPR(), true) + + if !strings.Contains(got, "**HUMAN DECISION NEEDED**") { + t.Error("publish prompt does not pin the HUMAN DECISION NEEDED marker") + } + if !strings.Contains(got, "requires_human or reject") { + t.Error("publish prompt does not tie the marker to the blocking verdicts") + } +} + +// TestRoutingDoesNotMentionBotAuthors is the point of the author check: the +// common case on a hive is an agent-authored PR, and mentioning the App that +// opened it notifies nobody while looking like the review was routed. +func TestRoutingDoesNotMentionBotAuthors(t *testing.T) { + for _, author := range []string{ + "kubestellar-hive[bot]", + "app/kubestellar-hive", + "", + } { + pr := testPR() + pr.Author = author + got := BuildPerspectivePromptOpts(PerspectiveCorrectness, pr, true) + + if strings.Contains(got, "(the PR author) so the person who can act") { + t.Errorf("author %q: prompt asks the reviewer to mention a non-human author", author) + } + if !strings.Contains(got, "Do NOT @-mention the PR author") { + t.Errorf("author %q: prompt omits the do-not-mention instruction", author) + } + } +} + +// TestRoutingMentionsHumanAuthors is the converse: when a person opened the PR, +// the mention is the fastest path from finding to decision. +func TestRoutingMentionsHumanAuthors(t *testing.T) { + pr := testPR() + pr.Author = "@clubanderson" + got := BuildPerspectivePromptOpts(PerspectiveCorrectness, pr, true) + + if !strings.Contains(got, "mention @clubanderson (the PR author)") { + t.Error("prompt does not ask the reviewer to mention the human PR author") + } + // The leading @ must not be doubled when the author already carries one. + if strings.Contains(got, "@@") { + t.Error("prompt double-prefixed the author handle with @") + } + if strings.Contains(got, "Do NOT @-mention the PR author") { + t.Error("prompt suppressed the mention for a human author") + } +} + +// TestRoutingAsksForReviewLimits covers the capability-honesty half: a reviewer +// that cannot judge something must say so rather than approve around it. +func TestRoutingAsksForReviewLimits(t *testing.T) { + got := BuildPerspectivePromptOpts(PerspectiveCorrectness, testPR(), true) + + if !strings.Contains(got, "Report the limits of your own review") { + t.Error("publish prompt does not ask the reviewer to report what it could not judge") + } +} + +// TestRoutingIsOptInWithPublish keeps routing on the same switch as publishing: +// a hive that has not opted into comments must not have its prompt changed. +func TestRoutingIsOptInWithPublish(t *testing.T) { + got := BuildPerspectivePrompt(PerspectiveCorrectness, testPR()) + + if strings.Contains(got, "HUMAN DECISION NEEDED") { + t.Error("default prompt contains the routing marker; routing must be opt-in with publishing") + } + if strings.Contains(got, "ROUTING") { + t.Error("default prompt contains the routing block; routing must be opt-in with publishing") + } +} From 21c64828710d9f949930d7e65c1c8ed2ec3c5759 Mon Sep 17 00:00:00 2001 From: Andy Anderson Date: Thu, 17 Sep 2026 23:18:31 -0400 Subject: [PATCH 05/17] =?UTF-8?q?=E2=9C=A8=20feat(review):=20spend=20revie?= =?UTF-8?q?w=20slots=20breadth-first=20on=20deep=20queues=20(#7528)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit PlanDispatch spends a fixed parallel-review budget in PR order, and an uncapped PR takes as many slots as it has missing perspectives. On a short queue that is exactly right - fanning every perspective out at once is the review swarm's designed behavior. On a deep one it inverts the intent: the head of the queue absorbs the entire budget, the PRs behind it get nothing this cycle, and adding reviewers buys more opinions on one PR rather than coverage across many. A spoke with hundreds of open PRs is the case where that matters, because there the scarce thing is PRs looked at, not depth per PR. Add review.max_perspectives_per_pr, which caps perspectives dispatched to one PR per cycle. It loses no coverage: a perspective skipped this cycle is still missing next cycle and gets dispatched then, so the cap schedules depth rather than dropping it. It also bounds how many comments a single PR can collect at once, which starts to matter now that reviewers publish their verdicts. Zero means no cap, so every existing hive keeps the behavior it has. The cap is subordinate to MaxParallelReviews, which still bounds total concurrency. Signed-off-by: Andrew Anderson Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- changelog.d/added-review-breadth-first.md | 1 + src/cmd/hive/main.go | 19 +++--- src/pkg/config/config.go | 13 ++++ src/pkg/review/dispatch.go | 30 +++++++-- src/pkg/review/dispatch_test.go | 81 +++++++++++++++++++++++ 5 files changed, 131 insertions(+), 13 deletions(-) create mode 100644 changelog.d/added-review-breadth-first.md diff --git a/changelog.d/added-review-breadth-first.md b/changelog.d/added-review-breadth-first.md new file mode 100644 index 0000000000..c50bfca39b --- /dev/null +++ b/changelog.d/added-review-breadth-first.md @@ -0,0 +1 @@ +- Added `review.max_perspectives_per_pr`, an opt-in cap on how many review perspectives one PR may be given per dispatch cycle. Without it the parallel review budget is spent in PR order, so the head of a deep queue absorbs every slot and adding reviewers buys more opinions on one PR instead of coverage across many. Capping spends the same budget breadth-first, and loses no coverage because skipped perspectives are still dispatched on later cycles. Zero (the default) keeps the existing fan-out behavior. diff --git a/src/cmd/hive/main.go b/src/cmd/hive/main.go index ef95c90503..bf3dfb75f8 100644 --- a/src/cmd/hive/main.go +++ b/src/cmd/hive/main.go @@ -9487,15 +9487,16 @@ func planReviewDispatch(cfg *config.Config, actionable *github.ActionableResult, }) } plan := review.PlanDispatch(prs, artifact, state, review.DispatchOptions{ - RequireApproval: cfg.Review.RequireApproval, - FanOut: cfg.Review.FanOut, - MaxParallelReviews: cfg.Review.EffectiveMaxParallelReviews(), - ReviewerAgents: cfg.Review.ReviewerAgents, - FixerAgent: cfg.Review.FixerAgent, - PostComments: cfg.Review.PostComments, - ProjectOrg: cfg.Project.Org, - AIAuthor: cfg.EffectiveAIAuthor(), - Agents: agents, + RequireApproval: cfg.Review.RequireApproval, + FanOut: cfg.Review.FanOut, + MaxParallelReviews: cfg.Review.EffectiveMaxParallelReviews(), + MaxPerspectivesPerPR: cfg.Review.MaxPerspectivesPerPR, + ReviewerAgents: cfg.Review.ReviewerAgents, + FixerAgent: cfg.Review.FixerAgent, + PostComments: cfg.Review.PostComments, + ProjectOrg: cfg.Project.Org, + AIAuthor: cfg.EffectiveAIAuthor(), + Agents: agents, }) if len(plan.ReviewKicks)+len(plan.FixKicks) > 0 { logger.Info("review swarm dispatch planned", "review_kicks", len(plan.ReviewKicks), "fix_kicks", len(plan.FixKicks)) diff --git a/src/pkg/config/config.go b/src/pkg/config/config.go index d5c6d72109..871d93d70e 100644 --- a/src/pkg/config/config.go +++ b/src/pkg/config/config.go @@ -6294,6 +6294,19 @@ type ReviewConfig struct { // aggregate has no consumer and the reviewer is silent by construction. // Turning this on is what makes a review reach the human who has to decide. PostComments bool `yaml:"post_comments,omitempty" json:"post_comments,omitempty"` + // MaxPerspectivesPerPR caps how many review perspectives one PR may be + // given in a single dispatch cycle. It exists because parallel review + // slots are a fixed budget spent in PR order: without a cap, the first PR + // in a deep queue absorbs every slot for its own perspectives, so adding + // reviewers buys more opinions on one PR instead of coverage across many. + // Capping it spends the same budget breadth-first. No coverage is lost — + // the perspectives skipped this cycle are still "missing" next cycle and + // get dispatched then — so this schedules depth rather than dropping it. + // It also bounds how many comments a single PR can collect at once, which + // matters once reviewers publish. Zero means no cap — fanning every + // perspective out at once stays the default, so this only changes a hive + // that opts in because its queue is too deep to review in depth. + MaxPerspectivesPerPR int `yaml:"max_perspectives_per_pr,omitempty" json:"max_perspectives_per_pr,omitempty"` } // DuplicateSweepConfig gates the cross-PR duplicate sweep diff --git a/src/pkg/review/dispatch.go b/src/pkg/review/dispatch.go index d6dd0f5f33..98ea572455 100644 --- a/src/pkg/review/dispatch.go +++ b/src/pkg/review/dispatch.go @@ -38,10 +38,13 @@ type DispatchOptions struct { RequireApproval bool FanOut bool MaxParallelReviews int - ReviewerAgents []string - FixerAgent string - ProjectOrg string - AIAuthor string + // MaxPerspectivesPerPR caps perspectives dispatched to one PR per cycle. + // Zero means DefaultMaxPerspectivesPerPR. + MaxPerspectivesPerPR int + ReviewerAgents []string + FixerAgent string + ProjectOrg string + AIAuthor string // PostComments carries config.ReviewConfig.PostComments into the prompt // builder, so reviewers are told to publish their verdict on the PR. PostComments bool @@ -169,6 +172,12 @@ func PlanDispatch(prs []PullRequest, artifact Artifact, state DispatchState, opt continue } limit := len(missing) + // Breadth before depth: the parallel budget is spent in PR order, so + // an uncapped first PR would take every slot for its own perspectives + // and leave the rest of the queue unreviewed this cycle. + if perPR := opts.effectiveMaxPerspectivesPerPR(); perPR > 0 && limit > perPR { + limit = perPR + } if len(reviewers) == 1 && limit > 1 { limit = 1 } @@ -293,6 +302,19 @@ func (a Artifact) AggregateFor(repo string, number int, headSHA string) (Aggrega return Aggregate{}, false } +// effectiveMaxPerspectivesPerPR returns the per-PR perspective cap, or 0 for +// "no cap". Unlimited is the default deliberately: fanning every perspective +// out at once is the review swarm's designed behavior, and a hive that wants +// depth on each PR should keep getting it. The cap is for the opposite +// situation — a queue too deep to review in depth — and is opt-in so no +// existing hive silently changes shape. +func (o DispatchOptions) effectiveMaxPerspectivesPerPR() int { + if o.MaxPerspectivesPerPR <= 0 { + return 0 + } + return o.MaxPerspectivesPerPR +} + func reviewCapableAgents(opts DispatchOptions) []AgentCapability { allowed := map[string]bool{} for _, name := range opts.ReviewerAgents { diff --git a/src/pkg/review/dispatch_test.go b/src/pkg/review/dispatch_test.go index 2448dbaede..b42d079c09 100644 --- a/src/pkg/review/dispatch_test.go +++ b/src/pkg/review/dispatch_test.go @@ -133,3 +133,84 @@ func TestConfigReviewDefaults(t *testing.T) { t.Fatalf("configured max parallel = %d, want 2", got) } } + +func dispatchPRNum(n int, sha string) PullRequest { + pr := dispatchPR(sha) + pr.Number = n + return pr +} + +// TestDispatchSpendsSlotsDepthFirstByDefault documents the status quo the cap +// exists to change: the parallel budget is spent in PR order, so the head of +// the queue absorbs every slot for its own perspectives and the PRs behind it +// get nothing this cycle. That is the right behavior when the queue is short. +func TestDispatchSpendsSlotsDepthFirstByDefault(t *testing.T) { + prs := []PullRequest{dispatchPRNum(1, "sha1"), dispatchPRNum(2, "sha2"), dispatchPRNum(3, "sha3")} + plan := PlanDispatch(prs, Artifact{}, DispatchState{}, DispatchOptions{ + RequireApproval: true, + FanOut: true, + MaxParallelReviews: 3, + ProjectOrg: "acme", + Agents: []AgentCapability{reviewer("r1"), reviewer("r2"), reviewer("r3")}, + }) + + if len(plan.ReviewKicks) != 3 { + t.Fatalf("got %d kicks, want 3 (the full slot budget)", len(plan.ReviewKicks)) + } + for _, k := range plan.ReviewKicks { + if k.Number != 1 { + t.Fatalf("uncapped dispatch should concentrate on the first PR, got a kick for #%d", k.Number) + } + } +} + +// TestMaxPerspectivesPerPRSpreadsAcrossPRs is the point of the cap: the same +// budget, spent breadth-first, reviews every PR in the queue once instead of +// one PR three times. No coverage is lost — the perspectives skipped here are +// still missing next cycle and get dispatched then. +func TestMaxPerspectivesPerPRSpreadsAcrossPRs(t *testing.T) { + prs := []PullRequest{dispatchPRNum(1, "sha1"), dispatchPRNum(2, "sha2"), dispatchPRNum(3, "sha3")} + plan := PlanDispatch(prs, Artifact{}, DispatchState{}, DispatchOptions{ + RequireApproval: true, + FanOut: true, + MaxParallelReviews: 3, + MaxPerspectivesPerPR: 1, + ProjectOrg: "acme", + Agents: []AgentCapability{reviewer("r1"), reviewer("r2"), reviewer("r3")}, + }) + + if len(plan.ReviewKicks) != 3 { + t.Fatalf("got %d kicks, want 3", len(plan.ReviewKicks)) + } + seen := map[int]int{} + for _, k := range plan.ReviewKicks { + seen[k.Number]++ + } + if len(seen) != 3 { + t.Fatalf("capped dispatch covered %d distinct PRs, want 3: %+v", len(seen), seen) + } + for num, n := range seen { + if n != 1 { + t.Errorf("PR #%d got %d perspectives, want 1 under the cap", num, n) + } + } +} + +// TestMaxPerspectivesPerPRNeverExceedsSlotBudget keeps the cap subordinate to +// the parallel budget: raising it must not let dispatch run more reviews at +// once than the hive allows. +func TestMaxPerspectivesPerPRNeverExceedsSlotBudget(t *testing.T) { + prs := []PullRequest{dispatchPRNum(1, "sha1"), dispatchPRNum(2, "sha2")} + plan := PlanDispatch(prs, Artifact{}, DispatchState{}, DispatchOptions{ + RequireApproval: true, + FanOut: true, + MaxParallelReviews: 2, + MaxPerspectivesPerPR: 4, + ProjectOrg: "acme", + Agents: []AgentCapability{reviewer("r1"), reviewer("r2")}, + }) + + if len(plan.ReviewKicks) != 2 { + t.Fatalf("got %d kicks, want 2 (the slot budget, not the per-PR cap)", len(plan.ReviewKicks)) + } +} From e96c72ce2cdbb055423d40be86aa4a75d5246aad Mon Sep 17 00:00:00 2001 From: "kubestellar-hive[bot]" <280983584+kubestellar-hive[bot]@users.noreply.github.com> Date: Thu, 17 Sep 2026 23:22:56 -0400 Subject: [PATCH 06/17] =?UTF-8?q?=F0=9F=94=92=20security:=20make=20the=20t?= =?UTF-8?q?oken-access=20audit=20log=20non-writable=20by=20agents=20(backp?= =?UTF-8?q?ort=20#6303=20to=20v4)=20(#7523)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Backport of #6303 (merged to v5) to the v4 release line, which still ships the vulnerable dev:node 0664 token-access.jsonl: every agent UID is in group node, so any prompt-injected agent could truncate the trail of its own token use, rewrite lines to blame a peer, or forge entries (CWE-284/CWE-345). Fixed on v5 on 2026-09-08; v4 has released ~20 versions since (HEAD is v4.54.0) without it. Deviations from the v5 commit, forced by branch divergence: - src/cmd/hive/notifywire.go does not exist on v4; the StartTokenAccessAuditWatcher call is wired into cmd/hive/main.go next to the existing PrepareRequestDirs call (same placement rationale). - .github/workflows/v2-ci.yml step ('Entrypoint token-access audit log not agent-writable') is NOT included: this agent tier cannot push workflow changes. The test script src/deploy/test_entrypoint_token_access_audit.sh IS included; wiring it into CI needs a human (see tracking issue). Verified: go build ./cmd/hive ./pkg/github ./pkg/dashboard, go vet, and go test ./pkg/github -run TokenAccess all pass on this branch. bin/test_git_credential_hive.sh: 21/22 pass; the one failure is the known live-host non-hermeticity (the UID map resolves the harness UID to the real agent, overriding the claimed name — the forge-guard working as designed, not a defect). Refs #6287 Signed-off-by: sec-check Signed-off-by: Andrew Anderson Co-authored-by: Andy Anderson Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .github/workflows/v2-ci.yml | 9 + bin/gh-wrapper.sh | 34 +- bin/git-credential-hive.sh | 25 +- bin/test_git_credential_hive.sh | 79 +++++ .../security-6287-token-access-audit-log.md | 1 + src/cmd/hive/main.go | 6 + src/deploy/entrypoint.sh | 31 +- .../test_entrypoint_token_access_audit.sh | 103 ++++++ src/pkg/dashboard/api.go | 8 +- src/pkg/github/token_access_audit.go | 317 ++++++++++++++++++ src/pkg/github/token_access_audit_test.go | 254 ++++++++++++++ 11 files changed, 836 insertions(+), 31 deletions(-) create mode 100644 changelog.d/security-6287-token-access-audit-log.md create mode 100755 src/deploy/test_entrypoint_token_access_audit.sh create mode 100644 src/pkg/github/token_access_audit.go create mode 100644 src/pkg/github/token_access_audit_test.go diff --git a/.github/workflows/v2-ci.yml b/.github/workflows/v2-ci.yml index 77fd3562c7..061e9e2838 100644 --- a/.github/workflows/v2-ci.yml +++ b/.github/workflows/v2-ci.yml @@ -145,6 +145,15 @@ jobs: - name: Entrypoint /data ownership invariant (#5369) run: bash deploy/test_entrypoint_data_ownership.sh + # #6287: the token-access audit log (GET /api/token-access) was + # dev:node 0664 while every agent UID is in group node, so the audited + # agents could truncate or forge their own trail. Fixed on v5 by #6303 + # and backported here by #7523. Pins the invariant on the shipped + # entrypoint and wrappers: the log is dev-owned 0600, the wrappers write + # only to the drop-box spool, and no chmod loosens it. + - name: Entrypoint token-access audit log not agent-writable (#6287) + run: bash deploy/test_entrypoint_token_access_audit.sh + # #5370: the arm64 lane probed the PUBLISHED image, so on a PR it # validated code already on v4 rather than the change proposed — a PR # fixing a startup bug stayed red, one introducing a startup bug went diff --git a/bin/gh-wrapper.sh b/bin/gh-wrapper.sh index e749af71a9..17a7ee2323 100755 --- a/bin/gh-wrapper.sh +++ b/bin/gh-wrapper.sh @@ -74,7 +74,16 @@ fi # Inject GitHub App token for agent gh calls (15k/hr vs PAT's 5k/hr). # Contributors keep their personal token — they fork+PR with their own identity. -TOKEN_ACCESS_LOG="/var/run/hive-metrics/token-access.jsonl" +# Token-access audit events (#6287). The durable log the dashboard serves +# (GET /api/token-access) is owned by the hive UID, mode 0600, and NO agent +# can open it: this wrapper runs as the agent UID, and an +# audit log the audited party can write is not an audit log. Events are +# dropped here as one file each and the hive ingests them, attributing each +# to the uid that OWNS the file rather than the uid the event claims. This +# path deliberately has no environment override (an override would let an +# agent redirect its own audit events into the void); it must match +# TokenAccessSpoolDir in pkg/github/token_access_audit.go. +TOKEN_ACCESS_SPOOL="/var/run/hive-metrics/token-access-events" if ! _contributor_mode; then # Per-agent scoped token (Phase 4) — 0640 dev:hive-, least-privilege, # readable ONLY by the owning agent's private group. This is the ONLY token an @@ -112,14 +121,21 @@ if ! _contributor_mode; then echo " The hive delivers a scoped token per agent; report this to the operator so token delivery is repaired." >&2 exit 1 fi - # The group wraps the append so a failed REDIRECTION is silenced too: `>> f - # 2>/dev/null` only mutes the printf, and when the log's directory is not - # writable by the agent UID the shell's own "Permission denied" line leaked - # into stderr on EVERY gh call, priming agents to read later denials as - # permission errors (#4043). - { printf '{"ts":"%s","agent":"%s","uid":%d,"op":"gh","cmd":"gh %s"}\n' \ - "$(date -u +%Y-%m-%dT%H:%M:%SZ)" "${HIVE_AGENT:-unknown}" "$(id -u)" "$*" \ - >> "$TOKEN_ACCESS_LOG"; } 2>/dev/null || true + # Write-then-rename so the ingester never sees a half-written event. The + # subshell pins umask 027: the spool is setgid to the hive's group, so + # 0640 is exactly "the hive can read it, nobody else"; without the pin an + # agent umask of 077 would hand the hive an event it cannot open. The group + # wraps everything so a failed REDIRECTION is silenced too: `> f 2>/dev/null` + # only mutes the printf, and when the spool is not writable by the agent + # UID the shell's own "Permission denied" line leaked into stderr on EVERY + # gh call, priming agents to read later denials as permission errors + # (#4043). + { + _evt="${TOKEN_ACCESS_SPOOL}/$(date -u +%s%N)-$$-${RANDOM}.json" + ( umask 027 && printf '{"ts":"%s","agent":"%s","uid":%d,"op":"gh","cmd":"gh %s"}\n' \ + "$(date -u +%Y-%m-%dT%H:%M:%SZ)" "${HIVE_AGENT:-unknown}" "$(id -u)" "$*" \ + > "${_evt}.tmp" ) && mv -f "${_evt}.tmp" "$_evt" + } 2>/dev/null || true fi # NOTE: the contributor-mode `gh auth` gate USED to live here, matching diff --git a/bin/git-credential-hive.sh b/bin/git-credential-hive.sh index 1c2b842d44..a7969436aa 100644 --- a/bin/git-credential-hive.sh +++ b/bin/git-credential-hive.sh @@ -145,7 +145,12 @@ if [ -z "$TOKEN" ]; then exit 1 fi -TOKEN_ACCESS_LOG="/var/run/hive-metrics/token-access.jsonl" +# Token-access audit events (#6287): see the matching block in gh-wrapper.sh. +# The durable log is hive-owned 0600 and not writable by this (agent) UID; +# events are dropped here one file each and ingested by the hive, which +# attributes each to the uid that owns the file. No environment override on +# purpose; must match TokenAccessSpoolDir in pkg/github/token_access_audit.go. +TOKEN_ACCESS_SPOOL="/var/run/hive-metrics/token-access-events" case "${1:-}" in get) @@ -154,16 +159,14 @@ case "${1:-}" in if [ -n "$REQUEST_PROTOCOL" ] && [ "$REQUEST_PROTOCOL" != "https" ]; then exit 0 fi - # The group wraps the append so a failed REDIRECTION is silenced too, the - # same shape gh-wrapper.sh uses (#4043): `>> f 2>/dev/null` only mutes the - # printf, and when the log cannot be opened by the agent UID — its - # directory is 0755 dev:node by design (#4044) and the file does not - # exist until the entrypoint pre-creates it — the shell's own - # "line N: .../token-access.jsonl: Permission denied" leaked into the - # agent's pane on every clone, fetch, and push. - { printf '{"ts":"%s","agent":"%s","uid":%d,"op":"git-credential","host":"%s"}\n' \ - "$(date -u +%Y-%m-%dT%H:%M:%SZ)" "${AGENT:-unknown}" "$(id -u)" "${REQUESTED_HOST:-unknown}" \ - >> "$TOKEN_ACCESS_LOG"; } 2>/dev/null || true + # Write-then-rename under a pinned umask (0640 into the setgid spool so + # the hive can read it and nobody else can); see gh-wrapper.sh. + { + _evt="${TOKEN_ACCESS_SPOOL}/$(date -u +%s%N)-$$-${RANDOM}.json" + ( umask 027 && printf '{"ts":"%s","agent":"%s","uid":%d,"op":"git-credential","host":"%s"}\n' \ + "$(date -u +%Y-%m-%dT%H:%M:%SZ)" "${AGENT:-unknown}" "$(id -u)" "${REQUESTED_HOST:-unknown}" \ + > "${_evt}.tmp" ) && mv -f "${_evt}.tmp" "$_evt" + } 2>/dev/null || true # Echo back the SAME host git asked about (github.com, github.ibm.com, or # any other configured GitHub Enterprise host) rather than a hardcoded # "github.com" — see the file header. entrypoint.sh only registers this diff --git a/bin/test_git_credential_hive.sh b/bin/test_git_credential_hive.sh index 4994945183..9f2b4518a8 100644 --- a/bin/test_git_credential_hive.sh +++ b/bin/test_git_credential_hive.sh @@ -187,6 +187,85 @@ else echo "PASS: store op does not leak the token" fi +# #6287: the helper must never write the durable audit log itself. It drops +# one event file per `get` into the hive-ingested spool (write-then-rename, +# 0640 so only the hive's group can read it) and the log path does not appear +# in the helper at all. Production deliberately has no environment override +# for the spool path, so redirect the constant in a temporary copy, the same +# way gh-wrapper.test.sh redirects CONTRIBUTOR_MODE_MARKER. +SPOOL="${WORK}/token-access-events" +AUDIT_LOG="${WORK}/token-access.jsonl" +mkdir -p "$SPOOL" +HELPER_COPY="${WORK}/git-credential-hive-spool.sh" +sed "s|TOKEN_ACCESS_SPOOL=\"/var/run/hive-metrics/token-access-events\"|TOKEN_ACCESS_SPOOL=\"${SPOOL}\"|" "$HELPER" >"$HELPER_COPY" +if ! grep -q "TOKEN_ACCESS_SPOOL=\"${SPOOL}\"" "$HELPER_COPY"; then + FAIL=$((FAIL + 1)) + echo "FAIL: could not redirect TOKEN_ACCESS_SPOOL in the test copy (helper constant changed?)" +else + SPOOL_OUT="$( + printf '%b' "$GET_STDIN" | \ + HIVE_AGENT="$TEST_AGENT" HIVE_AGENT_MODE=ADVISORY HIVE_ACMM_LEVEL=2 \ + HIVE_AGENT_TOKEN_CACHE="$TOKEN_CACHE" \ + bash "$HELPER_COPY" get 2>&1 + )" + EVENTS=("$SPOOL"/*.json) + if [[ "$SPOOL_OUT" == *"password=ghs_stubtoken"* ]] && [ -f "${EVENTS[0]:-}" ] && [ "${#EVENTS[@]}" -eq 1 ]; then + PASS=$((PASS + 1)) + echo "PASS: get drops exactly one event into the spool" + else + FAIL=$((FAIL + 1)) + echo "FAIL: get did not drop exactly one spool event (found: ${EVENTS[*]:-none}); output: $SPOOL_OUT" + fi + if [ -f "${EVENTS[0]:-}" ] && grep -q '"op":"git-credential"' "${EVENTS[0]}" \ + && grep -q "\"agent\":\"${TEST_AGENT}\"" "${EVENTS[0]}" \ + && grep -q '"host":"github.com"' "${EVENTS[0]}"; then + PASS=$((PASS + 1)) + echo "PASS: spool event records op, agent and host" + else + FAIL=$((FAIL + 1)) + echo "FAIL: spool event content is wrong: $(cat "${EVENTS[0]:-/dev/null}" 2>/dev/null)" + fi + if ! compgen -G "${SPOOL}/*.tmp" >/dev/null; then + PASS=$((PASS + 1)) + echo "PASS: no half-written .tmp event left behind" + else + FAIL=$((FAIL + 1)) + echo "FAIL: a .tmp event was left in the spool" + fi + EVENT_MODE="$(stat -c '%a' "${EVENTS[0]:-/dev/null}" 2>/dev/null || stat -f '%OLp' "${EVENTS[0]:-/dev/null}" 2>/dev/null || echo '?')" + if [ "$EVENT_MODE" = "640" ]; then + PASS=$((PASS + 1)) + echo "PASS: spool event is 0640 regardless of the caller's umask" + else + FAIL=$((FAIL + 1)) + echo "FAIL: spool event mode is ${EVENT_MODE}, want 640" + fi + if [ ! -e "$AUDIT_LOG" ] && ! grep -q "token-access.jsonl" "$HELPER"; then + PASS=$((PASS + 1)) + echo "PASS: helper never opens the durable audit log (only the hive writes it)" + else + FAIL=$((FAIL + 1)) + echo "FAIL: helper touched or references the durable audit log path" + fi + # An unwritable spool must not break the credential flow: the audit relay + # fails silent, the credential still flows (#4043 stderr-hygiene contract). + chmod 500 "$SPOOL" + RO_OUT="$( + printf '%b' "$GET_STDIN" | \ + HIVE_AGENT="$TEST_AGENT" HIVE_AGENT_MODE=ADVISORY HIVE_ACMM_LEVEL=2 \ + HIVE_AGENT_TOKEN_CACHE="$TOKEN_CACHE" \ + bash "$HELPER_COPY" get 2>&1 + )" + chmod 700 "$SPOOL" + if [[ "$RO_OUT" == *"password=ghs_stubtoken"* ]] && [[ "$RO_OUT" != *"Permission denied"* ]]; then + PASS=$((PASS + 1)) + echo "PASS: unwritable spool neither blocks the credential nor leaks EACCES to stderr" + else + FAIL=$((FAIL + 1)) + echo "FAIL: unwritable spool changed helper behaviour: $RO_OUT" + fi +fi + echo echo "=== $PASS passed, $FAIL failed ===" [ "$FAIL" -eq 0 ] || exit 1 diff --git a/changelog.d/security-6287-token-access-audit-log.md b/changelog.d/security-6287-token-access-audit-log.md new file mode 100644 index 0000000000..38276d1eed --- /dev/null +++ b/changelog.d/security-6287-token-access-audit-log.md @@ -0,0 +1 @@ +- The token-access audit log behind `GET /api/token-access` is no longer writable by the agents it audits ([#6287](https://github.com/hivecommons/hive/issues/6287)). The log records every gh CLI command and git credential lookup an agent makes and is gated at owner role for exactly that reason, yet it was fed by the audited parties: `bin/gh-wrapper.sh` and `bin/git-credential-hive.sh` run as the agent UID and appended straight into `/var/run/hive-metrics/token-access.jsonl`, which only works when the file is writable by every agent (dev:node 0664, and every agent's primary group is node). Append is indistinguishable from write at the permission level, so a compromised or prompt-injected agent could truncate the trail of its own token use, rewrite lines to blame a peer, or forge entries outright. The wrappers now drop one JSON event per call into `/var/run/hive-metrics/token-access-events`, a drop-box the agents can create files in but cannot list (0730, sticky, setgid), and the hive process ingests those events into the log, which is now hive-owned 0600 with no group or other bit at all. On ingest the hive replaces each event's self-reported `uid` with the uid that owns the event file, the same trust anchor the PR-request watcher uses, and keeps a disagreeing claim beside it as `claimed_uid`, so an entry forged in a peer's name lands attributed to the forger. Malformed or oversized events are rejected rather than appended. A 0664 log left by an older image is tightened on every boot and on every ingest pass instead of trusted. What remains open is narrower than before: an agent can still delete its own event during the two-second window before ingest, and can still flood the log with real calls; it can no longer alter anything already recorded. diff --git a/src/cmd/hive/main.go b/src/cmd/hive/main.go index bf3dfb75f8..dc62cd62d2 100644 --- a/src/cmd/hive/main.go +++ b/src/cmd/hive/main.go @@ -2008,6 +2008,12 @@ func main() { // installation ID, /gh-setup persists it, auto-discovery finds it later), so // this gap silently disarms agent writes on a hive that looks healthy. github.PrepareRequestDirs(logger) + // Token-access audit ingest (#6287): the per-UID wrappers record every gh + // call and credential lookup as an event file, and this loop folds them + // into the hive-owned 0600 audit log that GET /api/token-access serves. + // Unconditional, like the request dirs: agents touch tokens whether or + // not the App is usable, and the trail must never depend on App state. + github.StartTokenAccessAuditWatcher(ctx, logger) if ghClient != nil && cfg.GitHub.HasUsableApp() { // Attribution resolver: effective backend/model from the manager diff --git a/src/deploy/entrypoint.sh b/src/deploy/entrypoint.sh index 0c3ae797e3..7e3ebcda40 100644 --- a/src/deploy/entrypoint.sh +++ b/src/deploy/entrypoint.sh @@ -808,18 +808,29 @@ if [ "$(id -u)" = "0" ]; then # group-writable mode would let any agent swap that file and spoof the # identity the gate validates against. Re-asserted on every boot. chmod 755 /var/run/hive-metrics/agent-tokens 2>/dev/null || true - # The token-access audit log (GET /api/token-access) is APPENDED by the - # per-UID agent processes — gh-wrapper.sh on every gh call and - # git-credential-hive.sh on every credential lookup — but the directory - # above is deliberately not agent-writable, so an agent can never create - # the file and every append failed silently: the audit endpoint on a - # per-UID hive stayed empty forever. Pre-create it here, owned by dev with - # group "node" (every agent UID) writable, so the appends land. The - # directory itself stays 0755: only this one file opens up, the - # bot-identity file the gh-wrapper author gate trusts is untouched. + # The token-access audit log (GET /api/token-access) is NOT agent-writable + # (#6287). v4 pre-created it dev:node 0664 so the per-UID wrappers could + # append to it directly, which handed every agent (all in group node) the + # power to truncate or forge the trail of its own token use. The wrappers + # now drop one event file each into the spool below and the Go process + # (running as dev) ingests them into the log, attributing each event to + # the uid that owns the spool file. The log is therefore dev-owned 0600: + # no group bit at all, because group node IS the agents. Re-asserted on + # every boot so a 0664 file left by an older image is tightened, not + # trusted. The Go side (PrepareTokenAccessAudit) creates the spool with + # drop-box perms and re-tightens the log on every ingest pass; this is the + # root-phase belt to that braces, so the mode holds even if the file is + # somehow not dev-owned. NEVER give this file a group or other bit. touch /var/run/hive-metrics/token-access.jsonl 2>/dev/null || true chown dev:node /var/run/hive-metrics/token-access.jsonl 2>/dev/null || true - chmod 664 /var/run/hive-metrics/token-access.jsonl 2>/dev/null || true + chmod 600 /var/run/hive-metrics/token-access.jsonl 2>/dev/null || true + # Drop-box for the wrappers' events: dev-owned, group node can create and + # rename its own files but cannot list (0730), setgid so dropped files + # inherit group node (readable by dev), sticky so only a file's owner or + # dev can unlink it. Mirrors pkg/github's requestDirMode drop-box. + mkdir -p /var/run/hive-metrics/token-access-events 2>/dev/null || true + chown dev:node /var/run/hive-metrics/token-access-events 2>/dev/null || true + chmod 3730 /var/run/hive-metrics/token-access-events 2>/dev/null || true # Fix permissions on bind-mounted secret files (host may own them as # a different UID with mode 600, making them unreadable by dev/UID 1001) diff --git a/src/deploy/test_entrypoint_token_access_audit.sh b/src/deploy/test_entrypoint_token_access_audit.sh new file mode 100755 index 0000000000..53e2945946 --- /dev/null +++ b/src/deploy/test_entrypoint_token_access_audit.sh @@ -0,0 +1,103 @@ +#!/usr/bin/env bash +# #6287: the token-access audit log must never be agent-writable. +# +# The entrypoint's root phase pre-creates /var/run/hive-metrics/token-access.jsonl. +# v4 made it dev:node 0664, and every agent UID's primary group is node, so every +# agent could truncate or forge the operator-only trail behind GET /api/token-access. +# The log is now dev-owned 0600, the agent-side wrappers drop events into a +# drop-box spool instead, and the Go process ingests them. The permissions +# watcher never touches /var/run/hive-metrics, so nothing after boot loosens it. +# +# The entrypoint needs root to execute, so this pins the invariant STATICALLY on +# the shipped source: the log gets exactly mode 600 and no other chmod in the +# file ever hands it a group or other bit; the spool is a drop-box (no group +# read, sticky, setgid); and neither wrapper opens the log path at all. +# +# Run: bash src/deploy/test_entrypoint_token_access_audit.sh +set -uo pipefail + +PASS=0 +FAIL=0 +HERE="$(cd "$(dirname "$0")" && pwd)" +ENTRYPOINT="${HERE}/entrypoint.sh" +REPO_ROOT="$(cd "${HERE}/../.." && pwd)" +LOG_PATH="/var/run/hive-metrics/token-access.jsonl" +SPOOL_PATH="/var/run/hive-metrics/token-access-events" + +ok() { echo " PASS: $1"; PASS=$((PASS + 1)); } +bad() { echo " FAIL: $1"; [ -n "${2:-}" ] && echo " $2"; FAIL=$((FAIL + 1)); } + +echo "=== #6287: token-access audit log is not agent-writable ===" + +# 1. The log is created with mode 600, and 600 is the ONLY mode it ever gets. +log_chmods="$(grep -E "^[[:space:]]*chmod[[:space:]]+[0-7]+[[:space:]]+${LOG_PATH}" "$ENTRYPOINT" | awk '{print $2}' | sort -u)" +if [ "$log_chmods" = "600" ]; then + ok "entrypoint gives ${LOG_PATH} mode 600 and nothing else" +else + bad "entrypoint chmods ${LOG_PATH} to: '${log_chmods:-}' (want exactly 600)" \ + "every agent UID is in group node; any group bit on this file is an agent bit" +fi + +# 2. No chmod anywhere in the entrypoint grants group/other write to the log, +# including a chmod of the parent that could be recursive. +if grep -nE "chmod[[:space:]]+(-[a-zA-Z]*R[a-zA-Z]*[[:space:]]+)?[0-7]*[2367][0-7]?[[:space:]]+${LOG_PATH}" "$ENTRYPOINT" >/dev/null \ + || grep -nE "chmod[[:space:]]+-[a-zA-Z]*R[a-zA-Z]*[[:space:]]+[0-7]+[[:space:]]+/var/run/hive-metrics([[:space:]]|/?$|/?[[:space:]])" "$ENTRYPOINT" >/dev/null; then + bad "a chmod in the entrypoint can hand agents write access to the audit log" +else + ok "no chmod in the entrypoint grants group/other write on the audit log" +fi + +# 3. The log is owned by dev (the hive UID), never an agent. +if grep -qE "^[[:space:]]*chown[[:space:]]+dev:node[[:space:]]+${LOG_PATH}" "$ENTRYPOINT"; then + ok "audit log is chowned to dev (the hive UID)" +else + bad "audit log is not chowned to dev:node" +fi + +# 4. The spool is a drop-box: 3730 = setgid + sticky + owner rwx + group wx. +spool_chmod="$(grep -E "^[[:space:]]*chmod[[:space:]]+[0-7]+[[:space:]]+${SPOOL_PATH}" "$ENTRYPOINT" | awk '{print $2}' | sort -u)" +if [ "$spool_chmod" = "3730" ]; then + ok "event spool is a 3730 drop-box (agents create, cannot list; sticky; setgid)" +else + bad "event spool chmod is '${spool_chmod:-}' (want 3730)" +fi + +# 5. Neither agent-side wrapper references the log path: the only writer is the +# hive process. Both must point at the spool the Go ingester watches. +for wrapper in bin/gh-wrapper.sh bin/git-credential-hive.sh; do + f="${REPO_ROOT}/${wrapper}" + if grep -q "token-access.jsonl" "$f"; then + bad "${wrapper} still references the audit log path; agents must not write it" + elif grep -q "TOKEN_ACCESS_SPOOL=\"${SPOOL_PATH}\"" "$f"; then + ok "${wrapper} writes only to the event spool" + else + bad "${wrapper} does not point at ${SPOOL_PATH}" + fi + if grep -qE "TOKEN_ACCESS_SPOOL=\"\\\$\{" "$f"; then + bad "${wrapper} lets the environment override the spool path" \ + "an override would let an agent redirect its own audit events into the void" + fi +done + +# 6. The Go side agrees on both paths and on the 0600 mode. +go_src="${REPO_ROOT}/src/pkg/github/token_access_audit.go" +if grep -q "TokenAccessSpoolDir = \"${SPOOL_PATH}\"" "$go_src" \ + && grep -q "TokenAccessLogPath = \"${LOG_PATH}\"" "$go_src" \ + && grep -qE "tokenAccessLogMode[[:space:]]+os.FileMode[[:space:]]*=[[:space:]]*0o600" "$go_src"; then + ok "Go ingester uses the same spool, the same log, and mode 0600" +else + bad "Go ingester constants disagree with the entrypoint/wrappers" +fi + +# 7. The Go permissions watcher never has /var/run/hive-metrics in its roots, +# so nothing running as dev after boot can re-open the hole (#6238 / #6277 +# moved umask handling around; the watcher's roots are the only mode-fixer). +if grep -q '"/var/run/hive-metrics' "${REPO_ROOT}/src/pkg/agent/permissions_watcher.go"; then + bad "the permissions watcher walks /var/run/hive-metrics and could loosen the audit log" +else + ok "the permissions watcher does not walk /var/run/hive-metrics" +fi + +echo +echo "=== ${PASS} passed, ${FAIL} failed ===" +[ "$FAIL" -eq 0 ] || exit 1 diff --git a/src/pkg/dashboard/api.go b/src/pkg/dashboard/api.go index 1f815e1252..212086169e 100644 --- a/src/pkg/dashboard/api.go +++ b/src/pkg/dashboard/api.go @@ -2663,6 +2663,11 @@ const ( tokenAccessMaxEntries = 100 ) +// tokenAccessLogPath is github.TokenAccessLogPath. The file is written ONLY +// by the hive process (pkg/github's token-access ingester) and is hive-owned +// 0600: the agents whose gh calls it records cannot append to, truncate, or +// read it (#6287). The wrappers drop per-call events into a spool the hive +// ingests, attributing each to the uid that owns the event file. var tokenAccessLogPath = "/var/run/hive-metrics/token-access.jsonl" func (s *Server) handleTokenAccess(w http.ResponseWriter, r *http.Request) { @@ -2671,7 +2676,8 @@ func (s *Server) handleTokenAccess(w http.ResponseWriter, r *http.Request) { // --body ...). Without a role gate any authenticated user — including // read-only contributors — could enumerate the hive's full GitHub operation // history. Gate at owner-role, consistent with handleConfigDownload and - // handleSelfUpgrade which protect equivalent operator-only data. + // handleSelfUpgrade which protect equivalent operator-only data. The + // write side is protected too: see tokenAccessLogPath. if !requireOwnerRole(w, r) { return } diff --git a/src/pkg/github/token_access_audit.go b/src/pkg/github/token_access_audit.go new file mode 100644 index 0000000000..5f6d5a08d3 --- /dev/null +++ b/src/pkg/github/token_access_audit.go @@ -0,0 +1,317 @@ +package github + +import ( + "context" + "encoding/json" + "log/slog" + "os" + "path/filepath" + "sort" + "strings" + "time" +) + +// Token-access audit trail (#6287). +// +// THE DEFECT: GET /api/token-access is an operator-only security audit trail +// ("every gh CLI command an agent issued", gated at owner role, CWE-284) and +// it was fed by the audited parties writing straight into the log file. +// bin/gh-wrapper.sh and bin/git-credential-hive.sh run AS the agent UID and +// used to `>>` the JSONL file directly, which only works when the file is +// writable by every agent (v4 pre-created it dev:node 0664, and every agent's +// primary group is node). Append is indistinguishable from write at the file +// permission level, so any prompt-injected agent could truncate the trail of +// its own token use, rewrite lines to attribute its activity to a peer, or +// forge entries wholesale. An audit log the audited party can edit is not an +// audit log. +// +// THE FIX: the same drop-box relay every other agent->hive write already uses +// (pr-requests, issue-requests, merge-requests). The wrappers write one JSON +// event per gh call / credential lookup into TokenAccessSpoolDir, a directory +// agents can create files in but cannot list, and only the file's owner (or +// the hive, which owns the directory) can unlink. The hive process ingests +// those events into TokenAccessLogPath, which is owned by the hive UID with +// mode 0600: no agent can open it for writing, truncating, or appending, and +// no agent can read the fleet-wide history either (the endpoint is owner-only +// for exactly that reason). While ingesting, the hive replaces the event's +// self-reported "uid" with the spool file's OWNING uid, the same trust anchor +// the PR-request watcher uses: an agent can only ever speak for itself, so an +// entry forged in a peer's name lands attributed to the forger, with the +// claimed uid preserved alongside as evidence. +// +// What this does NOT close: an agent can still delete its OWN event during +// the poll window before the hive picks it up (a hive-side unix socket with +// SO_PEERCRED would close that too, and is the natural next step), and it can +// still flood the log with real gh calls. Neither lets it alter a line that +// has already been ingested, which is the invariant the tests assert. + +// TokenAccessSpoolDir is where the per-agent wrappers drop token-access +// events. Mirrors PRRequestDir. Changing it requires changing the constant in +// bin/gh-wrapper.sh and bin/git-credential-hive.sh, which deliberately carry +// no environment override: an override would let an agent redirect its own +// audit events into the void. +const TokenAccessSpoolDir = "/var/run/hive-metrics/token-access-events" + +// TokenAccessLogPath is the durable, hive-owned audit log the dashboard +// serves. Kept at its historical path so existing operator tooling and +// pkg/dashboard's reader keep working. +const TokenAccessLogPath = "/var/run/hive-metrics/token-access.jsonl" + +// tokenAccessLogMode is the ONLY acceptable permission set for the durable +// log: owner (the hive UID) read/write, nobody else anything. Every agent UID +// shares the hive's primary group ("node"), so any group bit at all is an +// agent bit. Re-asserted on every boot and on every ingest pass so a file +// left behind by a v4 image (dev:node 0664) is tightened rather than trusted. +const tokenAccessLogMode os.FileMode = 0o600 + +// tokenAccessSpoolDirMode is the drop-box mode for the spool directory: +// owner rwx, group write+search WITHOUT read. Agents (group node) can create +// and rename their own event files but cannot enumerate the directory, so +// one agent cannot discover a peer's not-yet-ingested events. setgid makes +// every dropped file inherit the node group so the hive (also group node) +// can read it regardless of the agent's umask; sticky keeps deletion to the +// file's owner and the directory's owner, as on /tmp. See requestDirMode for +// why the special bits must be os.Mode* and not raw octal. +const tokenAccessSpoolDirMode = 0o730 | os.ModeSetgid | os.ModeSticky + +// tokenAccessSpoolTmpSuffix marks an event still being written. The wrappers +// write to .tmp and rename into place, so the ingester never reads a +// half-written line. +const tokenAccessSpoolTmpSuffix = ".tmp" + +// tokenAccessSpoolStaleTmpAge is how old an abandoned .tmp file must be before +// the ingester removes it (a wrapper killed between write and rename). +const tokenAccessSpoolStaleTmpAge = 10 * time.Minute + +// tokenAccessMaxEventBytes bounds one event. A real event is under 1 KiB; the +// cap stops an agent turning the spool into a disk-filling channel or pushing +// a multi-megabyte "line" into the log the dashboard splits in memory. +const tokenAccessMaxEventBytes = 8 * 1024 + +// tokenAccessPollInterval is how often the ingester scans the spool. Short, +// because an event sitting in the spool is still deletable by its author; the +// window between drop and ingest is the only tamper window left. +var tokenAccessPollInterval = 2 * time.Second + +// tokenAccessSpoolDirForTest / tokenAccessLogPathForTest let tests redirect +// the ingester into a temp dir. Empty means production paths. +var ( + tokenAccessSpoolDirForTest string + tokenAccessLogPathForTest string +) + +func tokenAccessSpoolDir() string { + if tokenAccessSpoolDirForTest != "" { + return tokenAccessSpoolDirForTest + } + return TokenAccessSpoolDir +} + +func tokenAccessLogPath() string { + if tokenAccessLogPathForTest != "" { + return tokenAccessLogPathForTest + } + return TokenAccessLogPath +} + +// PrepareTokenAccessAudit creates the spool drop-box and the hive-owned log, +// and tightens the log to tokenAccessLogMode. Runs at boot regardless of +// GitHub App state: the wrappers emit events whenever an agent touches a +// token, App or not, and the trail must exist to receive them. Returns false +// only when the spool cannot be created, in which case the ingester has +// nothing to watch and disables itself. +func PrepareTokenAccessAudit(logger *slog.Logger) bool { + spool, logPath := tokenAccessSpoolDir(), tokenAccessLogPath() + if err := os.MkdirAll(spool, 0o777); err != nil { + if logger != nil { + logger.Warn("token-access audit: cannot create event spool; agent token use will not be recorded", + slog.String("dir", spool), slog.String("error", err.Error())) + } + return false + } + if err := os.Chmod(spool, tokenAccessSpoolDirMode); err != nil && logger != nil { + logger.Warn("token-access audit: could not set drop-box perms on event spool; agents may be unable to record token use", + slog.String("dir", spool), slog.String("error", err.Error())) + } + f, err := openTokenAccessLog(logPath) + if err != nil { + if logger != nil { + logger.Warn("token-access audit: cannot open audit log", + slog.String("path", logPath), slog.String("error", err.Error())) + } + return true + } + _ = f.Close() + return true +} + +// openTokenAccessLog opens the durable log append-only as the hive UID and +// enforces tokenAccessLogMode on it. The mode is enforced with an explicit +// Chmod, not just the create mode, because the file may pre-exist with the +// v4 group-writable mode and O_CREATE does not touch an existing file's bits. +func openTokenAccessLog(path string) (*os.File, error) { + f, err := os.OpenFile(path, os.O_WRONLY|os.O_APPEND|os.O_CREATE, tokenAccessLogMode) + if err != nil { + return nil, err + } + if fi, err := f.Stat(); err == nil && fi.Mode().Perm() != tokenAccessLogMode { + if err := f.Chmod(tokenAccessLogMode); err != nil { + _ = f.Close() + return nil, err + } + } + return f, nil +} + +// StartTokenAccessAuditWatcher runs the ingest loop until ctx is cancelled. +// Same contract as StartPRRequestWatcher: the returned channel closes when +// the loop exits, and the watcher disables itself when the spool cannot be +// created. +func StartTokenAccessAuditWatcher(ctx context.Context, logger *slog.Logger) <-chan struct{} { + done := make(chan struct{}) + if !PrepareTokenAccessAudit(logger) { + close(done) + return done + } + spool, logPath := tokenAccessSpoolDir(), tokenAccessLogPath() + // Captured before spawning for the same race-detector reason as + // StartPRRequestWatcher. + interval := tokenAccessPollInterval + go func() { + defer close(done) + t := time.NewTicker(interval) + defer t.Stop() + for { + select { + case <-ctx.Done(): + return + case <-t.C: + if ctx.Err() != nil { + return + } + IngestTokenAccessEventsOnce(logger, spool, logPath, time.Now()) + } + } + }() + return done +} + +// IngestTokenAccessEventsOnce moves every complete event in spool into the +// durable log, oldest first, and returns how many were appended. Events that +// are not a single JSON object, or exceed tokenAccessMaxEventBytes, are +// dropped with a warning rather than appended: the log must only ever hold +// lines the dashboard can hand back as json.RawMessage. +func IngestTokenAccessEventsOnce(logger *slog.Logger, spool, logPath string, now time.Time) int { + entries, err := os.ReadDir(spool) + if err != nil { + if logger != nil && !os.IsNotExist(err) { + logger.Warn("token-access audit: cannot read event spool", + slog.String("dir", spool), slog.String("error", err.Error())) + } + return 0 + } + names := make([]string, 0, len(entries)) + for _, e := range entries { + if !e.Type().IsRegular() { + continue + } + name := e.Name() + if strings.HasSuffix(name, tokenAccessSpoolTmpSuffix) { + if info, err := e.Info(); err == nil && now.Sub(info.ModTime()) > tokenAccessSpoolStaleTmpAge { + _ = os.Remove(filepath.Join(spool, name)) + } + continue + } + names = append(names, name) + } + if len(names) == 0 { + return 0 + } + // Wrapper file names start with a nanosecond timestamp, so lexical order + // is arrival order. + sort.Strings(names) + + out, err := openTokenAccessLog(logPath) + if err != nil { + if logger != nil { + logger.Warn("token-access audit: cannot open audit log; leaving events in spool", + slog.String("path", logPath), slog.String("error", err.Error())) + } + return 0 + } + defer out.Close() + + appended := 0 + for _, name := range names { + path := filepath.Join(spool, name) + line, ok := readTokenAccessEvent(logger, path) + if ok { + if _, err := out.Write(append(line, '\n')); err != nil { + if logger != nil { + logger.Warn("token-access audit: append failed; leaving event in spool", + slog.String("event", name), slog.String("error", err.Error())) + } + return appended + } + appended++ + } + // Consumed (or rejected): the hive owns the spool directory, so the + // sticky bit does not stop it unlinking an agent-owned file. + if err := os.Remove(path); err != nil && logger != nil { + logger.Warn("token-access audit: cannot remove consumed event", + slog.String("event", name), slog.String("error", err.Error())) + } + } + return appended +} + +// tokenAccessClaimedUIDKey is where a mismatching self-reported uid is kept +// when the ingester overrides it with the spool file's owner. +const tokenAccessClaimedUIDKey = "claimed_uid" + +// readTokenAccessEvent validates one spool file and returns the compact JSON +// line to append. The event's "uid" is replaced by the file's owning uid +// where the platform reports one (Linux in production); a self-reported uid +// that disagrees is preserved under "claimed_uid" so a forgery attempt is +// itself on the record. +func readTokenAccessEvent(logger *slog.Logger, path string) ([]byte, bool) { + fi, err := os.Lstat(path) + if err != nil { + return nil, false + } + if fi.Size() == 0 || fi.Size() > tokenAccessMaxEventBytes { + if logger != nil { + logger.Warn("token-access audit: dropping event of unacceptable size", + slog.String("event", filepath.Base(path)), slog.Int64("bytes", fi.Size())) + } + return nil, false + } + raw, err := os.ReadFile(path) + if err != nil { + if logger != nil { + logger.Warn("token-access audit: cannot read event", + slog.String("event", filepath.Base(path)), slog.String("error", err.Error())) + } + return nil, false + } + var event map[string]json.RawMessage + if err := json.Unmarshal(raw, &event); err != nil || event == nil { + if logger != nil { + logger.Warn("token-access audit: dropping event that is not a JSON object", + slog.String("event", filepath.Base(path))) + } + return nil, false + } + if owner := fileOwnerUID(fi); owner >= 0 { + ownerJSON, _ := json.Marshal(owner) + if claimed, present := event["uid"]; present && string(claimed) != string(ownerJSON) { + event[tokenAccessClaimedUIDKey] = claimed + } + event["uid"] = json.RawMessage(ownerJSON) + } + line, err := json.Marshal(event) + if err != nil { + return nil, false + } + return line, true +} diff --git a/src/pkg/github/token_access_audit_test.go b/src/pkg/github/token_access_audit_test.go new file mode 100644 index 0000000000..8db68eef72 --- /dev/null +++ b/src/pkg/github/token_access_audit_test.go @@ -0,0 +1,254 @@ +package github + +import ( + "context" + "encoding/json" + "os" + "path/filepath" + "runtime" + "strings" + "testing" + "time" + + "github.com/hivecommons/hive/internal/testutil" +) + +// #6287: the token-access audit log must not be writable by the agents it +// audits. These tests assert the INVARIANT on the durable log (owned by the +// hive process, no group/other bits at all, so no agent UID can open it for +// write, append or truncate) and the attribution rule on ingest (an event is +// attributed to the uid that OWNS the spool file, never to the uid it claims). +// They deliberately do not assert on log output: a warning is not a control. + +func testTokenAccessPaths(t *testing.T) (spool, logPath string) { + t.Helper() + root := t.TempDir() + spool = filepath.Join(root, "token-access-events") + logPath = filepath.Join(root, "token-access.jsonl") + prevSpool, prevLog := tokenAccessSpoolDirForTest, tokenAccessLogPathForTest + tokenAccessSpoolDirForTest, tokenAccessLogPathForTest = spool, logPath + t.Cleanup(func() { tokenAccessSpoolDirForTest, tokenAccessLogPathForTest = prevSpool, prevLog }) + return spool, logPath +} + +func dropTokenAccessEvent(t *testing.T, spool, name, body string) { + t.Helper() + if err := os.WriteFile(filepath.Join(spool, name), []byte(body), 0o640); err != nil { + t.Fatal(err) + } +} + +func readLogLines(t *testing.T, logPath string) []map[string]any { + t.Helper() + data, err := os.ReadFile(logPath) + if err != nil { + t.Fatal(err) + } + var out []map[string]any + for _, line := range strings.Split(strings.TrimSpace(string(data)), "\n") { + if line == "" { + continue + } + var m map[string]any + if err := json.Unmarshal([]byte(line), &m); err != nil { + t.Fatalf("log line is not a JSON object: %q: %v", line, err) + } + out = append(out, m) + } + return out +} + +// assertLogNotAgentWritable is the invariant: the log exists, is owned by the +// process running the hive, and carries no group or other permission bit. +// Every agent UID shares the hive's primary group, so a single group bit is +// an agent bit; 0600 is the only acceptable mode. +func assertLogNotAgentWritable(t *testing.T, logPath string) { + t.Helper() + fi, err := os.Stat(logPath) + if err != nil { + t.Fatalf("audit log missing: %v", err) + } + if got := fi.Mode().Perm(); got != tokenAccessLogMode { + t.Fatalf("audit log mode = %04o, want %04o (agents must have no access at all)", got, tokenAccessLogMode) + } + if got := fi.Mode().Perm() & 0o077; got != 0 { + t.Fatalf("audit log grants group/other bits %04o; an agent UID could write it", got) + } + if owner := fileOwnerUID(fi); owner >= 0 && owner != os.Getuid() { + t.Fatalf("audit log owner uid = %d, want the hive process uid %d", owner, os.Getuid()) + } +} + +// A log left behind by the previous design (dev:node 0664, group-writable by +// every agent) must be tightened at boot, not trusted. +func TestTokenAccessAudit_PrepareTightensLegacyGroupWritableLog(t *testing.T) { + _, logPath := testTokenAccessPaths(t) + if err := os.WriteFile(logPath, []byte(`{"op":"gh","uid":2001}`+"\n"), 0o664); err != nil { + t.Fatal(err) + } + if err := os.Chmod(logPath, 0o664); err != nil { // umask-proof + t.Fatal(err) + } + if !PrepareTokenAccessAudit(quietLogger()) { + t.Fatal("PrepareTokenAccessAudit returned false") + } + assertLogNotAgentWritable(t, logPath) + // Tightening must not lose what was already recorded. + if lines := readLogLines(t, logPath); len(lines) != 1 { + t.Fatalf("existing entries lost on tighten: got %d lines", len(lines)) + } +} + +// A fresh boot creates the log with the invariant already in force, and the +// spool as a drop-box agents can write into but not enumerate. +func TestTokenAccessAudit_PrepareCreatesHiveOwnedLogAndDropBox(t *testing.T) { + spool, logPath := testTokenAccessPaths(t) + if !PrepareTokenAccessAudit(quietLogger()) { + t.Fatal("PrepareTokenAccessAudit returned false") + } + assertLogNotAgentWritable(t, logPath) + + fi, err := os.Stat(spool) + if err != nil { + t.Fatalf("spool missing: %v", err) + } + if got := fi.Mode().Perm(); got != 0o730 { + t.Fatalf("spool perm = %04o, want 0730 (group create+search, no list)", got) + } + if fi.Mode()&os.ModeSticky == 0 { + t.Fatal("spool lacks the sticky bit; an agent could unlink a peer's queued event") + } + if runtime.GOOS == "linux" && fi.Mode()&os.ModeSetgid == 0 { + t.Fatal("spool lacks setgid; dropped events would not inherit the hive-readable group") + } +} + +// Ingest attributes each event to the uid that owns the spool file. An event +// claiming to be some other agent lands under the writer's real uid with the +// claim preserved as evidence, and the log stays non-writable afterwards. +func TestTokenAccessAudit_IngestAttributesByFileOwnerNotClaim(t *testing.T) { + if runtime.GOOS != "linux" && runtime.GOOS != "darwin" { + t.Skip("file ownership is only reported on unix") + } + spool, logPath := testTokenAccessPaths(t) + if !PrepareTokenAccessAudit(quietLogger()) { + t.Fatal("PrepareTokenAccessAudit returned false") + } + me := os.Getuid() + forgedUID := me + 1000 + dropTokenAccessEvent(t, spool, "1000000000000000001-11.json", + `{"ts":"2026-09-08T00:00:00Z","agent":"scanner","uid":`+itoa(me)+`,"op":"gh","cmd":"gh pr list"}`) + dropTokenAccessEvent(t, spool, "1000000000000000002-12.json", + `{"ts":"2026-09-08T00:00:01Z","agent":"peer","uid":`+itoa(forgedUID)+`,"op":"gh","cmd":"gh pr merge 1"}`) + + if n := IngestTokenAccessEventsOnce(quietLogger(), spool, logPath, time.Now()); n != 2 { + t.Fatalf("ingested %d events, want 2", n) + } + assertLogNotAgentWritable(t, logPath) + + lines := readLogLines(t, logPath) + if len(lines) != 2 { + t.Fatalf("got %d log lines, want 2", len(lines)) + } + if got := lines[0]["uid"]; got != float64(me) { + t.Fatalf("honest event uid = %v, want %d", got, me) + } + if _, present := lines[0][tokenAccessClaimedUIDKey]; present { + t.Fatal("honest event must not carry a claimed_uid") + } + if got := lines[1]["uid"]; got != float64(me) { + t.Fatalf("forged event attributed to uid %v, want the real writer %d", got, me) + } + if got := lines[1][tokenAccessClaimedUIDKey]; got != float64(forgedUID) { + t.Fatalf("forged event claimed_uid = %v, want %d", got, forgedUID) + } + if got := lines[1]["cmd"]; got != "gh pr merge 1" { + t.Fatalf("event payload not preserved: cmd = %v", got) + } + + // Consumed events leave the spool; nothing is left for the author to edit. + entries, err := os.ReadDir(spool) + if err != nil { + t.Fatal(err) + } + if len(entries) != 0 { + t.Fatalf("spool not drained: %d entries remain", len(entries)) + } +} + +// Only well-formed, bounded JSON objects reach the log: the dashboard hands +// lines back verbatim as json.RawMessage, so garbage or an oversized blob +// must be rejected and removed, never appended. In-progress .tmp files are +// left alone until they go stale. +func TestTokenAccessAudit_IngestRejectsMalformedAndOversized(t *testing.T) { + spool, logPath := testTokenAccessPaths(t) + if !PrepareTokenAccessAudit(quietLogger()) { + t.Fatal("PrepareTokenAccessAudit returned false") + } + dropTokenAccessEvent(t, spool, "1-notjson.json", "not json at all\n") + dropTokenAccessEvent(t, spool, "2-array.json", `[1,2,3]`) + dropTokenAccessEvent(t, spool, "3-huge.json", `{"cmd":"`+strings.Repeat("x", tokenAccessMaxEventBytes)+`"}`) + dropTokenAccessEvent(t, spool, "4-empty.json", "") + dropTokenAccessEvent(t, spool, "5-good.json", `{"op":"git-credential","host":"github.com"}`) + dropTokenAccessEvent(t, spool, "6-inflight.json"+tokenAccessSpoolTmpSuffix, `{"op":"gh"}`) + + now := time.Now() + if n := IngestTokenAccessEventsOnce(quietLogger(), spool, logPath, now); n != 1 { + t.Fatalf("ingested %d events, want exactly the one good event", n) + } + lines := readLogLines(t, logPath) + if len(lines) != 1 || lines[0]["op"] != "git-credential" { + t.Fatalf("log = %v, want only the good event", lines) + } + entries, err := os.ReadDir(spool) + if err != nil { + t.Fatal(err) + } + if len(entries) != 1 || entries[0].Name() != "6-inflight.json"+tokenAccessSpoolTmpSuffix { + t.Fatalf("spool after ingest = %v, want only the in-flight .tmp", entries) + } + // A .tmp older than the stale age is an abandoned write and gets swept. + if n := IngestTokenAccessEventsOnce(quietLogger(), spool, logPath, now.Add(2*tokenAccessSpoolStaleTmpAge)); n != 0 { + t.Fatalf("stale sweep appended %d events, want 0", n) + } + if entries, _ = os.ReadDir(spool); len(entries) != 0 { + t.Fatalf("stale .tmp not swept: %v", entries) + } + assertLogNotAgentWritable(t, logPath) +} + +// The boot entry point runs the ingest on a ticker and stops on ctx cancel. +func TestTokenAccessAudit_WatcherIngestsAndStops(t *testing.T) { + spool, logPath := testTokenAccessPaths(t) + prev := tokenAccessPollInterval + tokenAccessPollInterval = 5 * time.Millisecond + t.Cleanup(func() { tokenAccessPollInterval = prev }) + + ctx, cancel := context.WithCancel(context.Background()) + done := StartTokenAccessAuditWatcher(ctx, quietLogger()) + dropTokenAccessEvent(t, spool, "1-evt.json", `{"op":"gh","cmd":"gh issue list"}`) + + testutil.Eventually(t, 5*time.Second, func() bool { + data, err := os.ReadFile(logPath) + return err == nil && strings.Contains(string(data), "gh issue list") + }, "watcher never ingested the event") + cancel() + select { + case <-done: + case <-time.After(5 * time.Second): + t.Fatal("watcher did not stop on cancel") + } + assertLogNotAgentWritable(t, logPath) +} + +// A spool that cannot be created disables the watcher instead of spinning. +func TestTokenAccessAudit_WatcherDisabledWhenSpoolUncreatable(t *testing.T) { + testTokenAccessPaths(t) + tokenAccessSpoolDirForTest = blockedRequestDir(t) + done := StartTokenAccessAuditWatcher(context.Background(), quietLogger()) + select { + case <-done: + case <-time.After(5 * time.Second): + t.Fatal("watcher should have disabled itself on an uncreatable spool") + } +} From 238696983b1006ec96915823d20fbba081faddfe Mon Sep 17 00:00:00 2001 From: Douglas Baggett Date: Thu, 17 Sep 2026 22:25:05 -0400 Subject: [PATCH 07/17] =?UTF-8?q?=F0=9F=90=9B=20fix(hub):=20reconcile=20#7?= =?UTF-8?q?457's=20auth-url=20onto=20the=20hive-contribute=20Ingress=20of?= =?UTF-8?q?=20existing=20spokes?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit #7457 fixed the 401 on /api/contribute/me (#7453) in two halves: code on the hub and the spoke, and two nginx annotations — auth-url and auth-response-headers — on the hive-contribute Ingress in k8sManifestTemplate. The code half rolled out with the next image. The Ingress half is `kubectl apply`ed only by provisionHive, so every hosted spoke provisioned before the merge still serves /api/contribute through an Ingress with no auth-url: nginx never asks the hub who is calling, X-Hive-User never arrives, and /api/contribute/me answers 401 to the hive's own signed-in owner on a spoke whose served_sha is well past the fix (hosted-projectbluefin-common-nmq5, #7517). Add the missing re-apply as a hub reconcile on the NET_ADMIN / per-hive-env pattern (contribute_ingress_reconcile.go): - Every 15 minutes, for each hosted hive on an nginx cluster (OpenShift- Route clusters render Routes, not this Ingress; pull-only clusters have no kubectl path), read the live hive-contribute Ingress and merge-patch the two annotations on when absent or stale. Only the drifted keys go in the patch, so the issuer, timeouts and any vanity-host marks are untouched. A converged Ingress is a Debug no-op; an unparseable one is a Warn and no patch, never a blind one. An annotation patch rolls no pod, so there is no per-cycle cap. - The expected values come from the same inputs the template renders from (hubPublicURL, hive ID); TestContributeIngressReconcileMatchesThe- Template renders the template and pins them equal, so the sweep and provisioning cannot disagree about what converged means. - Wired into the SHA poller next to its siblings; a test pins the call site so the lane cannot be written, tested and never run (#2674). - Sweep accounting: a filter that selects nobody on a populated registry warns, and a sweep that patched or failed anything logs a summary. docs/security-model.md records the general rule the issue asks for: a template change to an existing object reaches only spokes provisioned after it, so it needs a reconcile path or a re-provision note. Not covered here: the sign-in bounce the issue mentions in passing — its redirect chain has not been captured yet, and the issue itself says it may be a separate cookie-scope question. Fixes #7517 Co-Authored-By: Claude Opus 5 (1M context) Signed-off-by: Douglas Baggett --- ...7-contribute-ingress-auth-url-reconcile.md | 1 + src/docs/security-model.md | 6 + src/pkg/hub/contribute_ingress_reconcile.go | 263 +++++++++++++++++ .../hub/contribute_ingress_reconcile_test.go | 273 ++++++++++++++++++ src/pkg/hub/saas_sha_poller.go | 8 + src/pkg/hub/server.go | 5 + 6 files changed, 556 insertions(+) create mode 100644 changelog.d/fixed-7517-contribute-ingress-auth-url-reconcile.md create mode 100644 src/pkg/hub/contribute_ingress_reconcile.go create mode 100644 src/pkg/hub/contribute_ingress_reconcile_test.go diff --git a/changelog.d/fixed-7517-contribute-ingress-auth-url-reconcile.md b/changelog.d/fixed-7517-contribute-ingress-auth-url-reconcile.md new file mode 100644 index 0000000000..8e5f59ed02 --- /dev/null +++ b/changelog.d/fixed-7517-contribute-ingress-auth-url-reconcile.md @@ -0,0 +1 @@ +- Hub: the `auth-url` / `auth-response-headers` annotations [#7457](https://github.com/hivecommons/hive/pull/7457) added to the `hive-contribute` Ingress now reach hosted spokes that were provisioned before that fix ([#7517](https://github.com/hivecommons/hive/issues/7517)). The provisioning template is applied only when a hive is created, so on every pre-existing spoke nginx never asked the hub who was calling and `/api/contribute/me` kept answering `401` to the hive's own signed-in owner — the #7453 symptom, on a spoke whose `served_sha` was well past the fix. A new 15-minute hub sweep (`contribute_ingress_reconcile.go`, alongside the NET_ADMIN and per-hive-env reconciles) reads each hosted spoke's live `hive-contribute` Ingress on nginx clusters and merge-patches the two annotations on when they are missing or stale; a converged Ingress is a no-op, the patch rolls no pod, and the expected values are pinned equal to what the template renders so the sweep and provisioning cannot disagree. OpenShift-Route and pull-only clusters are skipped. `src/docs/security-model.md` now records the general rule: a template change to an existing object needs a reconcile path or a re-provision note. diff --git a/src/docs/security-model.md b/src/docs/security-model.md index 2538c96ecf..69079ef1fd 100644 --- a/src/docs/security-model.md +++ b/src/docs/security-model.md @@ -122,6 +122,12 @@ Two gotchas: - **Empty is worse than absent.** The resolvers fall back on empty values just like missing ones, but an empty var can make convergence accounting report the hive as converged when it is not. Unset rather than blank. - Per-hive env material derives from the **current** master generation — relevant during rotation, below. +### Provisioning-template changes reach existing spokes only through a reconcile + +The provisioning template (`k8sManifestTemplate` in `src/pkg/hub/saas_provision.go`) is `kubectl apply`ed once, when a hive is provisioned. A change to it is born onto every spoke created afterwards and reaches **no spoke that already exists** — those keep whatever object the template rendered on their day. The per-hive env sweep above, the NET_ADMIN sweep, and the vanity-host patch are the reconcilers that close such gaps for the objects they own; a change to any other existing object needs one too, or the PR must say that existing spokes have to be re-provisioned. + +The case that made this rule: [#7457](https://github.com/hivecommons/hive/pull/7457) added `auth-url` / `auth-response-headers` to the `hive-contribute` Ingress so a signed-in visitor's identity reaches `/api/contribute/me`. The code half rolled out with the next image; the Ingress half reached only newly provisioned spokes, and `/api/contribute/me` kept answering `401` everywhere else ([#7517](https://github.com/hivecommons/hive/issues/7517)). The hub now reconciles those two annotations onto every hosted spoke's `hive-contribute` Ingress on nginx clusters (a 15-minute sweep, `contribute_ingress_reconcile.go`; a merge patch on the annotations, which rolls no pod). OpenShift-Route clusters have no nginx Ingress and are skipped. + ### Master key rotation The hub master secret supports **generations**: at most two live at once — one CURRENT (mints new material) and one PREVIOUS (verify-only, default window 7 days). Design details: [design/master-key-rotation.md](design/master-key-rotation.md). diff --git a/src/pkg/hub/contribute_ingress_reconcile.go b/src/pkg/hub/contribute_ingress_reconcile.go new file mode 100644 index 0000000000..592c6ff4be --- /dev/null +++ b/src/pkg/hub/contribute_ingress_reconcile.go @@ -0,0 +1,263 @@ +package hub + +import ( + "context" + "encoding/json" + "fmt" + "strings" + "time" +) + +// hive-contribute Ingress reconcile — closes the pre-#7457 auth-url drift. +// +// #7457 fixed the 401 on /api/contribute/me (#7453) in two halves: code on the +// hub and the spoke, and two nginx annotations on the hive-contribute Ingress +// in k8sManifestTemplate — auth-url, so nginx asks the hub who is calling, and +// auth-response-headers, so the answer (X-Hive-User / X-Hive-Role / +// X-Hive-Proxy-Auth) is copied onto the request. The code half rolled out with +// the next image; the Ingress half is `kubectl apply`ed ONLY by provisionHive, +// so every hosted spoke provisioned before it merged still serves +// /api/contribute through an Ingress with no auth-url. nginx never asks, the +// identity never arrives, and /api/contribute/me answers 401 to the hive's own +// owner — the exact symptom #7453 was filed on, now on a spoke whose +// served_sha is well past the fix (hivecommons/hive#7517). +// +// This sweep is the missing re-apply: for every hosted spoke on an nginx +// cluster it reads the live hive-contribute Ingress and merge-patches the two +// annotations on when they are absent or stale. It follows the NET_ADMIN and +// per-hive-env reconciles (netadmin_reconcile.go, perhive_env_reconcile.go), +// with two differences worth naming: an annotation patch does not roll the +// spoke's pod, so there is no per-cycle cap; and the expected values are +// derived from the same inputs the template renders from (hubPublicURL and +// the hive ID), with a test that pins them equal to the rendered Ingress so +// the sweep and the template cannot disagree about what "converged" means. +// +// More generally (from the issue): a change to k8sManifestTemplate that alters +// an existing object reaches only spokes provisioned after it, unless it has +// a reconcile path like this one. src/docs/security-model.md says so. + +const ( + // contributeIngressName is the Ingress that routes /api/contribute on a + // hosted spoke (k8sManifestTemplate, `name: hive-contribute`). + contributeIngressName = "hive-contribute" + + // The two nginx annotations #7457 added to that Ingress. Named here so + // the sweep's check and its patch can never disagree about the keys. + ingressAuthURLAnnotation = "nginx.ingress.kubernetes.io/auth-url" + ingressAuthResponseHeadersAnnotation = "nginx.ingress.kubernetes.io/auth-response-headers" + + // ingressAuthResponseHeaders is the header list every gated Ingress in the + // template forwards; hive-contribute carries the same one. + ingressAuthResponseHeaders = "X-Hive-User,X-Hive-Role,X-Hive-Proxy-Auth" + + // contributeIngressReconcileInterval throttles the sweep. The drift is + // static — an Ingress either carries the annotations or it does not, and + // a correctly provisioned one never loses them — so this is remediation, + // not a hot path. Same window as the NET_ADMIN sweep. + contributeIngressReconcileInterval = 15 * time.Minute + + // contributeIngressKubectlTimeout bounds each per-hive get/patch so one + // unreachable cluster cannot stall the whole sweep. + contributeIngressKubectlTimeout = 15 * time.Second +) + +// contributeIngressAuthAnnotations is what the hive-contribute Ingress of +// hive `hiveID` must carry for /api/contribute/me to learn who is calling: +// the per-hive auth-url and the identity headers nginx copies back. These are +// the values k8sManifestTemplate renders from the same two inputs +// (TestContributeIngressReconcileMatchesTheTemplate pins that), so a spoke the +// sweep converges is indistinguishable from one provisioned after #7457. +func contributeIngressAuthAnnotations(hubURL, hiveID string) map[string]string { + return map[string]string{ + ingressAuthURLAnnotation: hubURL + "/api/saas/auth-check?hive=" + hiveID + "&uri=$request_uri", + ingressAuthResponseHeadersAnnotation: ingressAuthResponseHeaders, + } +} + +// contributeIngressAnnotationPatch is the PURE reconcile decision: given the +// live Ingress as `kubectl get ingress -o json` prints it and the annotations +// it must carry, return the strategic-merge patch body that installs the +// missing or stale ones — or "" when the Ingress already carries every one, +// so a converged spoke is never patched. Only the annotations that differ are +// in the patch; a merge patch on metadata.annotations leaves every other key +// (cert-manager's issuer, the proxy timeouts, a vanity-host mirror's marks) +// untouched. +// +// An Ingress that does not parse is an error rather than a patch: patching +// blind onto an object the sweep could not read is how a typo becomes a +// fleet-wide outage. +func contributeIngressAnnotationPatch(raw []byte, want map[string]string) (string, error) { + var obj struct { + Metadata struct { + Annotations map[string]string `json:"annotations"` + } `json:"metadata"` + } + if err := json.Unmarshal(raw, &obj); err != nil { + return "", fmt.Errorf("parsing ingress: %w", err) + } + drift := map[string]string{} + for key, value := range want { + if obj.Metadata.Annotations[key] != value { + drift[key] = value + } + } + if len(drift) == 0 { + return "", nil + } + body, err := json.Marshal(map[string]any{ + "metadata": map[string]any{"annotations": drift}, + }) + if err != nil { + return "", fmt.Errorf("encoding patch: %w", err) + } + return string(body), nil +} + +// contributeIngressSweepEligible reports whether a hive should be examined. +// The only status excluded is "provisioning": its Ingress is being applied +// from the template right now and is born with the annotations. Every other +// status — including the unwritten "" most steady-state hives carry, and +// "available" placeholders, which must be right BEFORE they are claimed — is +// swept. See netAdminSweepEligible for why this is not a "running" check. +func contributeIngressSweepEligible(status string) bool { + return strings.TrimSpace(status) != "provisioning" +} + +// reconcileContributeIngressIfDue runs the sweep only if +// contributeIngressReconcileInterval has elapsed. Safe to call from the +// poller loop every tick; same shape and same guarding mutex as +// reconcileNetAdminIfDue — poller-loop-only state. +func (s *HubServer) reconcileContributeIngressIfDue() { + s.clusterUnreachableMu.Lock() + due := s.lastContributeIngressReconcile.IsZero() || + time.Since(s.lastContributeIngressReconcile) >= contributeIngressReconcileInterval + if due { + s.lastContributeIngressReconcile = time.Now() + } + s.clusterUnreachableMu.Unlock() + if !due { + return + } + s.reconcileContributeIngress() +} + +// reconcileContributeIngress sweeps every hub-managed hosted hive on an nginx +// cluster and, for any whose live hive-contribute Ingress is missing the +// auth-url or auth-response-headers annotation (or carries a stale value), +// merge-patches the template's values on. Idempotent — a converged Ingress is +// a Debug-level no-op — and non-fatal on kubectl errors, which are retried +// on the next sweep. An annotation patch does not restart anything: nginx +// re-reads the Ingress and starts asking the hub on the next request. +func (s *HubServer) reconcileContributeIngress() { + hives := listSaaSHives() + hubURL := hubPublicURL() + + // Sweep accounting, so "the filter selected nobody" and "the fleet is + // converged" are different, readable outcomes (the NET_ADMIN sweep + // shipped with a filter that selected nothing and nobody could tell). + considered := 0 + skippedByStatus := 0 + skippedNoIngress := 0 + converged := 0 + patched := 0 + failures := 0 + + for _, h := range hives { + if !contributeIngressSweepEligible(h.Status) { + skippedByStatus++ + continue + } + cluster := s.clusterForHive(&h) + if cluster == nil { + continue + } + // An OpenShift-Route cluster has no nginx Ingress to annotate: the + // template renders Routes there and /api/contribute has no auth-proxy + // at all (see the IsNginxIngress conditional in k8sManifestTemplate). + if cluster.IngressType == ingressTypeOpenShiftRoute { + skippedNoIngress++ + continue + } + // A pull-only cluster is reached only by answering its outbound + // heartbeat; the hub has no kubectl path into it. + if !cluster.KubectlReachable() { + skippedNoIngress++ + continue + } + // Skip clusters the hub just failed to dial — the same suppression the + // upgrade and NET_ADMIN paths use, so one down cluster does not cost a + // timeout per hive every sweep. Recovers on the next sweep after TTL. + if s.clusterRecentlyUnreachable(cluster.ID) { + continue + } + considered++ + + ns := hostedNamespaceForHive(&h) + if ns == "" { + continue + } + + ctx, cancel := context.WithTimeout(context.Background(), contributeIngressKubectlTimeout) + raw, err := kubectlForClusterContext(ctx, cluster, "get", "ingress", contributeIngressName, + "-n", ns, "-o", "json").Output() + cancel() + if err != nil { + // Ingress missing (a spoke that predates hive-contribute entirely, + // or one mid-teardown), cluster unreachable, or a transient kubectl + // error — all non-fatal. Debug, and the next sweep retries. + s.logger.Debug("contribute ingress reconcile: could not read hive-contribute ingress", + "hive_id", h.ID, "cluster", cluster.ID, "namespace", ns, "error", err) + continue + } + s.markClusterReachable(cluster.ID) + + patch, perr := contributeIngressAnnotationPatch(raw, contributeIngressAuthAnnotations(hubURL, h.ID)) + if perr != nil { + failures++ + s.logger.Warn("contribute ingress reconcile: could not parse hive-contribute ingress — not patching blind", + "hive_id", h.ID, "cluster", cluster.ID, "namespace", ns, "error", perr) + continue + } + if patch == "" { + converged++ + s.logger.Debug("contribute ingress reconcile: hive-contribute already carries the auth-url", + "hive_id", h.ID, "cluster", cluster.ID) + continue + } + + pctx, pcancel := context.WithTimeout(context.Background(), contributeIngressKubectlTimeout) + pout, perr := kubectlForClusterContext(pctx, cluster, "patch", "ingress", contributeIngressName, + "-n", ns, "--type", "merge", "-p", patch).CombinedOutput() + pcancel() + if perr != nil { + failures++ + s.markClusterUnreachable(cluster.ID) + s.logger.Warn("contribute ingress reconcile: patch failed — will retry next sweep", + "hive_id", h.ID, "cluster", cluster.ID, "namespace", ns, + "output", strings.TrimSpace(string(pout)), "error", perr) + continue + } + s.markClusterReachable(cluster.ID) + patched++ + s.logger.Info("reconciled auth-url onto hive-contribute ingress (#7517)", + "hive_id", h.ID, "cluster", cluster.ID, "namespace", ns) + } + + // A sweep that admitted no hives at all on a hub that hosts spokes is a + // bug signal, not a quiet no-op — the condition that made the NET_ADMIN + // lane dead code for its whole production life. + if considered == 0 && len(hives) > 0 { + s.logger.Warn("contribute ingress reconcile: selected NO hives — sweep is a no-op, /api/contribute/me stays 401 on drifted spokes", + "hives_in_registry", len(hives), "skipped_by_status", skippedByStatus, + "skipped_no_nginx_ingress", skippedNoIngress) + return + } + // The per-sweep summary is Info only when the sweep changed or failed + // something, so a converged fleet stays quiet. + if patched > 0 || failures > 0 { + s.logger.Info("contribute ingress reconcile sweep complete", + "considered", considered, "converged", converged, "patched", patched, + "failures", failures, "skipped_by_status", skippedByStatus, + "skipped_no_nginx_ingress", skippedNoIngress) + } +} diff --git a/src/pkg/hub/contribute_ingress_reconcile_test.go b/src/pkg/hub/contribute_ingress_reconcile_test.go new file mode 100644 index 0000000000..bb70656e14 --- /dev/null +++ b/src/pkg/hub/contribute_ingress_reconcile_test.go @@ -0,0 +1,273 @@ +package hub + +import ( + "encoding/json" + "os" + "path/filepath" + "strings" + "testing" + "time" + + "gopkg.in/yaml.v3" +) + +// hivecommons/hive#7517: #7457 put auth-url / auth-response-headers on the +// hive-contribute Ingress in the provisioning template, but the template is +// applied only at provision time, so every hosted spoke that already existed +// kept an Ingress with no auth-url and /api/contribute/me kept answering 401. +// These tests pin the reconcile that re-applies the two annotations: what +// "converged" means (the template's own values), what the sweep patches (only +// the drifted keys, by merge patch), what it leaves alone, and that the +// poller actually runs it. + +// The reconcile's idea of the annotations MUST be the template's: if they +// ever disagree, a spoke the sweep has "converged" is still not what a freshly +// provisioned one looks like, and the 401 comes back on a hub URL change or a +// header rename nobody remembered to mirror. Render the template with the same +// two inputs and compare byte for byte. +func TestContributeIngressReconcileMatchesTheTemplate(t *testing.T) { + for _, useWildcard := range []bool{false, true} { + blocks := ingressBlocks(t, renderManifestWildcard(t, useWildcard)) + raw, ok := blocks[contributeIngressName] + if !ok { + t.Fatalf("useWildcard=%v: no %s Ingress in the template", useWildcard, contributeIngressName) + } + var doc struct { + Metadata struct { + Annotations map[string]string `yaml:"annotations"` + } `yaml:"metadata"` + } + if err := yaml.Unmarshal([]byte(raw), &doc); err != nil { + t.Fatalf("%s does not parse: %v\n%s", contributeIngressName, err, raw) + } + // renderManifestWildcard renders ID hosted-hive-x against + // https://hive.hivecommons.dev. + want := contributeIngressAuthAnnotations("https://hive.hivecommons.dev", "hosted-hive-x") + for key, value := range want { + if got := doc.Metadata.Annotations[key]; got != value { + t.Errorf("useWildcard=%v: reconcile expects %s = %q but the template renders %q — a converged spoke would not match a provisioned one", + useWildcard, key, value, got) + } + } + } +} + +// liveIngressJSON is a hive-contribute Ingress as `kubectl get -o json` prints +// it, with whatever annotations a test wants on it. The rest of the object is +// the shape provisionHive leaves behind, so the parse is exercised on a real +// document rather than a bare annotations map. +func liveIngressJSON(t *testing.T, annotations map[string]string) []byte { + t.Helper() + obj := map[string]any{ + "apiVersion": "networking.k8s.io/v1", + "kind": "Ingress", + "metadata": map[string]any{ + "name": contributeIngressName, + "namespace": "hive-hosted-hosted-projectbluefin-common-nmq5", + "annotations": annotations, + }, + "spec": map[string]any{ + "ingressClassName": "nginx", + "rules": []any{map[string]any{ + "host": "hosted-projectbluefin-common-nmq5.hive.hivecommons.dev", + }}, + }, + } + raw, err := json.Marshal(obj) + if err != nil { + t.Fatal(err) + } + return raw +} + +func decodePatch(t *testing.T, patch string) map[string]string { + t.Helper() + var body struct { + Metadata struct { + Annotations map[string]string `json:"annotations"` + } `json:"metadata"` + } + if err := json.Unmarshal([]byte(patch), &body); err != nil { + t.Fatalf("patch %q is not a merge patch on metadata.annotations: %v", patch, err) + } + return body.Metadata.Annotations +} + +// The issue's spoke: provisioned before #7457, so the Ingress carries the +// cert-manager issuer and the proxy timeouts but no auth-url. Both annotations +// go on, and ONLY those two — the merge patch must not touch what is there. +func TestContributeIngressPatchAddsTheMissingAnnotations(t *testing.T) { + want := contributeIngressAuthAnnotations("https://hive.hivecommons.dev", "hosted-projectbluefin-common-nmq5") + pre7457 := map[string]string{ + "cert-manager.io/cluster-issuer": "letsencrypt-dns01", + "nginx.ingress.kubernetes.io/proxy-read-timeout": "3600", + "nginx.ingress.kubernetes.io/proxy-send-timeout": "3600", + } + patch, err := contributeIngressAnnotationPatch(liveIngressJSON(t, pre7457), want) + if err != nil { + t.Fatal(err) + } + if patch == "" { + t.Fatal("a pre-#7457 Ingress with no auth-url was judged converged — this is the production bug: /api/contribute/me stays 401") + } + got := decodePatch(t, patch) + if len(got) != 2 { + t.Errorf("patch touches %d annotations, want exactly the two #7457 added: %v", len(got), got) + } + if got[ingressAuthURLAnnotation] != "https://hive.hivecommons.dev/api/saas/auth-check?hive=hosted-projectbluefin-common-nmq5&uri=$request_uri" { + t.Errorf("auth-url = %q", got[ingressAuthURLAnnotation]) + } + if got[ingressAuthResponseHeadersAnnotation] != "X-Hive-User,X-Hive-Role,X-Hive-Proxy-Auth" { + t.Errorf("auth-response-headers = %q", got[ingressAuthResponseHeadersAnnotation]) + } + for key := range pre7457 { + if _, touched := got[key]; touched { + t.Errorf("patch rewrites %s, which was not drifted", key) + } + } + // No auth-signin: /api/contribute is public and fetch() could not follow + // the redirect anyway (#7457). A reconcile that added one would turn an + // anonymous leaderboard load into a cross-origin bounce. + if _, has := got["nginx.ingress.kubernetes.io/auth-signin"]; has { + t.Error("patch adds auth-signin to a public path") + } + // The value nginx needs is the literal $request_uri, not a shell- or + // template-expanded one. + if !strings.Contains(patch, `$request_uri`) { + t.Errorf("patch lost the literal $request_uri: %s", patch) + } +} + +// A spoke provisioned after #7457 (or one this sweep already fixed) is a +// no-op: no patch body, so the sweep never issues a pointless kubectl patch +// and can run every 15 minutes forever without touching a converged fleet. +func TestContributeIngressPatchIsEmptyWhenConverged(t *testing.T) { + want := contributeIngressAuthAnnotations("https://hive.hivecommons.dev", "hosted-hive-x") + live := map[string]string{ + "cert-manager.io/cluster-issuer": "letsencrypt-dns01", + "nginx.ingress.kubernetes.io/proxy-read-timeout": "3600", + } + for key, value := range want { + live[key] = value + } + patch, err := contributeIngressAnnotationPatch(liveIngressJSON(t, live), want) + if err != nil { + t.Fatal(err) + } + if patch != "" { + t.Errorf("a converged Ingress produced a patch: %s", patch) + } +} + +// Only the drifted key goes in the patch. A spoke whose auth-url points at a +// previous hub URL (or a hand-edited one) is corrected; the headers it already +// carries correctly are left out of the body. +func TestContributeIngressPatchReplacesOnlyTheStaleValue(t *testing.T) { + want := contributeIngressAuthAnnotations("https://hive.hivecommons.dev", "hosted-hive-x") + live := map[string]string{ + ingressAuthURLAnnotation: "https://hive.kubestellar.io/api/saas/auth-check?hive=hosted-hive-x&uri=$request_uri", + ingressAuthResponseHeadersAnnotation: ingressAuthResponseHeaders, + } + patch, err := contributeIngressAnnotationPatch(liveIngressJSON(t, live), want) + if err != nil { + t.Fatal(err) + } + got := decodePatch(t, patch) + if len(got) != 1 || got[ingressAuthURLAnnotation] != want[ingressAuthURLAnnotation] { + t.Errorf("patch = %v, want only the corrected auth-url", got) + } +} + +// An Ingress with no annotations at all (kubectl prints no `annotations` key) +// still gets both — the absent map must read as "everything missing", not as +// a parse failure. +func TestContributeIngressPatchHandlesNoAnnotationsAtAll(t *testing.T) { + want := contributeIngressAuthAnnotations("https://hive.hivecommons.dev", "hosted-hive-x") + patch, err := contributeIngressAnnotationPatch([]byte(`{"metadata":{"name":"hive-contribute"}}`), want) + if err != nil { + t.Fatal(err) + } + if got := decodePatch(t, patch); len(got) != 2 { + t.Errorf("patch = %v, want both annotations", got) + } +} + +// Unparseable output is an error, never a patch: the sweep must not patch +// blind onto an object it could not read. +func TestContributeIngressPatchRefusesToPatchBlind(t *testing.T) { + want := contributeIngressAuthAnnotations("https://hive.hivecommons.dev", "hosted-hive-x") + for _, raw := range []string{"", "not json", "Error from server (NotFound): ingresses.networking.k8s.io \"hive-contribute\" not found"} { + patch, err := contributeIngressAnnotationPatch([]byte(raw), want) + if err == nil || patch != "" { + t.Errorf("raw %q: patch=%q err=%v, want an error and no patch", raw, patch, err) + } + } +} + +// The values are per hive and per hub: two hives never share an auth-url +// (the hub scopes the auth-check by ?hive=), and the hub URL is whatever +// hubPublicURL() says at sweep time, not a constant. +func TestContributeIngressAuthAnnotationsArePerHive(t *testing.T) { + a := contributeIngressAuthAnnotations("https://hub.example", "hive-a") + b := contributeIngressAuthAnnotations("https://hub.example", "hive-b") + if a[ingressAuthURLAnnotation] == b[ingressAuthURLAnnotation] { + t.Error("two hives share an auth-url; the hub could not tell whose grant to check") + } + if !strings.HasPrefix(a[ingressAuthURLAnnotation], "https://hub.example/api/saas/auth-check?hive=hive-a&") { + t.Errorf("auth-url = %q", a[ingressAuthURLAnnotation]) + } +} + +// Same predicate shape as netAdminSweepEligible, for the same reason: the +// steady-state fleet carries "" and "available", not "running", and a +// placeholder must be right before it is claimed. +func TestContributeIngressSweepEligible(t *testing.T) { + for _, status := range []string{"", "available", "assigned", "running", "error", " available "} { + if !contributeIngressSweepEligible(status) { + t.Errorf("status %q is not swept, but live spokes carry it", status) + } + } + if contributeIngressSweepEligible("provisioning") { + t.Error("a hive still provisioning was selected; its Ingress is being applied from the template right now") + } +} + +// The poller-loop throttle lets the sweep run once per interval. Driven on +// the timestamp directly, as the NET_ADMIN throttle test is, rather than +// running the kubectl-shelling body. +func TestContributeIngressReconcileThrottle(t *testing.T) { + s := &HubServer{clusterUnreachableUntil: map[string]time.Time{}} + due := func() bool { + s.clusterUnreachableMu.Lock() + defer s.clusterUnreachableMu.Unlock() + d := s.lastContributeIngressReconcile.IsZero() || + time.Since(s.lastContributeIngressReconcile) >= contributeIngressReconcileInterval + if d { + s.lastContributeIngressReconcile = time.Now() + } + return d + } + if !due() { + t.Fatal("first call must be due") + } + if due() { + t.Fatal("second call inside the interval must not be due") + } + s.lastContributeIngressReconcile = time.Now().Add(-contributeIngressReconcileInterval - time.Second) + if !due() { + t.Fatal("a call after the interval must be due again") + } +} + +// The sweep exists only if the poller calls it. A reconcile that is written, +// tested and never wired is the NET_ADMIN story again (#2674), so pin the +// call site in saas_sha_poller.go next to its siblings. +func TestContributeIngressReconcileIsWiredIntoThePoller(t *testing.T) { + src, err := os.ReadFile(filepath.Join("saas_sha_poller.go")) + if err != nil { + t.Fatal(err) + } + if !strings.Contains(string(src), "s.reconcileContributeIngressIfDue()") { + t.Fatal("saas_sha_poller.go never calls reconcileContributeIngressIfDue(); the hive-contribute auth-url drift is never repaired") + } +} diff --git a/src/pkg/hub/saas_sha_poller.go b/src/pkg/hub/saas_sha_poller.go index 92081a34db..9330194c77 100644 --- a/src/pkg/hub/saas_sha_poller.go +++ b/src/pkg/hub/saas_sha_poller.go @@ -700,6 +700,14 @@ func (s *HubServer) StartLatestSHAPoller(ctx context.Context) { // rate-limited to perHiveEnvMaxPatchesPerCycle patches per cycle, because // each patch rolls that hive's pod. See perhive_env_reconcile.go. s.reconcilePerHiveEnvIfDue() + // Put #7457's auth-url / auth-response-headers onto the hive-contribute + // Ingress of every hosted spoke provisioned before that fix, so + // /api/contribute/me learns who is calling there too (#7517). The + // template is applied only at provision time; nothing else re-applies an + // Ingress annotation. Throttled internally to + // contributeIngressReconcileInterval; an annotation patch rolls no pod. + // See contribute_ingress_reconcile.go. + s.reconcileContributeIngressIfDue() // Force-delete hive-namespace pods stuck in Terminating past // orphanedPodMinAge with no finalizers and a non-Running phase — the // residue of nodes disappearing without draining (#5328). Throttled diff --git a/src/pkg/hub/server.go b/src/pkg/hub/server.go index 3100ae5c7b..3569c2b978 100644 --- a/src/pkg/hub/server.go +++ b/src/pkg/hub/server.go @@ -1076,6 +1076,11 @@ type HubServer struct { // Same rationale and same guarding mutex as lastNetAdminReconcile above: // both are poller-loop-only state. lastPerHiveEnvReconcile time.Time + // lastContributeIngressReconcile throttles the hive-contribute Ingress + // auth-url reconcile (contribute_ingress_reconcile.go), which puts the + // #7457 annotations onto spokes provisioned before that fix (#7517). + // Same rationale and same guarding mutex as the two above. + lastContributeIngressReconcile time.Time // lastOrphanedPodReap throttles the orphaned Terminating-pod reaper // (orphaned_pod_reaper.go), which force-deletes hive-namespace pods left // behind when a node disappears without draining (#5328). Same rationale From 436a8bd34dd6208bd57eb434402799b559f2f591 Mon Sep 17 00:00:00 2001 From: clubanderson Date: Thu, 17 Sep 2026 23:03:53 -0400 Subject: [PATCH 08/17] =?UTF-8?q?=F0=9F=93=96=20docs:=20refresh=20api-refe?= =?UTF-8?q?rence=20citations=20drifted=20by=20the=20auth-url=20reconcile?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: clubanderson --- src/docs/api-reference.md | 48 +++++++++++++++++++-------------------- 1 file changed, 24 insertions(+), 24 deletions(-) diff --git a/src/docs/api-reference.md b/src/docs/api-reference.md index f008e9058f..4157ef369c 100644 --- a/src/docs/api-reference.md +++ b/src/docs/api-reference.md @@ -535,28 +535,28 @@ always resolved server-side from the validated token. | `GET` | `/api/hub/clusters` | Hub auth | List Clusters | `pkg/hub/saas.go:503` | | `GET` | `/api/hub/image-pulls` | Hub auth | Per-Release Image Pull Series | `pkg/hub/saas.go:369` | | `GET` | `/api/reach` | Hub admin | PR Reach Report (?pr=NNN or ?recent=K) | `pkg/hub/saas.go:494` | -| `GET` | `/fleet` | Hub handler-specific | My-Hives Fleet Page (static; data via `/api/saas/my-hives`) | `pkg/hub/server.go:1587` | -| `GET` | `/my-hives` | Hub handler-specific | 301 redirect to `/fleet` (query preserved) | `pkg/hub/server.go:1588` | -| `POST` | `/api/heartbeat` | Hub handler-specific | Heartbeat | `pkg/hub/server.go:1549` | -| `POST` | `/api/task-status` | Hub handler-specific | Task Status | `pkg/hub/server.go:1550` | -| `GET` | `/api/registry` | Hub handler-specific | Registry | `pkg/hub/server.go:1551` | -| `GET` | `/api/hub/leaderboard` | Hub handler-specific | Leaderboard | `pkg/hub/server.go:1552` | -| `GET` | `/api/hub/stats` | Hub handler-specific | Stats | `pkg/hub/server.go:1553` | -| `GET` | `/api/fleet-stats` | Hub handler-specific | Fleet Stats | `pkg/hub/server.go:1554` | -| `GET` | `/api/hub/version` | Hub handler-specific | Hub Version | `pkg/hub/server.go:1555` | -| `DELETE` | `/api/hub/registry/{id}` | Hub handler-specific | Registry Delete | `pkg/hub/server.go:1565` | -| `POST` | `/api/contribute/register` | Hub handler-specific | Contribute Proxy | `pkg/hub/server.go:1566` | -| `GET` | `/api/contribute/status` | Hub handler-specific | Contribute Status | `pkg/hub/server.go:1567` | -| `GET` | `/api/contribute/ws` | Hub handler-specific | Contribute WSProxy | `pkg/hub/server.go:1568` | -| `POST` | `/api/github/webhook` | Hub handler-specific | GitHub Webhook | `pkg/hub/server.go:1569` | -| `GET` | `/gh-setup` | Hub handler-specific | GitHub App Setup Router | `pkg/hub/server.go:1570` | -| `GET` | `/learn` | Hub handler-specific | Static HTML page | `pkg/hub/server.go:1571` | -| `GET` | `/get-started` | Hub handler-specific | Static HTML page | `pkg/hub/server.go:1572` | -| `GET` | `/api/docs` | Hub handler-specific | Static HTML page | `pkg/hub/server.go:1573` | -| `GET` | `/api/reading-list` | Hub handler-specific | Reading List | `pkg/hub/server.go:1574` | -| `GET` | `/reading` | Hub handler-specific | Static HTML page | `pkg/hub/server.go:1575` | -| `GET` | `/cncf-reference-architecture` | Hub handler-specific | Static HTML page | `pkg/hub/server.go:1578` | -| `GET` | `/{$}` | Hub handler-specific | Static HTML page | `pkg/hub/server.go:1595` | -| `GET` | `/og-card.png` | Hub handler-specific | OGCard | `pkg/hub/server.go:1600` | +| `GET` | `/fleet` | Hub handler-specific | My-Hives Fleet Page (static; data via `/api/saas/my-hives`) | `pkg/hub/server.go:1592` | +| `GET` | `/my-hives` | Hub handler-specific | 301 redirect to `/fleet` (query preserved) | `pkg/hub/server.go:1593` | +| `POST` | `/api/heartbeat` | Hub handler-specific | Heartbeat | `pkg/hub/server.go:1554` | +| `POST` | `/api/task-status` | Hub handler-specific | Task Status | `pkg/hub/server.go:1555` | +| `GET` | `/api/registry` | Hub handler-specific | Registry | `pkg/hub/server.go:1556` | +| `GET` | `/api/hub/leaderboard` | Hub handler-specific | Leaderboard | `pkg/hub/server.go:1557` | +| `GET` | `/api/hub/stats` | Hub handler-specific | Stats | `pkg/hub/server.go:1558` | +| `GET` | `/api/fleet-stats` | Hub handler-specific | Fleet Stats | `pkg/hub/server.go:1559` | +| `GET` | `/api/hub/version` | Hub handler-specific | Hub Version | `pkg/hub/server.go:1560` | +| `DELETE` | `/api/hub/registry/{id}` | Hub handler-specific | Registry Delete | `pkg/hub/server.go:1570` | +| `POST` | `/api/contribute/register` | Hub handler-specific | Contribute Proxy | `pkg/hub/server.go:1571` | +| `GET` | `/api/contribute/status` | Hub handler-specific | Contribute Status | `pkg/hub/server.go:1572` | +| `GET` | `/api/contribute/ws` | Hub handler-specific | Contribute WSProxy | `pkg/hub/server.go:1573` | +| `POST` | `/api/github/webhook` | Hub handler-specific | GitHub Webhook | `pkg/hub/server.go:1574` | +| `GET` | `/gh-setup` | Hub handler-specific | GitHub App Setup Router | `pkg/hub/server.go:1575` | +| `GET` | `/learn` | Hub handler-specific | Static HTML page | `pkg/hub/server.go:1576` | +| `GET` | `/get-started` | Hub handler-specific | Static HTML page | `pkg/hub/server.go:1577` | +| `GET` | `/api/docs` | Hub handler-specific | Static HTML page | `pkg/hub/server.go:1578` | +| `GET` | `/api/reading-list` | Hub handler-specific | Reading List | `pkg/hub/server.go:1579` | +| `GET` | `/reading` | Hub handler-specific | Static HTML page | `pkg/hub/server.go:1580` | +| `GET` | `/cncf-reference-architecture` | Hub handler-specific | Static HTML page | `pkg/hub/server.go:1583` | +| `GET` | `/{$}` | Hub handler-specific | Static HTML page | `pkg/hub/server.go:1600` | +| `GET` | `/og-card.png` | Hub handler-specific | OGCard | `pkg/hub/server.go:1605` | | `GET` | `/api/hub/delegation-keys` | Public | Delegation-chain verification material (Ed25519 public keys + generation numbers, JWKS-equivalent); deliberately unauthenticated so a tenant or their auditor can verify a chain without a hive credential - see [delegation chain](delegation-chain.md) | `pkg/hub/server.go:1564` | -| `GET` | `/` | Public | Static asset fallback (`http.FileServerFS` over the embedded `static/` tree) for any path no other route claims | `pkg/hub/server.go:1601` | +| `GET` | `/` | Public | Static asset fallback (`http.FileServerFS` over the embedded `static/` tree) for any path no other route claims | `pkg/hub/server.go:1606` | From 8cba385ec79248b6438b2b13ac7a3ff427ba3441 Mon Sep 17 00:00:00 2001 From: clubanderson Date: Thu, 17 Sep 2026 23:04:31 -0400 Subject: [PATCH 09/17] =?UTF-8?q?=F0=9F=93=96=20docs:=20re-point=20delegat?= =?UTF-8?q?ion-keys=20citation=20at=20its=20HandleFunc=20line?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: clubanderson --- src/docs/api-reference.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/docs/api-reference.md b/src/docs/api-reference.md index 4157ef369c..37513a7cdd 100644 --- a/src/docs/api-reference.md +++ b/src/docs/api-reference.md @@ -558,5 +558,5 @@ always resolved server-side from the validated token. | `GET` | `/cncf-reference-architecture` | Hub handler-specific | Static HTML page | `pkg/hub/server.go:1583` | | `GET` | `/{$}` | Hub handler-specific | Static HTML page | `pkg/hub/server.go:1600` | | `GET` | `/og-card.png` | Hub handler-specific | OGCard | `pkg/hub/server.go:1605` | -| `GET` | `/api/hub/delegation-keys` | Public | Delegation-chain verification material (Ed25519 public keys + generation numbers, JWKS-equivalent); deliberately unauthenticated so a tenant or their auditor can verify a chain without a hive credential - see [delegation chain](delegation-chain.md) | `pkg/hub/server.go:1564` | +| `GET` | `/api/hub/delegation-keys` | Public | Delegation-chain verification material (Ed25519 public keys + generation numbers, JWKS-equivalent); deliberately unauthenticated so a tenant or their auditor can verify a chain without a hive credential - see [delegation chain](delegation-chain.md) | `pkg/hub/server.go:1569` | | `GET` | `/` | Public | Static asset fallback (`http.FileServerFS` over the embedded `static/` tree) for any path no other route claims | `pkg/hub/server.go:1606` | From 180ddadf988e46f17ae138edfe528db562122b63 Mon Sep 17 00:00:00 2001 From: Andrew Anderson Date: Thu, 17 Sep 2026 22:59:27 -0400 Subject: [PATCH 10/17] =?UTF-8?q?=F0=9F=8C=B1=20dashboard:=20name=20the=20?= =?UTF-8?q?branch-protection=20rule=20behind=20a=20blocked=20PR=20pill?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit GitHub folds every unsatisfied branch-protection rule into the single word "blocked", so a PR pill that hovered as "blocked — all sweep gates pass; a branch-protection rule is unsatisfied" (the step-1 placeholder from #7516) still sent the operator to GitHub to learn which rule — the trip the pill exists to save. The sweep now collects the two facts GitHub's REST PR payload does not carry, both at enumeration time and neither per PR: - GitHub's own reviewDecision plus each reviewer's current position, from ONE GraphQL query per repository per pass (paginated 100 PRs at a time); - the base branch's required status-check set — from the operator's auto_merge.required_checks when installed, which costs no API call at all — compared against the check runs EnrichCIStatus already walks. github.PullRequest.BranchProtectionBlockReason is the one place that turns those facts into words. It is deliberately conservative: it returns ok=false whenever nothing on hand identifies a rule, and it refuses to claim a required check "has not reported" on a repository where no required context was seen as a check run at all, because there the context is a commit status and every one of them would look absent. The dashboard needs no frontend change: prMergeNote already renders a blocked verdict's reason verbatim. Refs #7515 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Andrew Anderson --- ...hanged-7515-blocked-pill-names-the-rule.md | 1 + src/cmd/hive/main.go | 22 +- src/cmd/hive/merge_verdict_rule_7515_test.go | 170 ++++++++++ .../pr_pill_merge_verdict_7478_test.go | 22 ++ src/pkg/github/client.go | 150 +++++---- src/pkg/github/merge_block_reason.go | 163 ++++++++++ .../github/merge_block_reason_7515_test.go | 215 +++++++++++++ src/pkg/github/protection_facts.go | 216 +++++++++++++ src/pkg/github/protection_facts_7515_test.go | 292 ++++++++++++++++++ 9 files changed, 1189 insertions(+), 62 deletions(-) create mode 100644 changelog.d/changed-7515-blocked-pill-names-the-rule.md create mode 100644 src/cmd/hive/merge_verdict_rule_7515_test.go create mode 100644 src/pkg/github/merge_block_reason.go create mode 100644 src/pkg/github/merge_block_reason_7515_test.go create mode 100644 src/pkg/github/protection_facts.go create mode 100644 src/pkg/github/protection_facts_7515_test.go diff --git a/changelog.d/changed-7515-blocked-pill-names-the-rule.md b/changelog.d/changed-7515-blocked-pill-names-the-rule.md new file mode 100644 index 0000000000..5da4836afe --- /dev/null +++ b/changelog.d/changed-7515-blocked-pill-names-the-rule.md @@ -0,0 +1 @@ +- Hovering a blocked PR pill on the dashboard now names the branch-protection rule that is holding the PR, instead of GitHub's one-word `blocked` state ([#7515](https://github.com/hivecommons/hive/issues/7515)). GitHub folds every unsatisfied rule — a red required check, a required check that never ran, a missing approval, a reviewer who asked for changes — into that single word, so an operator had to open the PR on GitHub to learn which one, the trip the pill exists to save. The sweep now collects GitHub's own review decision (one GraphQL query per repository per cycle, not one per PR) and compares the base branch's required-check set against the check runs it already walks, and the tooltip reads `blocked — changes requested by @reviewer`, `blocked — required check "validate" has not reported`, or `blocked — required check "build" is failing`. When the facts on hand genuinely do not identify a rule the tooltip still says so plainly rather than guessing, and on a repository that reports its required contexts as commit statuses no "has not reported" claim is made at all. diff --git a/src/cmd/hive/main.go b/src/cmd/hive/main.go index dc62cd62d2..87b0ba6530 100644 --- a/src/cmd/hive/main.go +++ b/src/cmd/hive/main.go @@ -9226,8 +9226,10 @@ const mergeabilityUnknownReason = "mergeability not yet computed by GitHub — r // // - "blocked" folds every unsatisfied branch-protection rule into one // word. When the sweep has a reason it is almost always the rule -// ("blocked — CI failing: build"); without one, say that a rule we do -// not read is unsatisfied rather than nothing at all. +// ("blocked — CI failing: build"); when GitHub's own facts name a +// different rule (a review decision, a required check that never +// reported) that rule is named too; with neither, say that a rule we +// cannot read is unsatisfied rather than nothing at all. // - "dirty" and "behind" name the base branch and the fix (rebase / // update); a sweep reason is appended, since it still stands once the // branch is fixed. @@ -9240,8 +9242,22 @@ func notMergeableReason(pr github.PullRequest, sweepReason string) string { var msg string switch pr.MergeableState { case "blocked": - if sweepReason == "" { + // The branch-protection rule GitHub is hiding behind the word + // "blocked", when the sweep collected enough to name it + // (hivecommons/hive#7515 step 2). The wording itself lives in + // github.PullRequest.BranchProtectionBlockReason — one place, under + // test — not in this switch and not in the dashboard's JS. + rule, ruleKnown := pr.BranchProtectionBlockReason() + switch { + case sweepReason == "" && ruleKnown: + return "blocked — " + rule + case sweepReason == "": return "blocked — all sweep gates pass; a branch-protection rule is unsatisfied" + case ruleKnown && !strings.Contains(sweepReason, rule): + // Both are true and neither subsumes the other: the sweep's own + // gate is what it will act on, and GitHub's rule is what the + // operator must also clear. + return "blocked — " + sweepReason + "; GitHub also requires: " + rule } return "blocked — " + sweepReason case "dirty": diff --git a/src/cmd/hive/merge_verdict_rule_7515_test.go b/src/cmd/hive/merge_verdict_rule_7515_test.go new file mode 100644 index 0000000000..271c9aad16 --- /dev/null +++ b/src/cmd/hive/merge_verdict_rule_7515_test.go @@ -0,0 +1,170 @@ +package main + +import ( + "testing" + + "github.com/hivecommons/hive/pkg/github" +) + +// hivecommons/hive#7515 step 2: when GitHub says "blocked", the tooltip must +// name the branch-protection rule that is unsatisfied, not the word +// "blocked". Step 1 (#7516) kept the SWEEP's own gate when it had one; these +// cases pin what happens with GitHub's own facts — a review decision, a red +// or absent required check — which the sweep now collects during enumeration. +// +// The wording itself lives in github.PullRequest.BranchProtectionBlockReason +// (pinned by TestBranchProtectionBlockReason); what is pinned here is that +// the classifier reaches for it, and that it still declines to guess. +func TestClassifyMergeEligibility_BlockedNamesTheProtectionRule(t *testing.T) { + blocked := func(p *github.ProtectionFacts) github.PullRequest { + return github.PullRequest{ + Number: 1, + Mergeable: github.MergeableNo, + MergeableState: "blocked", + BaseRef: "v4", + CIStatus: "success", + Protection: p, + } + } + + cases := []struct { + name string + pr github.PullRequest + gates mergeGates + wantReason string // exact + }{ + { + name: "changes requested is named where the placeholder used to be", + pr: blocked(&github.ProtectionFacts{ + ReviewDecision: github.ReviewDecisionChangesRequested, + ChangesRequestedBy: []string{"reviewer"}, + }), + wantReason: "blocked — changes requested by @reviewer", + }, + { + name: "a required check that never reported is named", + pr: blocked(&github.ProtectionFacts{ + RequiredChecksKnown: true, + MissingRequiredChecks: []string{"validate"}, + }), + wantReason: `blocked — required check "validate" has not reported`, + }, + { + name: "a review GitHub requires is named", + pr: blocked(&github.ProtectionFacts{ + ReviewDecision: github.ReviewDecisionReviewRequired, + }), + wantReason: "blocked — an approving review is required by branch protection (0 given)", + }, + { + // NEGATIVE CONTROL. Facts were gathered and they explain + // nothing: the honest placeholder must survive. A build that + // always prints a derived rule fails here. + name: "no rule derivable keeps the honest placeholder", + pr: blocked(&github.ProtectionFacts{ + RequiredChecksKnown: true, + ReviewDecision: github.ReviewDecisionApproved, + }), + wantReason: "blocked — all sweep gates pass; a branch-protection rule is unsatisfied", + }, + { + // NEGATIVE CONTROL. No facts at all (an older governor, a repo + // whose GraphQL query failed) must read exactly as it did after + // step 1 — this is the #7516 regression guard. + name: "no facts at all keeps the step-1 placeholder", + pr: blocked(nil), + wantReason: "blocked — all sweep gates pass; a branch-protection rule is unsatisfied", + }, + } + + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + bucket, verdict, _ := classifyMergeEligibility(tc.pr, false, "org/repo", tc.gates) + if bucket != mergeBucketSkip { + t.Errorf("bucket = %v, want %v", bucket, mergeBucketSkip) + } + if verdict.State != github.MergeVerdictBlocked { + t.Errorf("state = %q, want %q", verdict.State, github.MergeVerdictBlocked) + } + if verdict.Reason != tc.wantReason { + t.Errorf("reason = %q\n want %q", verdict.Reason, tc.wantReason) + } + }) + } +} + +// When the sweep has its own gate AND GitHub names a different rule, the +// operator needs both: clearing the sweep's gate alone will not merge the PR. +// When the two say the same thing, it must not be said twice. +func TestClassifyMergeEligibility_BlockedKeepsBothReasons(t *testing.T) { + base := func(p *github.ProtectionFacts) github.PullRequest { + return github.PullRequest{ + Number: 2, + Mergeable: github.MergeableNo, + MergeableState: "blocked", + BaseRef: "v4", + CIStatus: "failure", + FailingChecks: []string{"build"}, + Protection: p, + } + } + gates := mergeGates{requiredChecks: map[string]bool{"build": true}} + + t.Run("different rules are both reported", func(t *testing.T) { + pr := base(&github.ProtectionFacts{ + ReviewDecision: github.ReviewDecisionChangesRequested, + ChangesRequestedBy: []string{"reviewer"}, + }) + _, verdict, _ := classifyMergeEligibility(pr, false, "org/repo", gates) + want := "blocked — CI failing: build; GitHub also requires: changes requested by @reviewer" + if verdict.Reason != want { + t.Errorf("reason = %q\n want %q", verdict.Reason, want) + } + }) + + t.Run("the same rule is not said twice", func(t *testing.T) { + // The sweep's reason already contains the derived rule verbatim. + pr := github.PullRequest{ + Number: 3, + Mergeable: github.MergeableNo, + MergeableState: "blocked", + BaseRef: "v4", + CIStatus: "success", + Protection: &github.ProtectionFacts{ + ReviewDecision: github.ReviewDecisionChangesRequested, + ChangesRequestedBy: []string{"reviewer"}, + }, + } + // Feed the intent gate a reason that literally contains the rule. + _, verdict, _ := classifyMergeEligibility(pr, true, "org/repo", mergeGates{}) + if verdict.Reason != "blocked — held: a hold label keeps it out of the sweep; GitHub also requires: changes requested by @reviewer" { + t.Errorf("held+changes-requested reason = %q", verdict.Reason) + } + }) +} + +// A state OTHER than "blocked" must not sprout a branch-protection rule: +// "dirty" is a conflict and "behind" is a stale branch, and neither is a +// protection rule the operator can clear by getting a review. +func TestClassifyMergeEligibility_ProtectionRuleOnlyAppliesToBlocked(t *testing.T) { + facts := &github.ProtectionFacts{ + ReviewDecision: github.ReviewDecisionChangesRequested, + ChangesRequestedBy: []string{"reviewer"}, + } + for _, tc := range []struct { + state string + want string + }{ + {"dirty", "has merge conflicts with v4 — needs a rebase"}, + {"behind", "behind v4 — needs an update from the base branch"}, + } { + pr := github.PullRequest{ + Number: 4, Mergeable: github.MergeableNo, MergeableState: tc.state, + BaseRef: "v4", CIStatus: "success", Protection: facts, + } + _, verdict, _ := classifyMergeEligibility(pr, false, "org/repo", mergeGates{}) + if verdict.Reason != tc.want { + t.Errorf("%s reason = %q\n want %q", tc.state, verdict.Reason, tc.want) + } + } +} diff --git a/src/pkg/dashboard/pr_pill_merge_verdict_7478_test.go b/src/pkg/dashboard/pr_pill_merge_verdict_7478_test.go index 567b50bbd4..b6a80f238c 100644 --- a/src/pkg/dashboard/pr_pill_merge_verdict_7478_test.go +++ b/src/pkg/dashboard/pr_pill_merge_verdict_7478_test.go @@ -199,6 +199,28 @@ const blockedReview = { mergeable: 'no', mergeable_state: 'blocked', merge_verdi check('blocked tooltip names the missing review', prMergeNote(blockedReview).includes('awaiting review approval')); check('draft tooltip says what to do', prMergeNote({ mergeable: 'yes', merge_verdict: { state: 'blocked', reason: 'draft — mark ready for review to enter the sweep' } }).includes('mark ready for review')); check('unknown tooltip says not yet computed', prMergeNote({ mergeable: '', merge_verdict: { state: 'unknown', reason: 'mergeability not yet computed by GitHub — re-checked next tick; CI pending' } }).includes('not yet computed')); +// hivecommons/hive#7515 step 2: when no sweep gate explains a "blocked" PR, +// the verdict now names the branch-protection rule GitHub was hiding. The +// tooltip must show that rule verbatim — the mapping is Go's, and the JS +// must not re-word, truncate or re-derive any of it. +const rules = [ + 'blocked — changes requested by @reviewer', + 'blocked — required check "validate" has not reported', + 'blocked — required checks "build", "lint" are failing', + 'blocked — an approving review is required by branch protection (0 given)', + 'blocked — CI failing: build; GitHub also requires: changes requested by @reviewer', +]; +rules.forEach(reason => { + const p = { mergeable: 'no', mergeable_state: 'blocked', merge_verdict: { state: 'blocked', reason } }; + check('tooltip shows the derived rule verbatim: ' + reason, prMergeNote(p).includes(reason)); + check('no raw GitHub state appended to: ' + reason, !prMergeNote(p).includes('GitHub state')); + check('a blocked rule never reads as eligible: ' + reason, !prMergeNote(p).includes('eligible')); +}); +// Negative control: the placeholder is still shown when the governor could +// NOT name a rule. A frontend that invented one would fail here. +const unnamed = { mergeable: 'no', mergeable_state: 'blocked', merge_verdict: { state: 'blocked', reason: 'blocked — all sweep gates pass; a branch-protection rule is unsatisfied' } }; +check('an underivable rule still says so honestly', prMergeNote(unnamed).includes('a branch-protection rule is unsatisfied')); +check('an underivable rule does not name a check', !prMergeNote(unnamed).includes('required check')); check('no undefined anywhere', ![green, bluefin1253, dirty, blocked, { mergeable: 'yes' }, {}, { merge_verdict: {} }].some(p => prMergeNote(p).includes('undefined'))); if (fails) { console.log(fails + ' check(s) failed'); process.exit(1); } diff --git a/src/pkg/github/client.go b/src/pkg/github/client.go index 6305e7327b..2c91f5ff93 100644 --- a/src/pkg/github/client.go +++ b/src/pkg/github/client.go @@ -374,6 +374,12 @@ type PullRequest struct { // src/docs/review-queue-triage.md (#6183). Derived from Title + Labels // at enumeration time; never read by the governor or any agent. ReviewClass ReviewClass `json:"review_class,omitempty"` + // Protection carries the branch-protection facts behind GitHub's + // one-word "blocked" state — which required checks are red or absent, + // and GitHub's own review decision. It is nil when none of it could be + // determined, and BranchProtectionBlockReason then declines to guess + // (hivecommons/hive#7515). Display only: no merge gate reads it. + Protection *ProtectionFacts `json:"protection,omitempty"` } // HasFailingRequiredCheck reports whether this PR has a completed, non-meta @@ -933,81 +939,107 @@ func (c *Client) fetchPRs(ctx context.Context, repo string) (actionable []PullRe // EnrichCIStatus fetches check-run results for each PR's HEAD commit // and sets the CIStatus field to "success", "failure", or "pending". // The "tide" check is skipped — it reports Prow merge-bot state, not CI. +// +// It also collects the branch-protection facts behind a "blocked" PR +// (hivecommons/hive#7515, step 2) — which required checks are red or have +// never reported, and GitHub's own review decision — so that the dashboard +// pill can name the unsatisfied rule instead of repeating the word +// "blocked". Those facts are gathered from data this pass already walks +// plus at most one GraphQL query per repository; nothing here is per-hover +// and nothing here is per-PR beyond the calls that were already made. func (c *Client) EnrichCIStatus(ctx context.Context, prs []PullRequest) { if c == nil { return } + facts := newProtectionCollector(c) + for i := range prs { + reported := c.enrichPRCI(ctx, &prs[i]) + facts.attach(ctx, &prs[i], reported) + } +} + +// enrichPRCI is EnrichCIStatus's per-PR body. It returns the names of every +// check run observed on the PR's head SHA — meta checks included, because a +// required status-check context may well be one of them — or nil when no +// check-run listing was obtained at all. The caller distinguishes the two: +// "no listing" must never be read as "the required check never reported". +func (c *Client) enrichPRCI(ctx context.Context, pr *PullRequest) map[string]bool { const ciStatusSuccess = "success" const ciStatusFailure = "failure" const ciStatusPending = "pending" - for i := range prs { - if prs[i].HeadSHA == "" { - prs[i].CIStatus = ciStatusPending - continue - } - owner, repoName := c.splitRepo(prs[i].Repo) - - // Fetch the PR individually to learn its mergeability. The list - // endpoint that produced these PullRequests never populates - // "mergeable"/"mergeable_state" — GitHub computes them per-PR and - // returns them only from this single-PR GET. On error we leave the - // field as MergeableUnknown rather than guessing. - if full, _, err := c.client.PullRequests.Get(ctx, owner, repoName, prs[i].Number); err != nil { - c.logger.Warn("failed to fetch PR mergeability", "repo", prs[i].Repo, "pr", prs[i].Number, "error", err) - } else { - prs[i].Mergeable = mergeableFromState(full.GetMergeableState(), full.Mergeable) - prs[i].MergeableState = full.GetMergeableState() - } + if pr.HeadSHA == "" { + pr.CIStatus = ciStatusPending + return nil + } + owner, repoName := c.splitRepo(pr.Repo) - checkRuns, _, err := c.client.Checks.ListCheckRunsForRef(ctx, owner, repoName, prs[i].HeadSHA, &gh.ListCheckRunsOptions{ - ListOptions: gh.ListOptions{PerPage: 100}, - }) - if err != nil { - c.logger.Warn("failed to fetch check runs", "repo", prs[i].Repo, "pr", prs[i].Number, "error", err) - prs[i].CIStatus = ciStatusPending - continue + // Fetch the PR individually to learn its mergeability. The list + // endpoint that produced these PullRequests never populates + // "mergeable"/"mergeable_state" — GitHub computes them per-PR and + // returns them only from this single-PR GET. On error we leave the + // field as MergeableUnknown rather than guessing. + if full, _, err := c.client.PullRequests.Get(ctx, owner, repoName, pr.Number); err != nil { + c.logger.Warn("failed to fetch PR mergeability", "repo", pr.Repo, "pr", pr.Number, "error", err) + } else { + pr.Mergeable = mergeableFromState(full.GetMergeableState(), full.Mergeable) + pr.MergeableState = full.GetMergeableState() + } + + checkRuns, _, err := c.client.Checks.ListCheckRunsForRef(ctx, owner, repoName, pr.HeadSHA, &gh.ListCheckRunsOptions{ + ListOptions: gh.ListOptions{PerPage: 100}, + }) + if err != nil { + c.logger.Warn("failed to fetch check runs", "repo", pr.Repo, "pr", pr.Number, "error", err) + pr.CIStatus = ciStatusPending + return nil + } + reported := make(map[string]bool, len(checkRuns.CheckRuns)) + for _, cr := range checkRuns.CheckRuns { + if name := cr.GetName(); name != "" { + reported[name] = true } - if checkRuns.GetTotal() == 0 { - prs[i].CIStatus = ciStatusPending + } + if checkRuns.GetTotal() == 0 { + pr.CIStatus = ciStatusPending + return reported + } + hasFail := false + allDone := true + ciChecksFound := 0 + var failingNames []string + var failingIDs []int64 + for _, cr := range checkRuns.CheckRuns { + if isMetaCheck(cr.GetName()) { continue } - hasFail := false - allDone := true - ciChecksFound := 0 - var failingNames []string - var failingIDs []int64 - for _, cr := range checkRuns.CheckRuns { - if isMetaCheck(cr.GetName()) { - continue - } - ciChecksFound++ - if cr.GetStatus() != "completed" { - allDone = false - continue - } - conclusion := cr.GetConclusion() - if conclusion == "failure" || conclusion == "action_required" { - hasFail = true - failingNames = append(failingNames, cr.GetName()) - failingIDs = append(failingIDs, cr.GetID()) - } - } - if ciChecksFound == 0 { - prs[i].CIStatus = ciStatusPending + ciChecksFound++ + if cr.GetStatus() != "completed" { + allDone = false continue } - switch { - case hasFail: - prs[i].CIStatus = ciStatusFailure - prs[i].FailingChecks = failingNames - prs[i].CIFailureExcerpt = c.fetchFailureExcerpt(ctx, owner, repoName, failingIDs, failingNames) - case allDone: - prs[i].CIStatus = ciStatusSuccess - default: - prs[i].CIStatus = ciStatusPending + conclusion := cr.GetConclusion() + if conclusion == "failure" || conclusion == "action_required" { + hasFail = true + failingNames = append(failingNames, cr.GetName()) + failingIDs = append(failingIDs, cr.GetID()) } } + if ciChecksFound == 0 { + pr.CIStatus = ciStatusPending + return reported + } + switch { + case hasFail: + pr.CIStatus = ciStatusFailure + pr.FailingChecks = failingNames + pr.CIFailureExcerpt = c.fetchFailureExcerpt(ctx, owner, repoName, failingIDs, failingNames) + case allDone: + pr.CIStatus = ciStatusSuccess + default: + pr.CIStatus = ciStatusPending + } + return reported } // isMetaCheck reports check runs that are merge-gates or deploy-status diff --git a/src/pkg/github/merge_block_reason.go b/src/pkg/github/merge_block_reason.go new file mode 100644 index 0000000000..63e7d18a5f --- /dev/null +++ b/src/pkg/github/merge_block_reason.go @@ -0,0 +1,163 @@ +package github + +import ( + "sort" + "strconv" + "strings" +) + +// ReviewDecision is GitHub's own aggregate verdict on a PR's reviews, as +// reported by the GraphQL PullRequest.reviewDecision field. It is the only +// signal that distinguishes "a review is required and nobody has given one" +// from "a reviewer asked for changes" — REST exposes neither, and +// mergeable_state folds both into the single word "blocked". +type ReviewDecision string + +const ( + // ReviewDecisionNone is the zero value: GitHub said nothing, either + // because the base branch requires no review or because the field was + // never fetched. It must never be read as "approved". + ReviewDecisionNone ReviewDecision = "" + // ReviewDecisionApproved means the review requirement is satisfied. + ReviewDecisionApproved ReviewDecision = "APPROVED" + // ReviewDecisionChangesRequested means a reviewer asked for changes and + // has not dismissed that review. + ReviewDecisionChangesRequested ReviewDecision = "CHANGES_REQUESTED" + // ReviewDecisionReviewRequired means branch protection requires an + // approving review that has not been given. + ReviewDecisionReviewRequired ReviewDecision = "REVIEW_REQUIRED" +) + +// ProtectionFacts are the branch-protection observations gathered for one PR +// during the sweep's normal enumeration — no extra per-PR API call, and +// certainly no call per dashboard hover. Every field is optional: an empty +// one means "not determined", never "not required". +type ProtectionFacts struct { + // ReviewDecision is GitHub's aggregate review verdict, or "" when it was + // not fetched. + ReviewDecision ReviewDecision `json:"review_decision,omitempty"` + // ChangesRequestedBy are the logins whose latest review on this PR asked + // for changes. + ChangesRequestedBy []string `json:"changes_requested_by,omitempty"` + // ApprovalsGiven counts distinct reviewers whose latest opinionated + // review is an approval. + ApprovalsGiven int `json:"approvals_given,omitempty"` + // RequiredChecksKnown records that the required status-check set for the + // PR's base branch was determined (from auto_merge.required_checks or + // from the branch-protection API). Without it, FailingRequiredChecks and + // MissingRequiredChecks say nothing. + RequiredChecksKnown bool `json:"required_checks_known,omitempty"` + // FailingRequiredChecks are the required checks observed red on the head + // commit. + FailingRequiredChecks []string `json:"failing_required_checks,omitempty"` + // MissingRequiredChecks are required checks for which NO check run exists + // on the head commit. It is only ever populated when at least one OTHER + // required check WAS observed as a check run: on a repository that + // reports its required contexts as commit statuses rather than check + // runs, every required context looks absent from a check-run listing, and + // "required check X has not reported" would be confidently wrong for all + // of them. See protectionCollector.attach. + MissingRequiredChecks []string `json:"missing_required_checks,omitempty"` +} + +// BranchProtectionBlockReason names, in one place, the branch-protection rule +// that GitHub's "blocked" mergeable_state is hiding (hivecommons/hive#7515). +// +// This is THE state → explanation mapping for the blocked case: the dashboard +// pill's tooltip renders whatever this returns, so the wording lives in Go, +// under test, and never in JS string concatenation. +// +// ok is false when the facts on hand do not identify a rule. The caller must +// then say so honestly rather than guess — a confidently wrong explanation +// sends the operator to fix something that is not broken, which is worse than +// admitting we do not know. Order is by what the operator acts on first: an +// explicit "changes requested" outranks a red check, which outranks a check +// that never ran, which outranks a merely missing approval. +func (p PullRequest) BranchProtectionBlockReason() (string, bool) { + f := p.Protection + if f == nil { + return "", false + } + if f.ReviewDecision == ReviewDecisionChangesRequested { + if who := mentionList(f.ChangesRequestedBy); who != "" { + return "changes requested by " + who, true + } + return "a reviewer requested changes", true + } + if f.RequiredChecksKnown { + if names := quotedList(f.FailingRequiredChecks); names != "" { + return "required " + checkNoun(f.FailingRequiredChecks) + " " + names + " " + isAre(f.FailingRequiredChecks) + " failing", true + } + if names := quotedList(f.MissingRequiredChecks); names != "" { + return "required " + checkNoun(f.MissingRequiredChecks) + " " + names + " " + hasHave(f.MissingRequiredChecks) + " not reported", true + } + } + if f.ReviewDecision == ReviewDecisionReviewRequired { + return "an approving review is required by branch protection (" + + strconv.Itoa(f.ApprovalsGiven) + " given)", true + } + return "", false +} + +// mentionList renders logins as "@a, @b", skipping blanks and duplicates. +func mentionList(logins []string) string { + seen := make(map[string]bool, len(logins)) + out := make([]string, 0, len(logins)) + for _, l := range logins { + l = strings.TrimSpace(strings.TrimPrefix(strings.TrimSpace(l), "@")) + if l == "" || seen[strings.ToLower(l)] { + continue + } + seen[strings.ToLower(l)] = true + out = append(out, "@"+l) + } + sort.Strings(out) + return strings.Join(out, ", ") +} + +// quotedList renders check names as `"build", "lint"` in a stable order. +func quotedList(names []string) string { + seen := make(map[string]bool, len(names)) + out := make([]string, 0, len(names)) + for _, n := range names { + n = strings.TrimSpace(n) + if n == "" || seen[n] { + continue + } + seen[n] = true + out = append(out, strconv.Quote(n)) + } + sort.Strings(out) + return strings.Join(out, ", ") +} + +func checkNoun(names []string) string { + if countDistinct(names) == 1 { + return "check" + } + return "checks" +} + +func isAre(names []string) string { + if countDistinct(names) == 1 { + return "is" + } + return "are" +} + +func hasHave(names []string) string { + if countDistinct(names) == 1 { + return "has" + } + return "have" +} + +func countDistinct(names []string) int { + seen := make(map[string]bool, len(names)) + for _, n := range names { + if n = strings.TrimSpace(n); n != "" { + seen[n] = true + } + } + return len(seen) +} diff --git a/src/pkg/github/merge_block_reason_7515_test.go b/src/pkg/github/merge_block_reason_7515_test.go new file mode 100644 index 0000000000..26cd87ff08 --- /dev/null +++ b/src/pkg/github/merge_block_reason_7515_test.go @@ -0,0 +1,215 @@ +package github + +import "testing" + +// TestBranchProtectionBlockReason is the pin for the ONE place that maps a +// blocked PR's branch-protection facts to the words an operator reads in the +// dashboard tooltip (hivecommons/hive#7515). Every row asserts the EXACT +// string, so a rewording anywhere has to come through here. +func TestBranchProtectionBlockReason(t *testing.T) { + tests := []struct { + name string + pr PullRequest + want string + wantOK bool + comment string + }{ + { + name: "no facts at all declines to guess", + pr: PullRequest{Repo: "org/repo", Number: 1}, + // Negative control: the function must NOT return a plausible + // default. "unknown" is the honest answer and the caller prints + // its own placeholder. + wantOK: false, + }, + { + name: "facts present but empty declines to guess", + pr: PullRequest{Protection: &ProtectionFacts{}}, + // Second negative control: a non-nil Protection whose every + // field is the zero value is still "we determined nothing". + wantOK: false, + }, + { + name: "required set known and everything green declines to guess", + pr: PullRequest{Protection: &ProtectionFacts{ + RequiredChecksKnown: true, + ReviewDecision: ReviewDecisionApproved, + }}, + // Third negative control: approved + no red/absent required + // check means the rule is one we do not read (a merge-queue + // rule, a CODEOWNERS rule). Do not invent one. + wantOK: false, + }, + { + name: "changes requested names the reviewer", + pr: PullRequest{Protection: &ProtectionFacts{ + ReviewDecision: ReviewDecisionChangesRequested, + ChangesRequestedBy: []string{"reviewer"}, + }}, + want: "changes requested by @reviewer", + wantOK: true, + }, + { + name: "changes requested by several, deduped, @-stripped, sorted", + pr: PullRequest{Protection: &ProtectionFacts{ + ReviewDecision: ReviewDecisionChangesRequested, + ChangesRequestedBy: []string{"zoe", "@alice", "zoe", ""}, + }}, + want: "changes requested by @alice, @zoe", + wantOK: true, + }, + { + name: "changes requested with no login still says so", + pr: PullRequest{Protection: &ProtectionFacts{ + ReviewDecision: ReviewDecisionChangesRequested, + ChangesRequestedBy: nil, + }}, + want: "a reviewer requested changes", + wantOK: true, + }, + { + name: "one failing required check", + pr: PullRequest{Protection: &ProtectionFacts{ + RequiredChecksKnown: true, + FailingRequiredChecks: []string{"build"}, + }}, + want: `required check "build" is failing`, + wantOK: true, + }, + { + name: "two failing required checks pluralise", + pr: PullRequest{Protection: &ProtectionFacts{ + RequiredChecksKnown: true, + FailingRequiredChecks: []string{"lint", "build"}, + }}, + want: `required checks "build", "lint" are failing`, + wantOK: true, + }, + { + name: "required check that never reported", + pr: PullRequest{Protection: &ProtectionFacts{ + RequiredChecksKnown: true, + MissingRequiredChecks: []string{"validate"}, + }}, + want: `required check "validate" has not reported`, + wantOK: true, + }, + { + name: "two required checks that never reported pluralise", + pr: PullRequest{Protection: &ProtectionFacts{ + RequiredChecksKnown: true, + MissingRequiredChecks: []string{"validate", "dco"}, + }}, + want: `required checks "dco", "validate" have not reported`, + wantOK: true, + }, + { + name: "missing required checks are ignored when the set is not known", + pr: PullRequest{Protection: &ProtectionFacts{ + RequiredChecksKnown: false, + MissingRequiredChecks: []string{"validate"}, + ReviewDecision: ReviewDecisionReviewRequired, + }}, + // Fourth negative control: without a known required set the + // check lists mean nothing, so the review rule is what is left. + want: "an approving review is required by branch protection (0 given)", + wantOK: true, + }, + { + name: "review required names the shortfall", + pr: PullRequest{Protection: &ProtectionFacts{ + ReviewDecision: ReviewDecisionReviewRequired, + ApprovalsGiven: 1, + }}, + want: "an approving review is required by branch protection (1 given)", + wantOK: true, + }, + { + name: "changes requested outranks a red required check", + pr: PullRequest{Protection: &ProtectionFacts{ + ReviewDecision: ReviewDecisionChangesRequested, + ChangesRequestedBy: []string{"alice"}, + RequiredChecksKnown: true, + FailingRequiredChecks: []string{"build"}, + }}, + want: "changes requested by @alice", + wantOK: true, + }, + { + name: "a red required check outranks one that never reported", + pr: PullRequest{Protection: &ProtectionFacts{ + RequiredChecksKnown: true, + FailingRequiredChecks: []string{"build"}, + MissingRequiredChecks: []string{"validate"}, + }}, + want: `required check "build" is failing`, + wantOK: true, + }, + { + name: "a red required check outranks a missing approval", + pr: PullRequest{Protection: &ProtectionFacts{ + ReviewDecision: ReviewDecisionReviewRequired, + RequiredChecksKnown: true, + FailingRequiredChecks: []string{"build"}, + }}, + want: `required check "build" is failing`, + wantOK: true, + }, + } + + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + got, ok := tt.pr.BranchProtectionBlockReason() + if ok != tt.wantOK { + t.Fatalf("ok = %v, want %v (reason %q)", ok, tt.wantOK, got) + } + if got != tt.want { + t.Errorf("reason = %q, want %q", got, tt.want) + } + }) + } +} + +// TestBranchProtectionBlockReason_NeverInventsAReason is the invariant behind +// the negative controls above, stated once: whenever the function claims a +// reason, some fact must support it. A build that returns a hardcoded string +// for everything passes each individual row's "want" only by accident; this +// one fails it outright. +func TestBranchProtectionBlockReason_NeverInventsAReason(t *testing.T) { + silent := []PullRequest{ + {}, + {Protection: &ProtectionFacts{}}, + {Protection: &ProtectionFacts{ReviewDecision: ReviewDecisionApproved}}, + {Protection: &ProtectionFacts{RequiredChecksKnown: true}}, + {Protection: &ProtectionFacts{ReviewDecision: ReviewDecisionNone, RequiredChecksKnown: true}}, + {Protection: &ProtectionFacts{FailingRequiredChecks: []string{"build"}}}, + {Protection: &ProtectionFacts{MissingRequiredChecks: []string{"validate"}}}, + } + for i, pr := range silent { + if got, ok := pr.BranchProtectionBlockReason(); ok { + t.Errorf("case %d: claimed %q with nothing to support it", i, got) + } + } + + // ...and the reasons it DOES give must differ from one another, so a + // single hardcoded string cannot satisfy the table above. + distinct := map[string]bool{} + for _, pr := range []PullRequest{ + {Protection: &ProtectionFacts{ReviewDecision: ReviewDecisionChangesRequested, ChangesRequestedBy: []string{"alice"}}}, + {Protection: &ProtectionFacts{RequiredChecksKnown: true, FailingRequiredChecks: []string{"build"}}}, + {Protection: &ProtectionFacts{RequiredChecksKnown: true, MissingRequiredChecks: []string{"validate"}}}, + {Protection: &ProtectionFacts{ReviewDecision: ReviewDecisionReviewRequired}}, + } { + got, ok := pr.BranchProtectionBlockReason() + if !ok { + t.Fatalf("expected a reason for %+v", pr.Protection) + } + if distinct[got] { + t.Errorf("reason %q is not specific to its facts", got) + } + distinct[got] = true + } + if len(distinct) != 4 { + t.Errorf("got %d distinct reasons, want 4", len(distinct)) + } +} diff --git a/src/pkg/github/protection_facts.go b/src/pkg/github/protection_facts.go new file mode 100644 index 0000000000..931dbe516c --- /dev/null +++ b/src/pkg/github/protection_facts.go @@ -0,0 +1,216 @@ +package github + +import ( + "context" + "strings" +) + +// protectionCollector gathers, once per enrichment pass, the per-repository +// data needed to name the branch-protection rule behind a "blocked" PR +// (hivecommons/hive#7515, step 2). +// +// Cost discipline: the required-check set comes from the operator's +// auto_merge.required_checks config when one is installed — zero API calls, +// the primary path — and falls back to the branch-protection API once per +// repo+branch. Review decisions come from ONE GraphQL query per repository, +// paginated 100 PRs at a time, not one per PR. Both are memoised for the +// lifetime of the pass. Nothing is fetched on a dashboard hover. +type protectionCollector struct { + c *Client + // required is keyed "owner/repo@branch". + required map[string]requiredSet + // reviews is keyed "owner/repo"; a nil map value records a repo whose + // GraphQL query failed, so it is attempted only once. + reviews map[string]map[int]prReviewState +} + +type requiredSet struct { + set map[string]bool + known bool +} + +// prReviewState is GitHub's review verdict for one PR. +type prReviewState struct { + decision ReviewDecision + changesRequestedBy []string + approvals int +} + +func newProtectionCollector(c *Client) *protectionCollector { + return &protectionCollector{ + c: c, + required: make(map[string]requiredSet), + reviews: make(map[string]map[int]prReviewState), + } +} + +// attach populates pr.Protection from the facts on hand. reported is the set +// of check-run names observed on the PR's head commit, or nil when no +// check-run listing was obtained — the distinction matters, because a missing +// listing is not evidence that a required check never ran. +// +// pr.Protection is left nil when nothing at all was determined, so that +// BranchProtectionBlockReason declines to guess rather than inventing a rule. +func (pc *protectionCollector) attach(ctx context.Context, pr *PullRequest, reported map[string]bool) { + if pc == nil || pc.c == nil || pr == nil { + return + } + var f ProtectionFacts + + if rs, ok := pc.reviewState(ctx, pr.Repo, pr.Number); ok { + f.ReviewDecision = rs.decision + f.ChangesRequestedBy = rs.changesRequestedBy + f.ApprovalsGiven = rs.approvals + } + + if reported != nil { + req := pc.requiredChecks(ctx, pr.Repo, pr.BaseRef) + if req.known { + f.RequiredChecksKnown = true + failing := make(map[string]bool, len(pr.FailingChecks)) + for _, n := range pr.FailingChecks { + failing[n] = true + } + // observedRequired proves this repository reports its required + // contexts as CHECK RUNS. Without that proof a required context + // absent from the check-run listing may simply be a commit + // status, which this pass does not fetch; claiming it "has not + // reported" would be confidently wrong for every required + // context on such a repo. + observedRequired := 0 + var missing, red []string + for name := range req.set { + switch { + case failing[name]: + observedRequired++ + red = append(red, name) + case reported[name]: + observedRequired++ + default: + missing = append(missing, name) + } + } + f.FailingRequiredChecks = red + if observedRequired > 0 { + f.MissingRequiredChecks = missing + } + } + } + + if f.ReviewDecision == ReviewDecisionNone && !f.RequiredChecksKnown { + return + } + pr.Protection = &f +} + +func (pc *protectionCollector) requiredChecks(ctx context.Context, repo, branch string) requiredSet { + key := repo + "@" + branch + if rs, ok := pc.required[key]; ok { + return rs + } + owner, name := pc.c.splitRepo(repo) + set, known := pc.c.requiredStatusCheckContexts(ctx, owner, name, branch) + rs := requiredSet{set: set, known: known} + pc.required[key] = rs + return rs +} + +func (pc *protectionCollector) reviewState(ctx context.Context, repo string, number int) (prReviewState, bool) { + byNumber, ok := pc.reviews[repo] + if !ok { + byNumber = pc.c.fetchReviewDecisions(ctx, repo) + pc.reviews[repo] = byNumber + } + if byNumber == nil { + return prReviewState{}, false + } + rs, ok := byNumber[number] + return rs, ok +} + +// reviewDecisionQuery asks for every open PR's review verdict in one request. +// latestOpinionatedReviews is GitHub's own per-author deduplication: one +// entry per reviewer, carrying that reviewer's current position, which is +// exactly the "0 approvals given" / "changes requested by @x" the operator +// needs and is not derivable from a raw review list. +const reviewDecisionQuery = `query($owner:String!,$name:String!,$cursor:String){ + repository(owner:$owner,name:$name){ + pullRequests(states:OPEN,first:100,after:$cursor){ + pageInfo{hasNextPage endCursor} + nodes{ + number + reviewDecision + latestOpinionatedReviews(first:50){nodes{state author{login}}} + } + } + } +}` + +type reviewDecisionResponse struct { + Repository struct { + PullRequests struct { + PageInfo struct { + HasNextPage bool `json:"hasNextPage"` + EndCursor string `json:"endCursor"` + } `json:"pageInfo"` + Nodes []struct { + Number int `json:"number"` + ReviewDecision string `json:"reviewDecision"` + LatestOpinionatedReviews struct { + Nodes []struct { + State string `json:"state"` + Author *struct { + Login string `json:"login"` + } `json:"author"` + } `json:"nodes"` + } `json:"latestOpinionatedReviews"` + } `json:"nodes"` + } `json:"pullRequests"` + } `json:"repository"` +} + +// fetchReviewDecisions returns GitHub's review verdict for every open PR in +// repo, or nil when the query could not be answered (no GraphQL scope, a GHE +// that does not serve the field, a transport error). nil means "not +// determined" and is never read as "approved". +func (c *Client) fetchReviewDecisions(ctx context.Context, repo string) map[int]prReviewState { + if c == nil || c.client == nil { + return nil + } + owner, name := c.splitRepo(repo) + if owner == "" || name == "" { + return nil + } + out := make(map[int]prReviewState) + vars := map[string]any{"owner": owner, "name": name} + for page := 0; page < 20; page++ { + var resp reviewDecisionResponse + if err := c.graphQL(ctx, reviewDecisionQuery, vars, &resp); err != nil { + c.logger.Warn("failed to fetch PR review decisions", "repo", repo, "error", err) + return nil + } + for _, n := range resp.Repository.PullRequests.Nodes { + st := prReviewState{decision: ReviewDecision(strings.ToUpper(strings.TrimSpace(n.ReviewDecision)))} + for _, r := range n.LatestOpinionatedReviews.Nodes { + login := "" + if r.Author != nil { + login = r.Author.Login + } + switch strings.ToUpper(strings.TrimSpace(r.State)) { + case "APPROVED": + st.approvals++ + case "CHANGES_REQUESTED": + if login != "" { + st.changesRequestedBy = append(st.changesRequestedBy, login) + } + } + } + out[n.Number] = st + } + if !resp.Repository.PullRequests.PageInfo.HasNextPage || resp.Repository.PullRequests.PageInfo.EndCursor == "" { + break + } + vars["cursor"] = resp.Repository.PullRequests.PageInfo.EndCursor + } + return out +} diff --git a/src/pkg/github/protection_facts_7515_test.go b/src/pkg/github/protection_facts_7515_test.go new file mode 100644 index 0000000000..98acc6dd0d --- /dev/null +++ b/src/pkg/github/protection_facts_7515_test.go @@ -0,0 +1,292 @@ +package github + +import ( + "context" + "encoding/json" + "io" + "net/http" + "net/http/httptest" + "strings" + "sync" + "testing" +) + +// protectionServer is a fake forge that answers the three endpoints +// EnrichCIStatus touches: the single-PR GET (mergeability), the check-run +// listing, and the GraphQL query that carries review decisions. +type protectionServer struct { + mu sync.Mutex + graphQLCalls int + checkRuns []map[string]any + mergeableState string + reviewDecision string + reviews []map[string]any + graphQLFails bool +} + +func (p *protectionServer) start(t *testing.T) *httptest.Server { + t.Helper() + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + switch { + case strings.HasSuffix(r.URL.Path, "/graphql"): + p.mu.Lock() + p.graphQLCalls++ + fails := p.graphQLFails + p.mu.Unlock() + body, _ := io.ReadAll(r.Body) + if !strings.Contains(string(body), "reviewDecision") { + t.Errorf("graphql query did not ask for reviewDecision: %s", body) + } + if fails { + w.WriteHeader(http.StatusForbidden) + _, _ = w.Write([]byte(`{"message":"no scope"}`)) + return + } + _ = json.NewEncoder(w).Encode(map[string]any{ + "data": map[string]any{ + "repository": map[string]any{ + "pullRequests": map[string]any{ + "pageInfo": map[string]any{"hasNextPage": false, "endCursor": ""}, + "nodes": []map[string]any{{ + "number": 1, + "reviewDecision": p.reviewDecision, + "latestOpinionatedReviews": map[string]any{"nodes": p.reviews}, + }}, + }, + }, + }, + }) + case strings.HasSuffix(r.URL.Path, "/pulls/1"): + _ = json.NewEncoder(w).Encode(map[string]any{ + "number": 1, + "mergeable": false, + "mergeable_state": p.mergeableState, + }) + case strings.Contains(r.URL.Path, "/check-runs"): + _ = json.NewEncoder(w).Encode(map[string]any{ + "total_count": len(p.checkRuns), + "check_runs": p.checkRuns, + }) + default: + w.WriteHeader(http.StatusNotFound) + } + })) + t.Cleanup(srv.Close) + return srv +} + +func enrichOnePR(t *testing.T, p *protectionServer, required map[string]bool) PullRequest { + t.Helper() + srv := p.start(t) + c := newTestClient(t, srv, "org", []string{"repo"}) + if required != nil { + c.SetRequiredChecks(required) + } + prs := []PullRequest{{Repo: "org/repo", Number: 1, HeadSHA: "abc123", BaseRef: "v4"}} + c.EnrichCIStatus(context.Background(), prs) + return prs[0] +} + +// A red REQUIRED check is the commonest thing behind GitHub's "blocked", and +// the sweep already knows the required set from auto_merge.required_checks — +// no branch-protection API call is needed for this path at all. +func TestEnrichCIStatus_ProtectionNamesFailingRequiredCheck(t *testing.T) { + p := &protectionServer{ + mergeableState: "blocked", + reviewDecision: "APPROVED", + checkRuns: []map[string]any{ + {"name": "build", "status": "completed", "conclusion": "failure"}, + {"name": "lint", "status": "completed", "conclusion": "success"}, + }, + } + pr := enrichOnePR(t, p, map[string]bool{"build": true, "lint": true}) + if pr.Protection == nil { + t.Fatal("Protection is nil; the blocked pill has nothing to say") + } + if !pr.Protection.RequiredChecksKnown { + t.Error("RequiredChecksKnown = false, want true (the config set is installed)") + } + got, ok := pr.BranchProtectionBlockReason() + if !ok || got != `required check "build" is failing` { + t.Errorf("reason = %q (ok=%v), want `required check \"build\" is failing`", got, ok) + } +} + +// A required check that never produced a check run is "expected" in GitHub's +// vocabulary and invisible in every REST field the sweep reads. +func TestEnrichCIStatus_ProtectionNamesRequiredCheckThatNeverReported(t *testing.T) { + p := &protectionServer{ + mergeableState: "blocked", + reviewDecision: "APPROVED", + checkRuns: []map[string]any{ + {"name": "build", "status": "completed", "conclusion": "success"}, + }, + } + pr := enrichOnePR(t, p, map[string]bool{"build": true, "validate": true}) + if pr.Protection == nil { + t.Fatal("Protection is nil") + } + got, ok := pr.BranchProtectionBlockReason() + if !ok || got != `required check "validate" has not reported` { + t.Errorf("reason = %q (ok=%v), want `required check \"validate\" has not reported`", got, ok) + } +} + +// The guard that keeps this honest: on a repository whose required contexts +// arrive as COMMIT STATUSES, not check runs, every required context is absent +// from the check-run listing. Claiming all of them "never reported" would be +// confidently wrong, so nothing at all is claimed. +func TestEnrichCIStatus_NoRequiredCheckObservedMakesNoMissingClaim(t *testing.T) { + p := &protectionServer{ + mergeableState: "blocked", + reviewDecision: "APPROVED", + checkRuns: []map[string]any{ + {"name": "some-unrelated-check", "status": "completed", "conclusion": "success"}, + }, + } + pr := enrichOnePR(t, p, map[string]bool{"validate": true, "dco": true}) + if pr.Protection == nil { + t.Fatal("Protection is nil") + } + if len(pr.Protection.MissingRequiredChecks) != 0 { + t.Errorf("MissingRequiredChecks = %v, want none: no required context was seen as a check run, so their absence proves nothing", pr.Protection.MissingRequiredChecks) + } + if got, ok := pr.BranchProtectionBlockReason(); ok { + t.Errorf("reason = %q, want none (nothing supports a claim here)", got) + } +} + +// A check-run listing that could not be fetched must never be read as "the +// required check never reported". +func TestEnrichCIStatus_CheckRunFetchFailureMakesNoCheckClaim(t *testing.T) { + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + switch { + case strings.HasSuffix(r.URL.Path, "/graphql"): + _ = json.NewEncoder(w).Encode(map[string]any{"data": map[string]any{ + "repository": map[string]any{"pullRequests": map[string]any{ + "pageInfo": map[string]any{"hasNextPage": false}, + "nodes": []map[string]any{{ + "number": 1, "reviewDecision": "APPROVED", + "latestOpinionatedReviews": map[string]any{"nodes": []map[string]any{}}, + }}, + }}, + }}) + case strings.HasSuffix(r.URL.Path, "/pulls/1"): + _ = json.NewEncoder(w).Encode(map[string]any{"number": 1, "mergeable_state": "blocked"}) + default: + w.WriteHeader(http.StatusInternalServerError) + } + })) + defer srv.Close() + c := newTestClient(t, srv, "org", []string{"repo"}) + c.SetRequiredChecks(map[string]bool{"validate": true}) + prs := []PullRequest{{Repo: "org/repo", Number: 1, HeadSHA: "abc123", BaseRef: "v4"}} + c.EnrichCIStatus(context.Background(), prs) + + if prs[0].Protection != nil && prs[0].Protection.RequiredChecksKnown { + t.Error("RequiredChecksKnown = true with no check-run listing; absence of evidence is not evidence") + } + if got, ok := prs[0].BranchProtectionBlockReason(); ok { + t.Errorf("reason = %q, want none", got) + } +} + +// GitHub's own review verdict is the only signal that separates "nobody has +// reviewed" from "somebody said no", and REST exposes neither. +func TestEnrichCIStatus_ProtectionCarriesReviewDecision(t *testing.T) { + t.Run("changes requested names the reviewer", func(t *testing.T) { + p := &protectionServer{ + mergeableState: "blocked", + reviewDecision: "CHANGES_REQUESTED", + reviews: []map[string]any{ + {"state": "CHANGES_REQUESTED", "author": map[string]any{"login": "octo"}}, + }, + checkRuns: []map[string]any{{"name": "build", "status": "completed", "conclusion": "success"}}, + } + pr := enrichOnePR(t, p, nil) + if pr.Protection == nil || pr.Protection.ReviewDecision != ReviewDecisionChangesRequested { + t.Fatalf("ReviewDecision not carried: %+v", pr.Protection) + } + got, ok := pr.BranchProtectionBlockReason() + if !ok || got != "changes requested by @octo" { + t.Errorf("reason = %q (ok=%v), want %q", got, ok, "changes requested by @octo") + } + }) + + t.Run("review required counts the approvals given", func(t *testing.T) { + p := &protectionServer{ + mergeableState: "blocked", + reviewDecision: "REVIEW_REQUIRED", + reviews: []map[string]any{ + {"state": "APPROVED", "author": map[string]any{"login": "one"}}, + }, + checkRuns: []map[string]any{{"name": "build", "status": "completed", "conclusion": "success"}}, + } + pr := enrichOnePR(t, p, nil) + if pr.Protection == nil { + t.Fatal("Protection is nil") + } + if pr.Protection.ApprovalsGiven != 1 { + t.Errorf("ApprovalsGiven = %d, want 1", pr.Protection.ApprovalsGiven) + } + got, ok := pr.BranchProtectionBlockReason() + want := "an approving review is required by branch protection (1 given)" + if !ok || got != want { + t.Errorf("reason = %q (ok=%v), want %q", got, ok, want) + } + }) +} + +// A forge that will not answer the GraphQL query (no scope, an older GHE) +// must leave the review decision undetermined — never "approved" — and must +// not be asked again for every PR in the repository. +func TestEnrichCIStatus_ReviewDecisionFailureIsOnePerRepoAndNotApproved(t *testing.T) { + p := &protectionServer{ + mergeableState: "blocked", + graphQLFails: true, + checkRuns: []map[string]any{{"name": "build", "status": "completed", "conclusion": "success"}}, + } + srv := p.start(t) + c := newTestClient(t, srv, "org", []string{"repo"}) + prs := []PullRequest{ + {Repo: "org/repo", Number: 1, HeadSHA: "abc123", BaseRef: "v4"}, + {Repo: "org/repo", Number: 1, HeadSHA: "abc123", BaseRef: "v4"}, + {Repo: "org/repo", Number: 1, HeadSHA: "abc123", BaseRef: "v4"}, + } + c.EnrichCIStatus(context.Background(), prs) + + p.mu.Lock() + calls := p.graphQLCalls + p.mu.Unlock() + if calls != 1 { + t.Errorf("graphQL called %d times for one repository, want 1: a per-PR query is unaffordable at a 400-PR queue", calls) + } + for i := range prs { + if prs[i].Protection != nil && prs[i].Protection.ReviewDecision == ReviewDecisionApproved { + t.Errorf("pr %d: an unanswerable query became %q", i, ReviewDecisionApproved) + } + } +} + +// One GraphQL query serves every PR in the repository. +func TestEnrichCIStatus_ReviewDecisionIsOneQueryPerRepo(t *testing.T) { + p := &protectionServer{ + mergeableState: "blocked", + reviewDecision: "REVIEW_REQUIRED", + checkRuns: []map[string]any{{"name": "build", "status": "completed", "conclusion": "success"}}, + } + srv := p.start(t) + c := newTestClient(t, srv, "org", []string{"repo"}) + prs := make([]PullRequest, 5) + for i := range prs { + prs[i] = PullRequest{Repo: "org/repo", Number: 1, HeadSHA: "abc123", BaseRef: "v4"} + } + c.EnrichCIStatus(context.Background(), prs) + + p.mu.Lock() + calls := p.graphQLCalls + p.mu.Unlock() + if calls != 1 { + t.Errorf("graphQL called %d times for 5 PRs in one repository, want 1", calls) + } +} From ed4edfb065834df8500481f9499855e7039faba7 Mon Sep 17 00:00:00 2001 From: clubanderson Date: Thu, 17 Sep 2026 22:55:54 -0400 Subject: [PATCH 11/17] =?UTF-8?q?=F0=9F=8C=B1=20ci:=20cap=20Actions=20cach?= =?UTF-8?q?e=20segment=20downloads=20at=201=20minute?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The warm-cache restore is only a win when it is faster than the compile it replaces. On some cluster nodes the download from the Actions cache backend crawls — observed 16-minute 'Restore warm Go build cache' steps (run 35299669695, test agent 3/5 and dashboard shuffle 2/2) against a ~2-minute cold compile, making the two slow jobs the run's 21-minute long poles. The actions/cache default gives each segment 10 minutes; cap it at 1 so a slow restore aborts and the job builds from scratch instead. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: clubanderson --- .github/workflows/v2-ci.yml | 5 +++++ .github/workflows/v2-tests.yml | 5 +++++ 2 files changed, 10 insertions(+) diff --git a/.github/workflows/v2-ci.yml b/.github/workflows/v2-ci.yml index 061e9e2838..e2dd325b52 100644 --- a/.github/workflows/v2-ci.yml +++ b/.github/workflows/v2-ci.yml @@ -42,6 +42,11 @@ concurrency: group: ${{ github.workflow }}-${{ github.event_name == 'pull_request' && github.event.pull_request.number || github.run_id }} cancel-in-progress: ${{ github.event_name == 'pull_request' }} +# A slow cache-segment download must never cost more than a recompile +# (see v2-tests.yml — observed 16-minute restore steps on some nodes). +env: + ACTIONS_CACHE_SEGMENT_DOWNLOAD_TIMEOUT_MINS: '1' + jobs: build-and-test: runs-on: ${{ (github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == github.repository) && fromJSON(vars.HIVE_RUNNER_LABELS || '["ubuntu-latest"]') || 'ubuntu-latest' }} diff --git a/.github/workflows/v2-tests.yml b/.github/workflows/v2-tests.yml index ad8ee7596f..6ef4aa7ee0 100644 --- a/.github/workflows/v2-tests.yml +++ b/.github/workflows/v2-tests.yml @@ -75,6 +75,11 @@ env: # Self-hosted runners sit inside OpenShift; keep these hermetic unit tests from auto-discovering that cluster. KUBERNETES_SERVICE_HOST: '' KUBERNETES_SERVICE_PORT: '' + # A slow cache-segment download must never cost more than a recompile. + # Some cluster nodes crawl when pulling from the Actions cache backend — + # observed 16-minute "Restore warm Go build cache" steps against a ~2-minute + # cold compile. Abort a segment after 1 minute and let the job build instead. + ACTIONS_CACHE_SEGMENT_DOWNLOAD_TIMEOUT_MINS: '1' # Cancel superseded PR runs only (#4623). A new push to a PR makes the From 7f1871e4ed2cea29d312166f6cb9a2c6d3888b85 Mon Sep 17 00:00:00 2001 From: Andy Anderson Date: Thu, 17 Sep 2026 23:39:04 -0400 Subject: [PATCH 12/17] refactor(agent): split consent/env/modes/routing/thrash/copilot_auth out of manager.go (#7532) Verbatim block moves matching pre-adoption v5 file boundaries. manager.go: 4,321 -> 1,793 lines. Part-of: #7303 Signed-off-by: Andrew Anderson Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../changed-7303-manager-domain-split.md | 1 + src/pkg/agent/copilot_auth_error_test.go | 2 +- src/pkg/agent/copilot_models_test.go | 6 +- src/pkg/agent/manager.go | 1628 ----------------- src/pkg/agent/manager_consent.go | 295 +++ src/pkg/agent/manager_copilot_auth.go | 48 + src/pkg/agent/manager_env.go | 420 +++++ src/pkg/agent/manager_modes.go | 354 ++++ src/pkg/agent/manager_routing.go | 456 +++++ src/pkg/agent/manager_thrash.go | 100 + src/pkg/agent/ringbuffer.go | 1 - src/pkg/agent/routing_coverage_test.go | 14 +- 12 files changed, 1685 insertions(+), 1640 deletions(-) create mode 100644 changelog.d/changed-7303-manager-domain-split.md create mode 100644 src/pkg/agent/manager_consent.go create mode 100644 src/pkg/agent/manager_copilot_auth.go create mode 100644 src/pkg/agent/manager_env.go create mode 100644 src/pkg/agent/manager_modes.go create mode 100644 src/pkg/agent/manager_routing.go create mode 100644 src/pkg/agent/manager_thrash.go diff --git a/changelog.d/changed-7303-manager-domain-split.md b/changelog.d/changed-7303-manager-domain-split.md new file mode 100644 index 0000000000..bba77b803d --- /dev/null +++ b/changelog.d/changed-7303-manager-domain-split.md @@ -0,0 +1 @@ +- Split six more domains out of `pkg/agent/manager.go` (consent, copilot auth, env, modes, routing, thrash) into per-domain files matching the v5 layout; manager.go shrinks from 4,321 to 1,793 lines. No behavior change — verbatim block moves. diff --git a/src/pkg/agent/copilot_auth_error_test.go b/src/pkg/agent/copilot_auth_error_test.go index bee45083d0..3bae76453d 100644 --- a/src/pkg/agent/copilot_auth_error_test.go +++ b/src/pkg/agent/copilot_auth_error_test.go @@ -23,7 +23,7 @@ func TestMatchesAuthError_DefinitiveRejectionsOnly(t *testing.T) { // These must NOT purge — they are login prompts / benign output that can // appear during a normal cold start with a valid token on disk. noPurge := []string{ - "Please re-authenticate to continue", // the removed over-broad pattern + "Please re-authenticate to continue", // the removed over-broad pattern "To sign in, use a web browser to open the page https://github.com/login/device", "Copilot CLI ready", "Enter the code below to authenticate", diff --git a/src/pkg/agent/copilot_models_test.go b/src/pkg/agent/copilot_models_test.go index c3a16b15c2..d545c6fcf1 100644 --- a/src/pkg/agent/copilot_models_test.go +++ b/src/pkg/agent/copilot_models_test.go @@ -79,9 +79,9 @@ func TestNormalizeModelNameCopilotDrift(t *testing.T) { {"claude-opus-4.6", "claude-opus-4.6"}, // legitimate dot unchanged {"gpt-5.5", "gpt-5.5"}, {"gemini-2.5-pro", "gemini-2.5-pro"}, - {"auto", "auto"}, // auto-select sentinel flows through - {"gpt-next", "gpt-next"}, // unknown id passthrough - {"custom-7", "custom-7"}, // unknown id: no blind dot-rewrite anymore + {"auto", "auto"}, // auto-select sentinel flows through + {"gpt-next", "gpt-next"}, // unknown id passthrough + {"custom-7", "custom-7"}, // unknown id: no blind dot-rewrite anymore } for _, tt := range tests { if got := normalizeModelName(tt.in, "copilot"); got != tt.want { diff --git a/src/pkg/agent/manager.go b/src/pkg/agent/manager.go index 8a6a07b619..3abfcf2048 100644 --- a/src/pkg/agent/manager.go +++ b/src/pkg/agent/manager.go @@ -2,14 +2,11 @@ package agent import ( "context" - "encoding/json" "fmt" "log/slog" "os" "os/exec" "path/filepath" - "regexp" - "strconv" "strings" "sync" "sync/atomic" @@ -17,7 +14,6 @@ import ( "github.com/hivecommons/hive/pkg/claude" "github.com/hivecommons/hive/pkg/config" - ghpkg "github.com/hivecommons/hive/pkg/github" "github.com/hivecommons/hive/pkg/pushbroker" "github.com/hivecommons/hive/pkg/sandbox" "github.com/hivecommons/hive/pkg/watchdog" @@ -404,14 +400,6 @@ type AgentProcess struct { kickHoldReason string } -// effectiveBackend returns the agent's backend accounting for any override. -func effectiveBackend(agent *AgentProcess) string { - if agent.BackendOverride != "" { - return agent.BackendOverride - } - return agent.Config.Backend -} - // ProjectContext holds project-level config injected into agent boot prompts. type ProjectContext struct { Org string @@ -639,65 +627,6 @@ type Manager struct { kickLogMaxBytes int64 } -// IsInferenceBackend returns true if the backend is a self-hosted inference -// backend (vllm, llm-d, litellm) rather than a CLI tool. Delegates to the -// canonical list in the config package (shared with the proxy package, -// which cannot be imported from here without a cycle). -func IsInferenceBackend(backend string) bool { - return config.IsInferenceBackend(backend) -} - -// SetInferenceCallbacks registers callbacks that the manager uses to -// configure/clear inference routing on the proxy when launching agents. -func (m *Manager) SetInferenceCallbacks( - setRoute func(agentName, backend, model string), - clearRoute func(agentName string), -) { - m.mu.Lock() - defer m.mu.Unlock() - m.inferenceRouteCallback = setRoute - m.clearInferenceRouteCallback = clearRoute -} - -// SetGatewayBackendChecker injects a predicate that reports whether a backend -// string names a configured model gateway. This makes an agent whose backend is -// a gateway name inference-routable, so its route is resolved via the inference -// callback exactly like the built-in litellm/vllm/llm-d backends. -func (m *Manager) SetGatewayBackendChecker(fn func(backend string) bool) { - // Atomic store — no m.mu — so routableBackend can read it lock-free from the - // lock-holding launch path without deadlocking (see isGatewayBackend docs). - m.isGatewayBackend.Store(&fn) -} - -// routableBackend reports whether a backend should be routed through the -// inference proxy: either a built-in inference backend, or a configured gateway -// name. Safe to call while holding m.mu (isGatewayBackend is read atomically). -func (m *Manager) routableBackend(backend string) bool { - if IsInferenceBackend(backend) { - return true - } - // Lock-free atomic read: this is invoked from the launch path while m.mu is - // already held, so it MUST NOT take m.mu (non-reentrant RWMutex → deadlock). - fnp := m.isGatewayBackend.Load() - return fnp != nil && *fnp != nil && (*fnp)(backend) -} - -// validateBackendName reports whether backend is one the launcher can actually -// dispatch: an agentic CLI, a model-gateway backend, or a configured gateway -// name. An empty backend is valid (it means "the hive default"). -// -// This is the manager-side half of the accept-then-fail fix. It dispatches on -// the SAME canonical lists as config.ValidateBackend and backendBinary, so a -// backend accepted by any write path is one the launch path can start. -// Safe to call while holding m.mu (routableBackend reads atomically). -func (m *Manager) validateBackendName(backend string) error { - if backend == "" || config.IsCLIBackend(backend) || m.routableBackend(backend) { - return nil - } - return fmt.Errorf("unsupported backend %q (supported: %s; or the name of a configured model gateway)", - backend, strings.Join(config.SupportedBackends(), ", ")) -} - // SetACMMLevel updates the cached ACMM level used by agentMode() when // launching agents. Call this whenever the ACMM level changes. func (m *Manager) SetACMMLevel(level int) { @@ -1002,70 +931,6 @@ func (m *Manager) tmuxPaneHasCLI(session string) bool { return paneHasCLIMarker(m.captureTmuxPane(session)) } -const ( - // consentConfirmFooter appears at the bottom of Claude Code interactive - // selection screens (consent dialogs, settings-error menus). - consentConfirmFooter = "Enter to confirm" - // bypassConsentTitle is the heading of the --dangerously-skip-permissions - // consent screen. Its default selection is "No, exit" — confirming it - // terminates the CLI and leaves a bare bash pane. - bypassConsentTitle = "Bypass Permissions mode" - // bypassConsentDefaultOption is the default (negative) option on the - // bypass-permissions consent screen. - bypassConsentDefaultOption = "No, exit" - // bypassConsentAcceptOption is the affirmative option on the - // bypass-permissions consent screen. Its position varies between CLI - // versions, so acceptance navigates by matching the selected-line text. - bypassConsentAcceptOption = "Yes, I accept" - // apiKeyPromptTitle is the heading of the custom-API-key approval prompt, - // shown when ANTHROPIC_API_KEY is not in customApiKeyResponses.approved. - // Its default selection is "No (recommended)" with the affirmative option - // above it. - apiKeyPromptTitle = "Detected a custom API key" - // apiKeyPromptAcceptOption is the affirmative option on the - // custom-API-key approval prompt. - apiKeyPromptAcceptOption = "Yes" - // cliWorkingMarker is shown while Claude Code is actively processing a - // request; a pane in this state is never a consent screen. - cliWorkingMarker = "esc to interrupt" -) - -// paneShowsConsentScreen reports whether the pane is showing an interactive -// consent/selection screen rather than a ready CLI input prompt. Such screens -// contain a "❯"-selected menu option (e.g. "❯ 1. No, exit"), so they satisfy -// marker-based CLI presence checks ("❯" is also a cliPaneMarkers entry) — a -// kick typed into one is consumed by the menu, or by bash once the default -// "No, exit" selection terminates the CLI. Callers should pass the visible -// pane only (no scrollback): dismissed consent screens linger in history. -func paneShowsConsentScreen(pane string) bool { - if pane == "" || strings.Contains(pane, cliWorkingMarker) { - return false - } - // A known startup-blocking menu is not a ready prompt either. The generic - // test below needs the "Enter to confirm" footer AND a "❯"-marked line; - // codex renders neither (its footer is "Press enter to continue" and its - // marker is "›" U+203A), so its update menu read as READY. Everything that - // gates on readiness — the startup kick, caveman activation — then typed - // into the menu, and the Enter confirmed its pre-selected option: - // "1. Update now", which runs `npm install -g` as the agent UID, fails, and - // kills the CLI. Blocking on these lets the prompt watcher answer them. - if paneHasBlockingPrompt(pane) { - return true - } - if strings.Contains(pane, bypassConsentTitle) && strings.Contains(pane, bypassConsentDefaultOption) { - return true - } - if !strings.Contains(pane, consentConfirmFooter) { - return false - } - for _, line := range strings.Split(pane, "\n") { - if strings.HasPrefix(strings.TrimSpace(line), "❯") { - return true - } - } - return false -} - // copilotGitHubWriteDenyFlags and claudeGitHubWriteDenyFlags are defined together // near the bottom of this file (alongside the codex/bob backend constants). v2 // independently added a copy of copilotGitHubWriteDenyFlags here; the v4 grouped @@ -1115,17 +980,6 @@ var acmmLevelNames = map[int]string{ 6: "Fully Autonomous", } -func (m *Manager) buildBootstrapPrompt(agent *AgentProcess) string { - // No boot prompt — the governor's first eval cycle (10s after startup) - // kicks all due agents via BuildKickMessages with fully substituted - // templates. Sending a boot prompt here caused unsubstituted ${ISSUE_LIST} - // and other vars to reach the agent. The policy-file path list this - // function used to assemble was dead code once the boot prompt was - // removed, so it is gone too. - _ = agent // signature kept for the call site; the arg is no longer read - return "" -} - // findACMMFragments returns paths to ACMM policy files the agent should read. // Order: base.md (shared rules) then l.md (level-specific). var acmmFragmentFallbackDirs = []string{ @@ -1214,75 +1068,6 @@ func (m *Manager) buildProjectPreamble(agent *AgentProcess) string { mode.Emoji(), mode.String(), prPolicy) } -// metricsCachePath is a var (not const) so tests can point it at a temp file -// to exercise readCoveragePreamble without a real /data volume. Production -// value is unchanged. -var metricsCachePath = "/data/metrics/agent-metrics-cache.json" - -func (m *Manager) readCoveragePreamble() string { - data, err := os.ReadFile(metricsCachePath) - if err != nil { - return "" - } - var metrics map[string]map[string]json.Number - if err := json.Unmarshal(data, &metrics); err != nil { - return "" - } - ci, ok := metrics["ci-maintainer"] - if !ok { - return "" - } - cov, err := ci["coverage"].Int64() - if err != nil { - return "" - } - target, err := ci["coverageTarget"].Int64() - if err != nil { - target = 91 - } - return fmt.Sprintf("[COVERAGE] Current: %d%% | Target: %d%%.", cov, target) -} - -// shellEnvVar formats KEY='value' with single-quoting so values containing -// spaces, parentheses, or other shell metacharacters are safe in inline env -// var assignments sent to tmux via send-keys. -func shellEnvVar(key, value string) string { - quoted := strings.ReplaceAll(value, "'", "'\"'\"'") - return fmt.Sprintf("%s='%s'", key, quoted) -} - -// applySecretEnv pushes only the Secret pairs into the agent's tmux session via -// set-environment. Values are passed as exec args (never through a shell), so -// they are not word-split and never land in the pane or in bash history. -// Failures are ignored for the same reason ensureTmuxSession ignores them: a -// missing session is handled by the launch path, not here. -func (m *Manager) applySecretEnv(agent *AgentProcess) { - if agent == nil || agent.tmuxSession == "" { - return - } - for _, p := range m.agentEnvPairs(agent) { - if !p.Secret { - continue - } - _ = m.tmuxCmd(agent, "set-environment", "-t", agent.tmuxSession, p.Key, p.Value).Run() - } -} - -func (m *Manager) buildEnvPrefix(agent *AgentProcess) string { - pairs := m.agentEnvPairs(agent) - var parts []string - for _, p := range pairs { - if p.Secret { - continue - } - parts = append(parts, shellEnvVar(p.Key, p.Value)) - } - if len(parts) == 0 { - return "" - } - return strings.Join(parts, " ") + " " -} - func (m *Manager) pollTmuxOutput(name, session string, buf *RingBuffer, ctx context.Context) { const pollInterval = 3 * time.Second ticker := time.NewTicker(pollInterval) @@ -1332,99 +1117,6 @@ func (m *Manager) pollTmuxOutput(name, session string, buf *RingBuffer, ctx cont } } -// Blocked-action thrash breaker: an agent that keeps hammering a policy wall -// (e.g. a push with no per-agent token, blocked every ~3s by -// git-credential-hive, or a proxy hard-deny) burns model tokens indefinitely -// with zero possible output — observed live 2026-08-04 on a hosted L2 hive -// whose guide agent retried a blocked push every 3 seconds. (Since #4289, -// ADVISORY-mode pushes are no longer blocked by the credential helper — the -// read-only token is served and GitHub rejects the push with 403 — but the -// helper still emits "git push blocked:" for unknown-UID and missing-token -// failures, which this breaker continues to catch.) The hub, not the model, -// breaks the loop: thrashThreshold blocked-action lines within thrashWindow -// pauses the session (visible, reversible, stops governor kicks) with the -// reason spelled out. -const ( - thrashWindow = 60 * time.Second - thrashThreshold = 5 - thrashCooldown = 10 * time.Minute -) - -// blockedActionMarkers are the policy-wall stderr lines that can never -// succeed by retrying. Keep in sync with bin/git-credential-hive.sh and the -// proxy's hard-deny responses. -var blockedActionMarkers = []string{ - "git push blocked:", - "blocked by hive policy", -} - -type thrashState struct { - times []time.Time - lastTrip time.Time -} - -// checkBlockedThrash records a blocked-action output line for the agent and, -// past the threshold, pauses the agent asynchronously (never inline: this is -// called from the output-capture goroutine and Pause takes m.mu). -func (m *Manager) checkBlockedThrash(agent, line string) { - matched := false - for _, marker := range blockedActionMarkers { - if strings.Contains(line, marker) { - matched = true - break - } - } - if !matched { - return - } - now := time.Now() - m.thrashMu.Lock() - if m.thrash == nil { - m.thrash = map[string]*thrashState{} - } - st := m.thrash[agent] - if st == nil { - st = &thrashState{} - m.thrash[agent] = st - } - trip := recordBlockedAndCheck(st, now, thrashWindow, thrashThreshold, thrashCooldown) - m.thrashMu.Unlock() - if !trip { - return - } - reason := fmt.Sprintf("blocked-action loop: %d+ policy-blocked attempts in %s — the block is terminal in this mode; paused to stop token burn", thrashThreshold, thrashWindow) - m.logger.Warn("thrash breaker tripped", "agent", agent, "line", truncateStr(line, 160)) - go func() { - if err := m.Pause(agent, "thrash-breaker", reason); err != nil { - m.logger.Warn("thrash breaker pause failed", "agent", agent, "error", err) - } - }() -} - -// recordBlockedAndCheck is the pure sliding-window decision: append now, drop -// entries older than window, and report whether the threshold is crossed -// outside the cooldown. Split out for direct unit testing. -func recordBlockedAndCheck(st *thrashState, now time.Time, window time.Duration, threshold int, cooldown time.Duration) bool { - st.times = append(st.times, now) - cutoff := now.Add(-window) - kept := st.times[:0] - for _, t := range st.times { - if t.After(cutoff) { - kept = append(kept, t) - } - } - st.times = kept - if len(st.times) < threshold { - return false - } - if !st.lastTrip.IsZero() && now.Sub(st.lastTrip) < cooldown { - return false - } - st.lastTrip = now - st.times = nil - return true -} - // waitForCLIReady polls the tmux pane until the CLI shows its ready prompt // or the timeout expires. Returns true if the CLI became ready. func (m *Manager) waitForCLIReady(session string) bool { @@ -1741,252 +1433,6 @@ func (m *Manager) RemoveAgent(name string) { )) } -// effectiveBackend is the backend this agent will actually launch with: the -// per-agent override when set, otherwise its configured backend. -func (a *AgentProcess) effectiveBackend() string { - if a.BackendOverride != "" { - return a.BackendOverride - } - return a.Config.Backend -} - -// effectiveModel is the model this agent will actually launch with: the -// per-agent override when set, otherwise its configured model. Returns the -// raw (un-normalized) name — the audit log should show what was ASKED for, -// since a bad model name is exactly the kind of misconfiguration being -// audited. -func (a *AgentProcess) effectiveModel() string { - if a.ModelOverride != "" { - return a.ModelOverride - } - return a.Config.Model -} - -// dismissInferencePrompts polls the tmux pane for Claude Code interactive -// prompts and auto-dismisses them. The "Bypass Permissions mode" consent -// screen and the custom-API-key approval prompt are handled first and -// explicitly (see confirmMenuOption): their default selections are negative -// ("No, exit" / "No (recommended)"), so confirming blind terminates the CLI -// or declines the seeded key. -// Other prompts are handled dynamically regardless of prompt text changes -// between Claude Code versions by: -// 1. Detecting "Enter to confirm" (universal prompt footer) -// 2. Finding the selected option (line with "❯" marker) -// 3. If selected option looks negative (contains "No" or "exit"), navigate -// away from it before confirming -// 4. For "Press Enter to continue" screens, just press Enter -// -// The pane is polled fast for the first 10s — the consent screen appears -// within ~5-8s of launch and every second it lingers is a window for a kick -// to be swallowed by the menu — then at a relaxed interval. -// -// Stops when the main Claude Code input prompt appears ("esc to interrupt"). -func (m *Manager) dismissInferencePrompts(agent *AgentProcess) { - const ( - // promptFastPollWindow covers the launch window in which the consent - // screen normally appears (~5-8s after CLI start). - promptFastPollWindow = 10 * time.Second - promptFastPollInterval = 250 * time.Millisecond - promptPollInterval = 1 * time.Second - promptDismissTimeout = 60 * time.Second - postKeystrokeDelay = 500 * time.Millisecond - ) - - start := time.Now() - timeout := promptDismissTimeout - if m.promptDismissTimeout > 0 { - timeout = m.promptDismissTimeout - } - deadline := start.Add(timeout) - lastPane := "" - - for time.Now().Before(deadline) { - interval := promptPollInterval - if time.Since(start) < promptFastPollWindow { - interval = promptFastPollInterval - } - m.sleepDuringPromptDismiss(interval) - - pane := m.captureVisiblePaneForAgent(agent) - if pane == "" { - continue - } - - // Bypass-permissions consent screen: handle first and explicitly, - // even if the pane is unchanged since the last poll (a mistimed - // keystroke must be retried, not skipped). The affirmative option - // sits below the default "No, exit". - if strings.Contains(pane, bypassConsentTitle) && !strings.Contains(pane, cliWorkingMarker) { - m.logger.Info("accepting bypass-permissions consent", "agent", agent.Name) - m.confirmMenuOption(agent, bypassConsentTitle, bypassConsentAcceptOption, "Down") - lastPane = "" // re-capture fresh on the next pass - continue - } - - // Custom-API-key approval prompt: the affirmative "Yes" sits ABOVE - // the default "No (recommended)" selection, so the generic - // Down-then-Enter fallback below would decline it. - if strings.Contains(pane, apiKeyPromptTitle) && !strings.Contains(pane, cliWorkingMarker) { - m.logger.Info("approving seeded inference API key", "agent", agent.Name) - m.confirmMenuOption(agent, apiKeyPromptTitle, apiKeyPromptAcceptOption, "Up") - lastPane = "" - continue - } - - if pane == lastPane { - continue - } - lastPane = pane - - // Main prompt visible — agent is ready - if strings.Contains(pane, "bypass permissions on") || strings.Contains(pane, "esc to interrupt") { - m.logger.Info("inference agent ready", "agent", agent.Name) - return - } - - // "Press Enter to continue" screens - if strings.Contains(pane, "Press Enter to continue") { - m.logger.Info("inference prompt: press enter", "agent", agent.Name) - m.tmuxSendKeysForAgent(agent, "Enter") - continue - } - - // Selection prompts have "Enter to confirm" footer - if !strings.Contains(pane, "Enter to confirm") { - continue - } - - // Find the currently selected option (marked with ❯) - selected := selectedMenuOption(pane) - - m.logger.Info("inference prompt detected", - "agent", agent.Name, - "selected", selected, - ) - - // If current selection looks negative, navigate away from it - selectedLower := strings.ToLower(selected) - if strings.Contains(selectedLower, "no,") || strings.Contains(selectedLower, "no ") || - strings.Contains(selectedLower, "exit") { - // Try moving down first (most prompts put the positive option below) - m.tmuxSendKeysForAgent(agent, "Down") - m.sleepDuringPromptDismiss(postKeystrokeDelay) - } else if strings.Contains(selectedLower, "fix with") { - // Settings error: skip past "Fix with Claude" and "Exit" to "Continue without" - m.tmuxSendKeysForAgent(agent, "Down") - m.sleepDuringPromptDismiss(postKeystrokeDelay) - m.tmuxSendKeysForAgent(agent, "Down") - m.sleepDuringPromptDismiss(postKeystrokeDelay) - } - - m.tmuxSendKeysForAgent(agent, "Enter") - } - - m.logger.Warn("inference prompt dismissal timed out", "agent", agent.Name) -} - -func (m *Manager) sleepDuringPromptDismiss(d time.Duration) { - m.term().Sleep(d) -} - -// selectedMenuOption returns the trimmed text of the "❯"-selected line of an -// interactive CLI menu, or "" if no line is selected. -func selectedMenuOption(pane string) string { - for _, line := range strings.Split(pane, "\n") { - trimmed := strings.TrimSpace(line) - if strings.HasPrefix(trimmed, "❯") { - return trimmed - } - } - return "" -} - -// confirmMenuOption drives an interactive CLI menu identified by title to the -// option whose text contains want, then confirms it with Enter. Navigation -// matches the "❯"-selected line text rather than pressing a fixed number of -// keys, so it lands on the right option whichever position it occupies (menu -// option order differs between Claude CLI versions). navKey is the arrow key -// to step with ("Down" or "Up"). Returns true once the option was confirmed -// or the screen is gone. -func (m *Manager) confirmMenuOption(agent *AgentProcess, title, want, navKey string) bool { - const ( - // menuMaxNavigateSteps bounds arrow-key navigation; the handled menus - // have 2 options, extra headroom covers future variants. - menuMaxNavigateSteps = 4 - postKeystrokeDelay = 500 * time.Millisecond - ) - for step := 0; step < menuMaxNavigateSteps; step++ { - pane := m.captureVisiblePaneForAgent(agent) - if !strings.Contains(pane, title) || strings.Contains(pane, cliWorkingMarker) { - return true // screen already dismissed - } - if strings.Contains(selectedMenuOption(pane), want) { - m.tmuxSendKeysForAgent(agent, "Enter") - m.sleepDuringPromptDismiss(postKeystrokeDelay) - return true - } - m.tmuxSendKeysForAgent(agent, navKey) - m.sleepDuringPromptDismiss(postKeystrokeDelay) - } - m.logger.Warn("inference menu: wanted option not reached", - "agent", agent.Name, "title", title, "want", want) - return false -} - -const ( - // consentStuckGracePeriod is how long a consent screen must stay visible - // across watcher passes before the agent counts as stuck. The launch-time - // dismissal goroutine runs for 60s, so a screen still visible this long - // after first being seen by the watcher means dismissal lost the race. - consentStuckGracePeriod = 30 * time.Second - // consentDismissCooldown is the minimum interval between watcher-triggered - // dismissal passes for one agent, so a stubborn screen can't spam - // keystroke goroutines (each dismissal pass itself polls for 60s). - consentDismissCooldown = 2 * time.Minute -) - -// clearConsentTracking resets the consent-stuck timer for an agent whose pane -// no longer shows a consent screen. -func (m *Manager) clearConsentTracking(name string) { - m.mu.Lock() - defer m.mu.Unlock() - if agent, ok := m.agents[name]; ok { - agent.consentSeenAt = time.Time{} - } -} - -// dismissConsentIfStuck re-runs dismissInferencePrompts for an inference agent -// whose pane has shown a consent screen for longer than the grace period, -// subject to a per-agent cooldown. Called from the watcher loop -// (CheckAndRestartCrashedAgents) so an agent that lands on a consent screen -// after launch — e.g. a crash-recovery restart whose launch-time dismissal -// timed out — recovers instead of sitting stuck while kicks appear to succeed. -func (m *Manager) dismissConsentIfStuck(name string) { - now := time.Now() - m.mu.Lock() - agent, ok := m.agents[name] - if !ok { - m.mu.Unlock() - return - } - if agent.consentSeenAt.IsZero() { - agent.consentSeenAt = now - m.mu.Unlock() - return - } - stuckFor := now.Sub(agent.consentSeenAt) - if stuckFor < consentStuckGracePeriod || now.Sub(agent.lastConsentDismiss) < consentDismissCooldown { - m.mu.Unlock() - return - } - agent.lastConsentDismiss = now - m.mu.Unlock() - - m.logger.Warn("inference agent stuck on consent screen, re-running prompt dismissal", - "name", name, "stuck_seconds", int(stuckFor.Seconds())) - go m.dismissInferencePrompts(agent) -} - func (m *Manager) markProviderErrorLocked(agent *AgentProcess, match providerErrorMatch, now time.Time) time.Duration { // BackendAuth (#6558) is updated on every observation of this match, // independent of the backoff early-return below: an operator watching the @@ -2180,454 +1626,18 @@ func (m *Manager) AllStatuses() map[string]*AgentProcess { return result } -// backendBinaryAliases names the backends whose binary is NOT simply the -// backend name. Only genuine aliases belong here: every other CLI backend is -// derived from config.CLIBackends by identity, and every routable model-gateway -// backend is resolved by Manager.backendBinaryName. Keeping this map to aliases -// only is what makes the accept-then-fail class of bug structurally impossible. -var backendBinaryAliases = map[string]string{ - // pi was previously aliased to "goose", which made every pi-configured - // agent exec the goose CLI instead of pi (the backend launch command - // switch now has a real pi case). pi is a first-class CLI backend - // (config.CLIBackends includes "pi"), so identity mapping applies. -} - -// backendBinaryName maps a config-independent agent backend to the NAME of the -// CLI binary that is exec'd for it, without touching the filesystem. Split out -// from backendBinary so the "every supported backend resolves" invariant can be -// tested without requiring each CLI to be installed on the test machine. -// -// Both canonical lists are derived rather than written out here: -// -// - config.CLIBackends (claude, copilot, goose, codex, pi, bob, aider, gemini) -// each launch a binary of the same name, except for the aliases above. -// - config.InferenceBackends (vllm, llm-d, litellm, watsonx) all launch the -// SAME claude CLI, pointed at hive's local OpenAI-compatible translator via -// ANTHROPIC_BASE_URL — the backend name selects the upstream route, not the -// binary. -// -// Deriving both means a backend added to either list can never again be -// accepted by config.ValidateBackend and then rejected hours later at kick time -// with "unknown backend". Previously only InferenceBackends was derived, so -// codex and aider were valid config values that failed at launch. -func backendBinaryName(backend string) (string, error) { - binaries := make(map[string]string, len(config.CLIBackends)+len(config.InferenceBackends)) - for _, b := range config.CLIBackends { - binaries[b] = b - } - for _, b := range config.InferenceBackends { - binaries[b] = "claude" - } - for backend, binary := range backendBinaryAliases { - binaries[backend] = binary - } - - binary, ok := binaries[backend] - if !ok { - return "", fmt.Errorf("unknown backend: %s", backend) - } - return binary, nil -} - -// backendBinaryName resolves both config-independent backends and live -// configured gateway names. A gateway name validates via Manager.routableBackend, -// so the launch path must use the same predicate and route it through claude. -func (m *Manager) backendBinaryName(backend string) (string, error) { - if binary, err := backendBinaryName(backend); err == nil { - return binary, nil - } - if m != nil && m.routableBackend(backend) { - return "claude", nil - } - return "", fmt.Errorf("unknown backend: %s", backend) -} - -// backendBinary resolves an agent backend to the absolute path of the CLI -// binary that is actually exec'd for it. -func backendBinary(backend string) (string, error) { - binary, err := backendBinaryName(backend) - if err != nil { - return "", err - } - - path, err := exec.LookPath(binary) - if err != nil { - return "", fmt.Errorf("backend %s not found in PATH: %w", backend, err) - } - - return path, nil -} - -func (m *Manager) backendBinary(backend string) (string, error) { - binary, err := m.backendBinaryName(backend) - if err != nil { - return "", err - } - - path, err := exec.LookPath(binary) - if err != nil { - return "", fmt.Errorf("backend %s binary %s not found in PATH: %w", backend, binary, err) - } - - return path, nil -} - -func (m *Manager) backendLaunchFailureMessage(backend string, err error) string { - binary, nameErr := m.backendBinaryName(backend) - if nameErr != nil { - return fmt.Sprintf( - "backend %s did not launch: %v. This backend is not a supported CLI, built-in inference backend, or configured model gateway; switch this agent to a supported backend or configure a matching model gateway.", - backend, err) - } - return fmt.Sprintf( - "backend %s did not launch: %v. The %s CLI required for this backend is not installed in this hive image — upgrade the hive image or switch this agent to a different backend.", - backend, err, binary) -} - -const ( - sharedConfigDesiredMode = 0o660 - // agyDefaultEffort is the reasoning effort passed alongside agy's --model - // when the agent has no usable reasoning_effort configured (see - // agyLaunchEffort). agy requires --effort whenever --model is given and - // otherwise ignores the model entirely; "low" is the effort agy defaults - // to on its own, so this makes the configured model take effect without - // changing behaviour. - agyDefaultEffort = "low" - - tokenRestartCooldownSec = 60 // minimum seconds between token-triggered restarts per agent - // loginPromptTailLines bounds the pane region the login-prompt detector - // reads: a prompt the CLI is stuck at sits at the pane bottom, while - // echoed kick text and startup flashes live in scrollback (see the poller). - loginPromptTailLines = 15 - // loginStreakRestartMin is how many consecutive polls (~3s apart) must see - // the login prompt before a token-triggered restart may fire — filters the - // CLI's transient startup "/login" flash. - loginStreakRestartMin = 3 - // tokenRestartMaxAttempts bounds CONSECUTIVE token-triggered restarts that - // fail to clear the login prompt. - // - // The three guards above answer WHEN to restart; none of them answered HOW - // MANY TIMES, so a restart that could never work was retried forever at the - // cooldown interval. #4596 is precisely that shape: the shared credential is - // valid (so configHasTokens() is true) while $HOME/.claude.json has lost its - // oauthAccount (so the CLI shows the login menu regardless), and each - // restart re-launched a CLI that rewrote the same contended file and asked - // again. Restarts are not free — they destroy in-flight work, which is the - // failure the kick grace above was added for. - // - // Three is deliberately generous: one restart genuinely does fix the case - // this feature was built for (an operator authenticates in one agent's - // terminal and the others need a nudge), so the cap only engages on a - // theory that has now failed repeatedly. - tokenRestartMaxAttempts = 3 - // tokenRestartKickGrace suppresses token-triggered restarts after a kick - // delivery so the restart can never destroy just-delivered work. - tokenRestartKickGrace = 10 * time.Minute - expiredTokenHangTimeoutSec = 180 // blank pane after this many seconds triggers token purge + restart - tlsErrorRestartCooldownSec = 120 // minimum seconds between TLS-error-triggered restarts per agent -) - // loginPromptPatterns are substrings that indicate an agent is stuck on a -// codexBackend is the backend name for the OpenAI Codex CLI. -const codexBackend = "codex" - -// bobBackend is the backend name for the IBM bobshell ("bob") CLI. -const bobBackend = "bob" - -// normalizeModelName converts YAML-friendly model names to the format each -// CLI backend expects. Claude CLI uses hyphens (claude-opus-4-7), while -// gemini/goose/agy-style backends use dots (claude-opus-4.7). -// -// copilot does NOT take the blind trailing-digits dot-rewrite below: the -// Copilot CLI's --model nomenclature mixes separators per model family -// (claude-fable-5 is DASHED, claude-opus-4.6 is DOTTED), so the rewrite -// corrupted every dashed-family id — verified live, copilot CLI v1.0.78 -// rejected the rewritten `claude-fable.5` ("is not available") and fell back -// to a different model (#4262). copilot instead uses the alias-based -// CanonicalizeCopilotModel (copilot_models.go), which normalizes separator -// drift against the known CLI-accepted list in both directions and passes -// unknown ids through verbatim. Applied here — at launch time — so an -// already-stored bad id self-corrects on existing spokes without operator -// action. -// -// Self-hosted inference backends (vllm, llm-d, litellm) and configured gateway -// names are the outbound gateway model id verbatim — the string must match an -// entitled model on the gateway EXACTLY (prefixes like "Azure/", dots vs -// hyphens, case). Rewriting it (e.g. "Azure/gpt-4" -> "Azure/gpt.4", -// "gpt-4o-2024-08-06" -> "gpt-4o-2024-08.06") produces a model the team is not -// entitled to and the gateway 403s ("team not allowed to access model") even -// for entitled models. So never normalize inference model names — pass them -// through untouched. -// -// bob is likewise excluded. bobLaunchCmd passes no --model at all (bob -// auto-selects), so this is defense-in-depth rather than the fix: the value is -// still computed and logged on the bob launch path, and the dot-rewrite is -// what turned a configured `claude-sonnet-4-6` into the unknown -// `claude-sonnet-4.6` that made bob die with "Cannot read properties of -// undefined (reading 'maxTokens')". Leaving it unrewritten keeps logs honest -// about what was configured and stops the corrupted id from being handed to a -// future bob consumer. -func normalizeModelNameForBackend(model, backend string, inferenceRoutable bool) string { - if backend == "claude" || backend == bobBackend || inferenceRoutable { - return model - } - if backend == "copilot" { - return CanonicalizeCopilotModel(model) - } - idx := strings.LastIndex(model, "-") - if idx < 0 || idx == len(model)-1 { - return model - } - suffix := model[idx+1:] - allDigits := true - for _, c := range suffix { - if c < '0' || c > '9' { - allDigits = false - break - } - } - if allDigits { - return model[:idx] + "." + suffix - } - return model -} - func normalizeModelName(model, backend string) string { return normalizeModelNameForBackend(model, backend, IsInferenceBackend(backend)) } -// ClearModeOverrides clears Config.Mode for the NAMED agents so that -// DefaultAgentMode determines their mode from the ACMM level. Call it before -// SyncModeFiles when applying a pack, because a pack agent's Config.Mode may -// have been set by a previous level's pack and would otherwise override the -// new level's expected default. -// -// Scoped to a name list — the pack's roster — on purpose (#7503). The previous -// ClearAllModeOverrides wiped every agent in the process table, including -// agents no pack manages. Their Mode was never pack-seeded, so there is no -// stale pack value to clear: what got cleared was the OPERATOR's setting. On -// the projectbluefin spoke `reviewer` was `mode: ADVISORY` in every config -// layer and ran as ISSUES_AND_PRS, the L5 default — two rungs more authority -// than anything on disk granted it — because the startup pack apply cleared it -// here and SyncModeFiles then wrote the default. Names not in the process -// table are ignored. -func (m *Manager) ClearModeOverrides(names []string) { - m.mu.Lock() - defer m.mu.Unlock() - for _, name := range names { - if agent, ok := m.agents[name]; ok { - agent.Config.Mode = "" - } - } -} - -// SyncModeFiles rewrites /tmp/.hive-mode-* for all running agents to reflect the given ACMM level. -func (m *Manager) SyncModeFiles(level int) { - m.mu.RLock() - defer m.mu.RUnlock() - for name, agent := range m.agents { - if agent.Paused { - continue - } - mode := DefaultAgentMode(name, level) - // converseConfigured is logged on both branches below so one grep by - // agent name shows whether the capability came from config or fell - // back to the default alongside the mode decision (#7503). - converseConfigured := agent.Config.Converse != nil - if modeStr := agent.Config.Mode; modeStr != "" { - if parsed, ok := ParseAgentMode(modeStr); ok { - m.logger.Info("SyncModeFiles: Config.Mode override", - "agent", name, "level", level, - "default", DefaultAgentMode(name, level).String(), - "override", modeStr, - "converse_configured", converseConfigured) - mode = parsed - } - } else { - // Log the fallback too, not only the override (#7503). Before this, - // an agent whose configured mode had been dropped on the way to the - // process table was indistinguishable from one that never set a - // mode: the only signal was reading /tmp/.hive-mode- and - // comparing by hand. One line saying "config said nothing, using - // the level default" turns that into a grep. `overlay_file` names - // the per-agent file when the entry came from one, so an operator - // can check what it says against what is being used. - m.logger.Info("SyncModeFiles: no Config.Mode, using level default", - "agent", name, "level", level, - "default", mode.String(), - "converse_configured", converseConfigured, - "overlay_file", agent.Config.SourceFile()) - } - modeFile := filepath.Join(agentStateDir, ".hive-mode-"+name) - if err := writeAgentStateFile(modeFile, []byte(mode.String())); err != nil { - m.logger.Warn("SyncModeFiles: write failed", "file", modeFile, "error", err) - } - // The capability file rides the same sync (#4492). It is level-independent - // today, but writing it here is what makes a `converse` change take effect - // on the next reconcile instead of only at the next agent launch. - caps := DefaultCapabilities(mode, level) - if agent.Config.Converse != nil { - caps.Converse = *agent.Config.Converse - } - m.writeAgentCapsFile(name, caps) - } -} - -// agentCapabilities returns the ORTHOGONAL capabilities for a given agent -// (#4492). Unlike agentMode there is no per-level default table: `converse` is -// opt-in everywhere, so an agent whose config says nothing gets the zero value -// and behaves exactly as it did before capabilities existed. -func (m *Manager) agentCapabilities(agent *AgentProcess) AgentCapabilities { - caps := DefaultCapabilities(m.agentMode(agent), m.project.ACMMLevel) - if agent.Config.Converse != nil { - caps.Converse = *agent.Config.Converse - } - return caps -} - -// writeAgentCapsFile persists the capability set the proxy reads on the request -// path. It is written for EVERY agent, including those with no capabilities, so -// a cleared `converse` actually revokes: leaving a stale file behind would keep -// granting the capability after the operator turned it off. -func (m *Manager) writeAgentCapsFile(name string, caps AgentCapabilities) { - capsFile := filepath.Join(agentStateDir, ".hive-caps-"+name) - if err := writeAgentStateFile(capsFile, []byte(caps.String())); err != nil { - m.logger.Warn("caps file write failed", "file", capsFile, "error", err) - } -} - -// agentMode returns the GitHub interaction mode for a given agent at the current ACMM level. -// If the agent has an explicit Mode in its config (hive.yaml or pack YAML), that takes precedence. -// Otherwise, the default table by ACMM level is used. -func (m *Manager) agentMode(agent *AgentProcess) AgentMode { - if modeStr := agent.Config.Mode; modeStr != "" { - if parsed, ok := ParseAgentMode(modeStr); ok { - return parsed - } - } - return DefaultAgentMode(agent.Name, m.project.ACMMLevel) -} - -// DefaultAgentMode returns the default mode for a given agent name and ACMM level, -// ignoring any hive.yaml override. Used by the dashboard to show "(default)" indicators. -func DefaultAgentMode(agentName string, level int) AgentMode { - if agentName == "supervisor" { - return ModeAdvisory - } - switch level { - case 1: - return ModeAdvisory - case 2: - return ModeAdvisory - case 3: - if agentName == "quality" { - return ModeIssuesAndPRs - } - return ModeAdvisory - case 4: - switch agentName { - case "quality", "sec-check", "ci-maintainer": - return ModeIssuesAndPRs - case "scanner", "guide": - return ModeIssuesOnly - default: - return ModeAdvisory - } - case 5: - return ModeIssuesAndPRs - case 6: - if agentName == "scanner" { - return ModeIssuesPRsMerge - } - return ModeIssuesAndPRs - default: - return ModeAdvisory - } -} - // agentCanWrite returns true if this agent is allowed to push branches and create PRs. // Deprecated: use agentMode() for granular mode checks. func (m *Manager) agentCanWrite(agent *AgentProcess) bool { return m.agentMode(agent).CanPush() } -// AuthorizePROpen enforces the policy for the hive-opens-PR watcher: an agent -// may open a PR (by dropping a request file) only if BOTH hold: -// -// 1. Forge-resistance — the request file's owning UID (fileUID) maps to the -// agent it claims to be (via the uid-map). One agent cannot open a PR "as" -// another, and a non-agent process (unknown UID) is refused. When per-agent -// UIDs are not in play (fileUID <= 0, e.g. shared-dev-UID mode with no map), -// ownership is unverifiable, so we fall back to the ACMM check alone rather -// than hard-failing — the same posture the credential helper takes. -// 2. ACMM write-gate — the agent must be push-capable at the hive's current -// ACMM level, i.e. exactly the CanPush() check that governs `gh pr create`. -// -// Returns nil to authorize, or an error describing the denial. This mirrors the -// direct PR path's policy so the request-file route grants no extra privilege. -func (m *Manager) AuthorizePROpen(agentName string, fileUID int) error { - if strings.TrimSpace(agentName) == "" { - return fmt.Errorf("no agent named in the request") - } - // Forge check: when we have a UID map and a real owning UID, the file owner - // must BE this agent. - if m.uidMap != nil && fileUID > 0 { - owner := m.uidMap.LookupByUID(fileUID) - if owner == "" { - return fmt.Errorf("request file owned by unknown uid %d (not a registered agent)", fileUID) - } - if owner != agentName { - return fmt.Errorf("request claims agent %q but file is owned by agent %q (uid %d)", agentName, owner, fileUID) - } - } - // ACMM write-gate: resolve the agent and check CanPush. - m.mu.RLock() - agent := m.agents[agentName] - m.mu.RUnlock() - if agent == nil { - return fmt.Errorf("unknown agent %q", agentName) - } - if !m.agentMode(agent).CanPush() { - return fmt.Errorf("agent %q is not push-capable at this ACMM level (mode %s) — advisory agents may not open PRs", - agentName, m.agentMode(agent).String()) - } - return nil -} - -// AuthorizeIssueOpen enforces the policy for the issue-request watcher, -// mirroring AuthorizePROpen with the mode gates that govern the direct gh -// paths: "issue" requests need CanCreateIssues() (mode >= ISSUES_ONLY); -// "comment" and "claim" requests need the same (commenting and claiming an -// issue are both issue-writes under the same tier). The same UID -// forge-resistance applies: the request file's owner must BE the claimed -// agent. A nil manager or unknown agent is denied. -func (m *Manager) AuthorizeIssueOpen(agentName string, fileUID int, kind string) error { - if strings.TrimSpace(agentName) == "" { - return fmt.Errorf("no agent named in the request") - } - if m.uidMap != nil && fileUID > 0 { - owner := m.uidMap.LookupByUID(fileUID) - if owner == "" { - return fmt.Errorf("request file owned by unknown uid %d (not a registered agent)", fileUID) - } - if owner != agentName { - return fmt.Errorf("request claims agent %q but file is owned by agent %q (uid %d)", agentName, owner, fileUID) - } - } - m.mu.RLock() - agent := m.agents[agentName] - m.mu.RUnlock() - if agent == nil { - return fmt.Errorf("unknown agent %q", agentName) - } - if !m.agentMode(agent).CanCreateIssues() { - return fmt.Errorf("agent %q may not create issues or comments at this ACMM level (mode %s)", - agentName, m.agentMode(agent).String()) - } - return nil -} - // AuthorizeReviewRequest enforces the policy for the review-request watcher. // It mirrors AuthorizePROpen's forge-resistance, but grants on capability as // well as mode (hivecommons/hive#7485). @@ -2681,128 +1691,6 @@ func (m *Manager) AuthorizeReviewRequest(agentName string, fileUID int) error { return nil } -// AuthorizeMerge enforces the policy for the hive-merges-PR watcher, mirroring -// AuthorizePROpen but with the stricter CanMerge() gate: the request's agent -// must own the request file (forge-resistance) AND be merge-capable at the -// hive's current ACMM level (ModeIssuesPRsMerge). This keeps the file-based -// merge relay under the exact same authority as a direct merge would require — -// an issues/PRs agent that can open PRs still cannot merge them unless its mode -// grants merge. A nil manager or unknown agent is denied. -func (m *Manager) AuthorizeMerge(agentName string, fileUID int) error { - if strings.TrimSpace(agentName) == "" { - return fmt.Errorf("no agent named in the request") - } - // Forge check: when we have a UID map and a real owning UID, the file owner - // must BE this agent. - if m.uidMap != nil && fileUID > 0 { - owner := m.uidMap.LookupByUID(fileUID) - if owner == "" { - return fmt.Errorf("request file owned by unknown uid %d (not a registered agent)", fileUID) - } - if owner != agentName { - return fmt.Errorf("request claims agent %q but file is owned by agent %q (uid %d)", agentName, owner, fileUID) - } - } - // ACMM merge-gate: resolve the agent and check CanMerge. - m.mu.RLock() - agent := m.agents[agentName] - m.mu.RUnlock() - if agent == nil { - return fmt.Errorf("unknown agent %q", agentName) - } - if !m.agentMode(agent).CanMerge() { - return fmt.Errorf("agent %q is not merge-capable at this ACMM level (mode %s) — only ISSUES_PRS_MERGE agents may merge PRs", - agentName, m.agentMode(agent).String()) - } - return nil -} - -// AgentCapabilities reports whether the named agent is ABLE — at the hive's -// current ACMM level and the agent's effective mode — to create issues, open -// PRs, and merge PRs. These are the EXACT gates AuthorizePROpen (CanPush) and -// AuthorizeMerge (CanMerge) enforce, so a hub capability badge derived from -// these can never claim a capability the merge/PR relay would actually refuse. -// ok=false when the agent is unknown to the manager (the caller then reports -// "unknown", not a false negative). Read-only under RLock. -func (m *Manager) AgentCapabilities(agentName string) (canOpenIssue, canOpenPR, canMerge, ok bool) { - m.mu.RLock() - agent, exists := m.agents[agentName] - m.mu.RUnlock() - if !exists || agent == nil { - return false, false, false, false - } - mode := m.agentMode(agent) - return mode.CanCreateIssues(), mode.CanPush(), mode.CanMerge(), true -} - -// EffectiveBackend reports the named agent's effective backend, honoring any -// runtime BackendOverride (see effectiveBackend). ok=false when the agent is -// unknown. Read-only under RLock — a small exported wrapper so callers outside -// the package (the heartbeat builder) need not reach into unexported state. -func (m *Manager) EffectiveBackend(agentName string) (backend string, ok bool) { - m.mu.RLock() - agent, exists := m.agents[agentName] - m.mu.RUnlock() - if !exists || agent == nil { - return "", false - } - return effectiveBackend(agent), true -} - -// InvocationMetadata reports the effective backend, model, and reasoning effort -// the hive invokes for the named agent, accounting for runtime overrides — the -// launch-time truth the invocation-attribution trail records (see pkg/github/attribution -// .go). ok=false when the agent is unknown to the manager (the caller then -// falls back to static config). Read-only under RLock; called from the -// PR-request watcher goroutine, never from the launch path. -func (m *Manager) InvocationMetadata(agentName string) (backend, model, effort string, ok bool) { - m.mu.RLock() - defer m.mu.RUnlock() - agent, exists := m.agents[agentName] - if !exists { - return "", "", "", false - } - backend = effectiveBackend(agent) - model = agent.Config.Model - if agent.ModelOverride != "" { - model = agent.ModelOverride - } - return backend, model, ResolveReasoningEffort(backend, model, agent.Config.ReasoningEffort), true -} - -// ResolveReasoningEffort reports the reasoning effort the hive actually launches -// a given backend/model pair with, given the agent's configured reasoning_effort. -// Exported because the attribution trail is -// resolved in TWO places — Manager.InvocationMetadata above for a running agent, -// and cmd/hive's fallback that reads straight from config when the Manager does -// not know the agent — and both must give the same answer. -// -// Before this existed the fallback carried its own hardcoded "low", so changing -// agyDefaultEffort here would have left cmd/hive silently stamping PRs with an -// effort agy was no longer being launched with. An attribution trail that -// misreports is worse than one that says nothing. -// -// The rules mirror the launch path exactly: -// - agy REQUIRES --effort whenever --model is given, so with a model it runs -// at agyLaunchEffort(configured) and with no model at no effort at all. -// - codex is launched with `-c model_reasoning_effort` only when an effort -// is configured; unset means codex's own default, which the hive does not -// resolve, so the honest answer is the configured value verbatim. -// - every other backend takes its effort from its own config, which the -// hive does not resolve here, so the honest answer is "". -func ResolveReasoningEffort(backend, model, configured string) string { - switch backend { - case "agy": - if model != "" { - return agyLaunchEffort(configured) - } - return "" - case codexBackend: - return configured - } - return "" -} - // filteredEnv returns os.Environ() with write-capable tokens removed for advisory agents. // COPILOT_GITHUB_TOKEN is kept for all agents (needed for AI auth); write access is // gated by --enable-all-github-mcp-tools flag. GH_TOKEN and GITHUB_TOKEN are stripped @@ -2823,522 +1711,6 @@ func (m *Manager) filteredEnv(agent *AgentProcess) []string { return filtered } -// embeddedTokenRe matches git remote URLs with embedded credentials: -// https://x-access-token:TOKEN@github.com/org/repo.git -var embeddedTokenRe = regexp.MustCompile(`^https://[^@]+@(github\.com/.+)$`) - -// sanitizeGitRemotes strips embedded tokens from git remote URLs in all repos -// under the agent's work directory. Copilot CLI embeds the GitHub App token -// directly in the remote URL when it clones, bypassing both the credential -// helper (Layer 1) and env var filtering (Layer 2). -func (m *Manager) sanitizeGitRemotes(agent *AgentProcess) { - if m.agentMode(agent).CanPush() { - return - } - agentDir := m.workDir + "/" + agent.Name - _ = filepath.WalkDir(agentDir, func(path string, d os.DirEntry, err error) error { - if err != nil || d.Name() != ".git" || !d.IsDir() { - return nil - } - repoDir := filepath.Dir(path) - out, err := exec.Command("git", "-C", repoDir, "remote", "get-url", "origin").Output() - if err != nil { - return filepath.SkipDir - } - url := strings.TrimSpace(string(out)) - if match := embeddedTokenRe.FindStringSubmatch(url); match != nil { - clean := "https://" + match[1] - _ = exec.Command("git", "-C", repoDir, "remote", "set-url", "origin", clean).Run() - m.logger.Info("stripped embedded token from git remote", - "agent", agent.Name, "repo", repoDir) - } - return filepath.SkipDir - }) -} - -// agentEnvPair is an unquoted key-value environment variable. -type agentEnvPair struct { - Key string - Value string - // Secret vars are set via tmux set-environment only, never on the command line. - Secret bool -} - -// inferenceQuietCLIEnv is the set of Claude CLI switches exported to -// inference-routed sessions so the CLI stops emitting non-inference traffic -// (telemetry, error reporting, nonessential lookups) to its Anthropic host. -var inferenceQuietCLIEnv = []string{ - "DISABLE_TELEMETRY", - "DISABLE_ERROR_REPORTING", - "CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC", -} - -func (m *Manager) agentEnvPairs(agent *AgentProcess) []agentEnvPair { - model := agent.Config.Model - if agent.ModelOverride != "" { - model = agent.ModelOverride - } - backend := agent.Config.Backend - if agent.BackendOverride != "" { - backend = agent.BackendOverride - } - displayName := agent.Config.DisplayName - if displayName == "" { - displayName = agent.Name - } - vars := []agentEnvPair{ - {"HIVE_AGENT", agent.Name, false}, - {"HIVE_AGENT_DISPLAY_NAME", displayName, false}, - {"HIVE_BACKEND", backend, false}, - {"HIVE_MODEL", model, false}, - } - if hiveID := os.Getenv("HIVE_ID"); hiveID != "" { - vars = append(vars, agentEnvPair{"HIVE_ID", hiveID, false}) - } - vars = append(vars, agentEnvPair{"HIVE_ACMM_LEVEL", fmt.Sprintf("%d", m.project.ACMMLevel), false}) - - mode := m.agentMode(agent) - if agent.Config.Tools != nil { - if effectiveMode := agent.Config.Tools.EffectiveMode(); effectiveMode != "" { - vars = append(vars, agentEnvPair{"HIVE_AGENT_MODE", effectiveMode, false}) - } else { - vars = append(vars, agentEnvPair{"HIVE_AGENT_MODE", mode.String(), false}) - } - } else { - vars = append(vars, agentEnvPair{"HIVE_AGENT_MODE", mode.String(), false}) - } - modeFile := filepath.Join(agentStateDir, ".hive-mode-"+agent.Name) - if err := writeAgentStateFile(modeFile, []byte(mode.String())); err != nil { - m.logger.Warn("agentBootstrapEnv: mode file write failed", "file", modeFile, "error", err) - } - m.writeAgentCapsFile(agent.Name, m.agentCapabilities(agent)) - // Plain proxy URL without userinfo — Claude Code's native binary fails - // to open a socket when the URL contains username:password@ (FailedToOpenSocket). - // Agent identification uses UID-based /proc/net/tcp lookup instead of - // Proxy-Authorization headers. GIT_TERMINAL_PROMPT=0 prevents git from - // prompting for proxy credentials. - proxyURL := fmt.Sprintf("http://127.0.0.1:%d", proxyListenPort) - vars = append(vars, agentEnvPair{"HTTPS_PROXY", proxyURL, false}) - vars = append(vars, agentEnvPair{"HTTP_PROXY", proxyURL, false}) - vars = append(vars, agentEnvPair{"HIVE_PROXY_AGENT", agent.Name, false}) - vars = append(vars, agentEnvPair{"GIT_TERMINAL_PROMPT", "0", false}) - vars = append(vars, agentEnvPair{"NODE_EXTRA_CA_CERTS", proxyCACertPath, false}) - if sha := os.Getenv("HIVE_SHA"); sha != "" { - vars = append(vars, agentEnvPair{"HIVE_SHA", sha, false}) - } - if advisory := os.Getenv("HIVE_ADVISORY_ISSUE"); advisory != "" { - vars = append(vars, agentEnvPair{"HIVE_ADVISORY_ISSUE", advisory, false}) - } - // HIVE_REPO / HIVE_REPOS: the shipped policy templates instruct agents to - // run `gh issue create --repo "$HIVE_REPO"`, but nothing ever exported it - // to hosted agents (only the OSS scheduler set a hardcoded "/hive"). - // Root-caused on a live hosted hive (2026-08-20): the sec-check agent saw - // HIVE_REPO unset, fell back to the git remote of its own workdir, and - // silently scanned only the primary repo — the other project repos were - // never touched. Export the primary repo and the full project repo list so - // templates and agents can target every configured repo. - if m.project.Org != "" && len(m.project.Repos) != 0 { - primary := m.project.PrimaryRepo() - if primary == "" { - primary = m.project.Repos[0] - } - vars = append(vars, agentEnvPair{"HIVE_REPO", m.project.Org + "/" + primary, false}) - full := make([]string, len(m.project.Repos)) - for i, r := range m.project.Repos { - full[i] = m.project.Org + "/" + r - } - vars = append(vars, agentEnvPair{"HIVE_REPOS", strings.Join(full, ","), false}) - } - // GH_HOST: point the gh CLI at the configured forge host for GHE spokes. - // See ProjectContext.GHHost. The gh wrapper pairs this with - // GH_ENTERPRISE_TOKEN so the per-agent scoped token authenticates there. - if m.project.GHHost != "" { - vars = append(vars, agentEnvPair{"GH_HOST", m.project.GHHost, false}) - } - if m.routableBackend(backend) { - const inferenceTranslatePort = 18444 - vars = append(vars, agentEnvPair{"ANTHROPIC_API_KEY", "sk-hive-" + agent.Name, false}) - baseURL := fmt.Sprintf("http://127.0.0.1:%d", inferenceTranslatePort) - vars = append(vars, agentEnvPair{"ANTHROPIC_BASE_URL", baseURL, false}) - vars = append(vars, agentEnvPair{"NO_PROXY", "127.0.0.1,localhost", false}) - // Cap the CLI output-token budget at a value every commercial model - // litellm may front will accept. A prior 128000 (chosen so verbose - // OSS models would not truncate) exceeds Azure GPT-4o's 16384 - // completion-token cap, so every request 400s with - // "max_tokens is too large: 128000. This model supports at most - // 16384 completion tokens". See inferenceMaxOutputTokensDefault. - // TODO: the gateway 400 body names the model's real cap ("supports - // at most N completion tokens"); a future enhancement could parse it - // to auto-adjust per-model instead of using a universal floor. - vars = append(vars, agentEnvPair{"CLAUDE_CODE_MAX_OUTPUT_TOKENS", strconv.Itoa(inferenceMaxOutputTokensDefault), false}) - // The Claude CLI sends telemetry batches, error reports, and other - // non-inference traffic to its configured Anthropic host. Routed at - // an OpenAI-compatible gateway that traffic has nowhere useful to go - // (the proxy now answers it locally rather than forwarding it — see - // classifyInferencePath), so switch it off at the source. Only for - // inference-routed sessions: subscription/Anthropic-direct sessions - // keep Anthropic's own telemetry. - for _, v := range inferenceQuietCLIEnv { - vars = append(vars, agentEnvPair{v, "1", false}) - } - } - if m.copilotAuthToken != "" { - vars = append(vars, agentEnvPair{copilotTokenEnvVar, m.copilotAuthToken, true}) - } - // Point the GitHub MCP server at the App installation token so PRs, issue - // comments, and merges are authored by the App bot ("[bot]") — NOT by - // the Copilot login user. COPILOT_GITHUB_TOKEN above stays as the Copilot - // OAuth token because it authenticates the AI model (a separate concern from - // GitHub write identity); leaving it untouched keeps the Copilot CLI login - // working. The Copilot CLI reads GH_TOKEN / GITHUB_TOKEN for GitHub API auth - // (per its README: GH_TOKEN or GITHUB_TOKEN, in that precedence), so setting - // GITHUB_TOKEN here makes the built-in GitHub MCP server act as the App bot. - // - // Gated on the opt-in flag first (default OFF → no behavior change on any - // hive that has not explicitly enabled App-bot authorship), then on CanPush(): - // advisory agents are deliberately kept GITHUB_TOKEN-less (see the -u - // GITHUB_TOKEN strip after the env loop) so they cannot write; only push- - // capable tiers — the ones that legitimately open/merge PRs — get the App - // token. m.appAuth != nil means an App is configured. The value is the - // per-agent tier-SCOPED App token, and refreshAgentTokens re-pushes it hourly - // so it never goes stale. - if m.project.AppAuthoredPRs && m.appAuth != nil && agent.UID > 0 && m.agentMode(agent).CanPush() { - if data, err := os.ReadFile(ghpkg.AgentTokenCachePath(agent.Name)); err == nil { - if tok := strings.TrimSpace(string(data)); tok != "" { - vars = append(vars, agentEnvPair{"GITHUB_TOKEN", tok, true}) - } - } - } - // Linear write credential for ISSUES_ONLY+ agents — see linearEnvPairs. - // Nil for advisory agents and for hives with no Linear credential, so a - // GitHub-only hive sees no change. - vars = append(vars, m.linearEnvPairs(agent)...) - // CLAUDE_CODE_OAUTH_TOKEN is a LAST RESORT, not the normal delivery path. - // - // Claude Code treats this variable as a static bearer token: when it is - // set the CLI uses it verbatim, never opens ~/.claude/.credentials.json, - // and therefore never refreshes. Measured in-container (2026-09-01): with - // the variable set to a bad value and a perfectly good credentials file - // beside it, the CLI answered "401 OAuth access token is invalid" — there - // is no fallback to the file. - // - // m.claudeAuthToken is a snapshot of the SHORT-LIVED access token, taken - // once at manager construction and refreshed only by ReloadClaudeToken() - // after a dashboard login. Injecting it therefore pinned every claude - // agent to the remaining life of whatever access token happened to be on - // disk when the container started — Claude access tokens live 8h, so the - // whole fleet 401'd within a day of every restart and the only recovery - // hive offered was an operator re-login, once per agent. That is the daily - // re-authentication treadmill of #5454. - // - // It is also unnecessary since per-agent homes (#4619): every agent's - // ~/.claude is a symlink to the shared /data/home/.claude, so the CLI can - // read the credential itself — and redeem its refresh grant on start, - // which is the one thing the env var makes impossible. - // - // So inject ONLY when the agent has no credential file it can read. That - // keeps the variable doing the job it was added for (#c5648bc9: deliver a - // dashboard-obtained token to an agent that cannot see the file) and stops - // it overriding a credential that can still refresh itself. - if m.claudeAuthToken != "" && backend == "claude" && !claudeCredentialReachable(agent, backend) { - vars = append(vars, agentEnvPair{"CLAUDE_CODE_OAUTH_TOKEN", m.claudeAuthToken, true}) - } - // bob reads its key from BOBSHELL_API_KEY. Secret: true keeps the value off - // the shell command line (out of `ps`, bash history, and pane scrollback); - // it reaches the CLI via tmux set-environment only. Gated on the backend so - // no other CLI's environment carries an IBM credential it has no use for. - if backend == bobBackend { - if key := m.bobAPIKey(); key != "" { - vars = append(vars, agentEnvPair{config.BobAPIKeyEnvVar, key, true}) - } - // BOBSHELL_DEFAULT_AUTH_TYPE is what actually selects API-key auth; - // without it bob defaults to W3ID SSO and parks at the interactive key - // prompt forever. Deliberately NOT Secret: the value is the literal - // non-credential string "api-key", and secret pairs only reach a - // freshly-created pane shell via tmux set-environment, whereas - // non-secret pairs are re-applied on EVERY launch through - // buildEnvPrefix. That asymmetry is exactly what caused the sibling - // bug fixed in #2228, so the auth type must ride the always-reapplied - // path or a relaunch into an existing session loses it. - vars = append(vars, agentEnvPair{config.BobAuthTypeEnvVar, config.BobAuthTypeAPIKey, false}) - } - // BD_DIR tells the `bd` CLI where to read/write beads. Without this, - // bd falls back to cwd (/data/agents/) instead of the configured - // beads_dir (/data/beads/), causing a path mismatch with the - // dashboard and advisory digest. - if agent.Config.BeadsDir != "" { - vars = append(vars, agentEnvPair{"BD_DIR", agent.Config.BeadsDir, false}) - } - if agent.Config.CavemanMode != "" { - vars = append(vars, agentEnvPair{"HIVE_CAVEMAN_MODE", agent.Config.CavemanMode, false}) - } - // Export the RESOLVED explain mode, not the raw config value, so an agent's - // skills and helper scripts see the same answer the kick suffix acted on - // (including inheritance from the hive-wide default and the off fallback for - // an invalid value). Always exported, off included, so a script can branch on - // it without having to re-derive the precedence rules itself. - vars = append(vars, agentEnvPair{config.ExplainModeEnvVar, resolveExplainMode(agent.Config, m.explainModeDefault()), false}) - // GIT_SSL_CAINFO only — NOT SSL_CERT_FILE (that breaks Copilot API TLS) - vars = append(vars, agentEnvPair{"GIT_SSL_CAINFO", proxyCACertPath, false}) - if agent.UID > 0 { - vars = append(vars, agentEnvPair{"HIVE_AGENT_TOKEN_CACHE", ghpkg.AgentTokenCachePath(agent.Name), false}) - } - if agent.UID > 0 { - // Per-UID agents get a per-agent HOME (#4596) — AgentHome is the single - // source of truth so the auth probe and this export can never diverge. - vars = append(vars, agentEnvPair{"HOME", AgentHome(agent.Name, agent.UID, backend), false}) - - // Per-agent XDG data/state roots (#6238), beneath the per-agent HOME. - // Every backend CLI keeps its session transcripts, run locks and - // caches under $XDG_DATA_HOME / $XDG_STATE_HOME; with the legacy - // shared /data/home/.local those were one contended tree owned by - // whichever agent wrote first. Exported explicitly (not left to the - // spec default under $HOME) so the answer cannot depend on how each - // CLI resolves XDG, and only when HOME itself is per-agent — the - // HIVE_SHARED_AGENT_HOME=1 escape hatch keeps the legacy layout whole. - // XDG_CONFIG_HOME is deliberately NOT set: ~/.config stays the shared - // credential/config bridge (gh hosts.yml, goose config.yaml). See - // setupAgentXDGDirs, which pre-creates these as the agent's own dirs. - if xdgHome, ok := perAgentXDGHome(agent.Name, agent.UID, backend); ok { - vars = append(vars, agentEnvPair{"XDG_DATA_HOME", agentXDGDataHome(xdgHome), false}) - vars = append(vars, agentEnvPair{"XDG_STATE_HOME", agentXDGStateHome(xdgHome), false}) - } - - // Under the per-agent-UID layout the global npm prefix is owned by the - // image's build user, so the Claude Code CLI's self-updater fails on - // every launch with "✘ Auto-update failed: no write permission to npm - // prefix" — a red line in every agent pane for an update the agent must - // not perform anyway (the CLI version is managed by the image, not by - // an in-pod npm write). Disabling the updater removes the failure at its - // source; a per-agent npm prefix would instead let an agent drift off - // the pinned image version. - vars = append(vars, agentEnvPair{"DISABLE_AUTOUPDATER", "1", false}) - } - - // Codex CLI 0.144.1's in-process app-server performs OWNER-gated operations - // on files under CODEX_HOME (helper-binary "PATH alias" symlinks under - // tmp/arg0, sqlite state). The shared /data/home/.codex is owned by dev - // (the entrypoint chowns it group-writable + setgid), which claude/copilot - // tolerate but Codex does not — every non-owner agent UID fails with - // "failed to start embedded app server: Operation not permitted (os error 1)". - // The manager launches the codex binary DIRECTLY (not via agent-launch.sh), - // so CODEX_HOME must be set here. Give each agent its own dir; codex will - // NOT create it (it errors "CODEX_HOME ... does not exist"), so it is - // pre-created AS the agent below in setupCodexHome. - if backend == codexBackend { - vars = append(vars, agentEnvPair{"CODEX_HOME", codexHomePath(agent.Name), false}) - } - - for _, conn := range agent.Config.Connections { - if conn.Type != "api" { - continue - } - envName := conn.EnvName - if envName == "" { - envName = "HIVE_CONN_" + strings.ToUpper(strings.ReplaceAll(conn.Name, "-", "_")) + "_URL" - } - vars = append(vars, agentEnvPair{envName, conn.URI, false}) - if conn.Auth != nil && conn.Auth.Type == "env" && conn.Auth.EnvVar != "" { - if tokenVal := os.Getenv(conn.Auth.EnvVar); tokenVal != "" { - vars = append(vars, agentEnvPair{conn.Auth.EnvVar, tokenVal, true}) - } - } - } - - return vars -} - -func (m *Manager) PinCLI(name, version string) error { - m.mu.Lock() - defer m.mu.Unlock() - - agent, ok := m.agents[name] - if !ok { - return fmt.Errorf("agent %s not found", name) - } - - agent.PinnedCLI = version - m.logger.Info("agent CLI pinned", "name", name, "version", version) - return nil -} - -func (m *Manager) UnpinCLI(name string) error { - m.mu.Lock() - defer m.mu.Unlock() - - agent, ok := m.agents[name] - if !ok { - return fmt.Errorf("agent %s not found", name) - } - - agent.PinnedCLI = "" - m.logger.Info("agent CLI unpinned", "name", name) - return nil -} - -func (m *Manager) PinModel(name, model string) error { - m.mu.Lock() - defer m.mu.Unlock() - - agent, ok := m.agents[name] - if !ok { - return fmt.Errorf("agent %s not found", name) - } - - prevModel := agent.effectiveModel() - agent.PinnedModel = model - agent.ModelOverride = model - m.logger.Info("agent model pinned", "name", name, "model", model) - if prevModel != model { - m.audit(AuditAgentModelSet, name, auditFields( - "outcome", "success", - "backend", agent.effectiveBackend(), - "model", model, - "previous_model", prevModel, - "trigger", "pin", - )) - } - return nil -} - -func (m *Manager) UnpinModel(name string) error { - m.mu.Lock() - defer m.mu.Unlock() - - agent, ok := m.agents[name] - if !ok { - return fmt.Errorf("agent %s not found", name) - } - - agent.PinnedModel = "" - m.logger.Info("agent model unpinned", "name", name) - return nil -} - -func (m *Manager) SetModelOverride(name, model string) error { - m.mu.Lock() - defer m.mu.Unlock() - - agent, ok := m.agents[name] - if !ok { - return fmt.Errorf("agent %s not found", name) - } - - // Store the CLI-accepted spelling for copilot so the persisted selection, - // the dropdown preselect, and auto-heal all agree on one canonical id - // (separator drift like claude-fable.5 vs claude-fable-5, #4262). Launch - // re-applies the same canonicalization, so even ids stored before this - // existed self-correct there. - if agent.effectiveBackend() == "copilot" { - model = CanonicalizeCopilotModel(model) - } - - // A pin blocks the governor's auto-selection, never a user's explicit - // switch: retarget the pin to the new model so the pin state is - // unchanged (still pinned) while the change takes effect. - if agent.PinnedModel != "" { - agent.PinnedModel = model - m.logger.Info("agent model pin retargeted by user switch", "name", name, "model", model) - } - - prevModel := agent.effectiveModel() - agent.ModelOverride = model - m.logger.Info("agent model override set", "name", name, "model", model) - // State CHANGES only — the governor re-asserts the current model on every - // evaluation cycle, so auditing unchanged writes would flood the ring. - if prevModel != model { - m.audit(AuditAgentModelSet, name, auditFields( - "outcome", "success", - "backend", agent.effectiveBackend(), - "model", model, - "previous_model", prevModel, - )) - } - - effectiveBackend := agent.Config.Backend - if agent.BackendOverride != "" { - effectiveBackend = agent.BackendOverride - } - if m.routableBackend(effectiveBackend) && m.inferenceRouteCallback != nil { - m.inferenceRouteCallback(name, effectiveBackend, model) - } - return nil -} - -func (m *Manager) SetBackendOverride(name, backend string) error { - m.mu.Lock() - defer m.mu.Unlock() - - agent, ok := m.agents[name] - if !ok { - return fmt.Errorf("agent %s not found", name) - } - - // Refuse a backend the launcher cannot dispatch, at SET time. Previously - // any string was accepted here and the agent was then restarted into it, - // failing only at launch with "unknown backend: " — the agent stops - // working and the operator gets no signal at the moment of the change. - // routableBackend covers configured gateway names (resolved live), so a - // gateway-named backend still passes. - if err := m.validateBackendName(backend); err != nil { - return err - } - - // Captured after validation so a rejected switch records no audit event: - // the override is only mutated below, once the backend is known routable. - prevBackend := agent.effectiveBackend() - agent.BackendOverride = backend - m.logger.Info("agent backend override set", "name", name, "backend", backend) - // Record only a real transition: /switch/{backend} is also re-applied on - // config reload with the value already in effect, and auditing those - // no-ops would bury the actual operator changes. - if prevBackend != backend { - m.audit(AuditAgentBackendSet, name, auditFields( - "outcome", "success", - "backend", backend, - "model", agent.effectiveModel(), - "previous_backend", prevBackend, - )) - } - - if m.routableBackend(backend) && m.inferenceRouteCallback != nil { - model := agent.ModelOverride - if model == "" { - model = agent.Config.Model - } - m.inferenceRouteCallback(name, backend, model) - } else if !IsInferenceBackend(backend) && m.clearInferenceRouteCallback != nil { - m.clearInferenceRouteCallback(name) - } - return nil -} - -// RefreshInferenceRoutes re-fires the inference route callback for every -// agent whose effective backend matches, so endpoint or credential changes -// (e.g. a governor LiteLLM config save) take effect on live agents without -// a restart. -func (m *Manager) RefreshInferenceRoutes(backend string) { - m.mu.Lock() - defer m.mu.Unlock() - if m.inferenceRouteCallback == nil || !m.routableBackend(backend) { - return - } - for name, agent := range m.agents { - effective := agent.Config.Backend - if agent.BackendOverride != "" { - effective = agent.BackendOverride - } - if effective != backend { - continue - } - model := agent.ModelOverride - if model == "" { - model = agent.Config.Model - } - m.inferenceRouteCallback(name, backend, model) - } -} - // GetBufferOutput returns output from the ring buffer directly, bypassing // the tmux pane capture. The ring buffer accumulates all output over time // (up to 500 lines) while the pane capture only has visible lines. diff --git a/src/pkg/agent/manager_consent.go b/src/pkg/agent/manager_consent.go new file mode 100644 index 0000000000..26db0fb34a --- /dev/null +++ b/src/pkg/agent/manager_consent.go @@ -0,0 +1,295 @@ +package agent + +import ( + "strings" + "time" +) + +const ( + // consentConfirmFooter appears at the bottom of Claude Code interactive + // selection screens (consent dialogs, settings-error menus). + consentConfirmFooter = "Enter to confirm" + // bypassConsentTitle is the heading of the --dangerously-skip-permissions + // consent screen. Its default selection is "No, exit" — confirming it + // terminates the CLI and leaves a bare bash pane. + bypassConsentTitle = "Bypass Permissions mode" + // bypassConsentDefaultOption is the default (negative) option on the + // bypass-permissions consent screen. + bypassConsentDefaultOption = "No, exit" + // bypassConsentAcceptOption is the affirmative option on the + // bypass-permissions consent screen. Its position varies between CLI + // versions, so acceptance navigates by matching the selected-line text. + bypassConsentAcceptOption = "Yes, I accept" + // apiKeyPromptTitle is the heading of the custom-API-key approval prompt, + // shown when ANTHROPIC_API_KEY is not in customApiKeyResponses.approved. + // Its default selection is "No (recommended)" with the affirmative option + // above it. + apiKeyPromptTitle = "Detected a custom API key" + // apiKeyPromptAcceptOption is the affirmative option on the + // custom-API-key approval prompt. + apiKeyPromptAcceptOption = "Yes" + // cliWorkingMarker is shown while Claude Code is actively processing a + // request; a pane in this state is never a consent screen. + cliWorkingMarker = "esc to interrupt" +) + +// paneShowsConsentScreen reports whether the pane is showing an interactive +// consent/selection screen rather than a ready CLI input prompt. Such screens +// contain a "❯"-selected menu option (e.g. "❯ 1. No, exit"), so they satisfy +// marker-based CLI presence checks ("❯" is also a cliPaneMarkers entry) — a +// kick typed into one is consumed by the menu, or by bash once the default +// "No, exit" selection terminates the CLI. Callers should pass the visible +// pane only (no scrollback): dismissed consent screens linger in history. +func paneShowsConsentScreen(pane string) bool { + if pane == "" || strings.Contains(pane, cliWorkingMarker) { + return false + } + // A known startup-blocking menu is not a ready prompt either. The generic + // test below needs the "Enter to confirm" footer AND a "❯"-marked line; + // codex renders neither (its footer is "Press enter to continue" and its + // marker is "›" U+203A), so its update menu read as READY. Everything that + // gates on readiness — the startup kick, caveman activation — then typed + // into the menu, and the Enter confirmed its pre-selected option: + // "1. Update now", which runs `npm install -g` as the agent UID, fails, and + // kills the CLI. Blocking on these lets the prompt watcher answer them. + if paneHasBlockingPrompt(pane) { + return true + } + if strings.Contains(pane, bypassConsentTitle) && strings.Contains(pane, bypassConsentDefaultOption) { + return true + } + if !strings.Contains(pane, consentConfirmFooter) { + return false + } + for _, line := range strings.Split(pane, "\n") { + if strings.HasPrefix(strings.TrimSpace(line), "❯") { + return true + } + } + return false +} + +// dismissInferencePrompts polls the tmux pane for Claude Code interactive +// prompts and auto-dismisses them. The "Bypass Permissions mode" consent +// screen and the custom-API-key approval prompt are handled first and +// explicitly (see confirmMenuOption): their default selections are negative +// ("No, exit" / "No (recommended)"), so confirming blind terminates the CLI +// or declines the seeded key. +// Other prompts are handled dynamically regardless of prompt text changes +// between Claude Code versions by: +// 1. Detecting "Enter to confirm" (universal prompt footer) +// 2. Finding the selected option (line with "❯" marker) +// 3. If selected option looks negative (contains "No" or "exit"), navigate +// away from it before confirming +// 4. For "Press Enter to continue" screens, just press Enter +// +// The pane is polled fast for the first 10s — the consent screen appears +// within ~5-8s of launch and every second it lingers is a window for a kick +// to be swallowed by the menu — then at a relaxed interval. +// +// Stops when the main Claude Code input prompt appears ("esc to interrupt"). +func (m *Manager) dismissInferencePrompts(agent *AgentProcess) { + const ( + // promptFastPollWindow covers the launch window in which the consent + // screen normally appears (~5-8s after CLI start). + promptFastPollWindow = 10 * time.Second + promptFastPollInterval = 250 * time.Millisecond + promptPollInterval = 1 * time.Second + promptDismissTimeout = 60 * time.Second + postKeystrokeDelay = 500 * time.Millisecond + ) + + start := time.Now() + timeout := promptDismissTimeout + if m.promptDismissTimeout > 0 { + timeout = m.promptDismissTimeout + } + deadline := start.Add(timeout) + lastPane := "" + + for time.Now().Before(deadline) { + interval := promptPollInterval + if time.Since(start) < promptFastPollWindow { + interval = promptFastPollInterval + } + m.sleepDuringPromptDismiss(interval) + + pane := m.captureVisiblePaneForAgent(agent) + if pane == "" { + continue + } + + // Bypass-permissions consent screen: handle first and explicitly, + // even if the pane is unchanged since the last poll (a mistimed + // keystroke must be retried, not skipped). The affirmative option + // sits below the default "No, exit". + if strings.Contains(pane, bypassConsentTitle) && !strings.Contains(pane, cliWorkingMarker) { + m.logger.Info("accepting bypass-permissions consent", "agent", agent.Name) + m.confirmMenuOption(agent, bypassConsentTitle, bypassConsentAcceptOption, "Down") + lastPane = "" // re-capture fresh on the next pass + continue + } + + // Custom-API-key approval prompt: the affirmative "Yes" sits ABOVE + // the default "No (recommended)" selection, so the generic + // Down-then-Enter fallback below would decline it. + if strings.Contains(pane, apiKeyPromptTitle) && !strings.Contains(pane, cliWorkingMarker) { + m.logger.Info("approving seeded inference API key", "agent", agent.Name) + m.confirmMenuOption(agent, apiKeyPromptTitle, apiKeyPromptAcceptOption, "Up") + lastPane = "" + continue + } + + if pane == lastPane { + continue + } + lastPane = pane + + // Main prompt visible — agent is ready + if strings.Contains(pane, "bypass permissions on") || strings.Contains(pane, "esc to interrupt") { + m.logger.Info("inference agent ready", "agent", agent.Name) + return + } + + // "Press Enter to continue" screens + if strings.Contains(pane, "Press Enter to continue") { + m.logger.Info("inference prompt: press enter", "agent", agent.Name) + m.tmuxSendKeysForAgent(agent, "Enter") + continue + } + + // Selection prompts have "Enter to confirm" footer + if !strings.Contains(pane, "Enter to confirm") { + continue + } + + // Find the currently selected option (marked with ❯) + selected := selectedMenuOption(pane) + + m.logger.Info("inference prompt detected", + "agent", agent.Name, + "selected", selected, + ) + + // If current selection looks negative, navigate away from it + selectedLower := strings.ToLower(selected) + if strings.Contains(selectedLower, "no,") || strings.Contains(selectedLower, "no ") || + strings.Contains(selectedLower, "exit") { + // Try moving down first (most prompts put the positive option below) + m.tmuxSendKeysForAgent(agent, "Down") + m.sleepDuringPromptDismiss(postKeystrokeDelay) + } else if strings.Contains(selectedLower, "fix with") { + // Settings error: skip past "Fix with Claude" and "Exit" to "Continue without" + m.tmuxSendKeysForAgent(agent, "Down") + m.sleepDuringPromptDismiss(postKeystrokeDelay) + m.tmuxSendKeysForAgent(agent, "Down") + m.sleepDuringPromptDismiss(postKeystrokeDelay) + } + + m.tmuxSendKeysForAgent(agent, "Enter") + } + + m.logger.Warn("inference prompt dismissal timed out", "agent", agent.Name) +} + +func (m *Manager) sleepDuringPromptDismiss(d time.Duration) { + m.term().Sleep(d) +} + +// selectedMenuOption returns the trimmed text of the "❯"-selected line of an +// interactive CLI menu, or "" if no line is selected. +func selectedMenuOption(pane string) string { + for _, line := range strings.Split(pane, "\n") { + trimmed := strings.TrimSpace(line) + if strings.HasPrefix(trimmed, "❯") { + return trimmed + } + } + return "" +} + +// confirmMenuOption drives an interactive CLI menu identified by title to the +// option whose text contains want, then confirms it with Enter. Navigation +// matches the "❯"-selected line text rather than pressing a fixed number of +// keys, so it lands on the right option whichever position it occupies (menu +// option order differs between Claude CLI versions). navKey is the arrow key +// to step with ("Down" or "Up"). Returns true once the option was confirmed +// or the screen is gone. +func (m *Manager) confirmMenuOption(agent *AgentProcess, title, want, navKey string) bool { + const ( + // menuMaxNavigateSteps bounds arrow-key navigation; the handled menus + // have 2 options, extra headroom covers future variants. + menuMaxNavigateSteps = 4 + postKeystrokeDelay = 500 * time.Millisecond + ) + for step := 0; step < menuMaxNavigateSteps; step++ { + pane := m.captureVisiblePaneForAgent(agent) + if !strings.Contains(pane, title) || strings.Contains(pane, cliWorkingMarker) { + return true // screen already dismissed + } + if strings.Contains(selectedMenuOption(pane), want) { + m.tmuxSendKeysForAgent(agent, "Enter") + m.sleepDuringPromptDismiss(postKeystrokeDelay) + return true + } + m.tmuxSendKeysForAgent(agent, navKey) + m.sleepDuringPromptDismiss(postKeystrokeDelay) + } + m.logger.Warn("inference menu: wanted option not reached", + "agent", agent.Name, "title", title, "want", want) + return false +} + +const ( + // consentStuckGracePeriod is how long a consent screen must stay visible + // across watcher passes before the agent counts as stuck. The launch-time + // dismissal goroutine runs for 60s, so a screen still visible this long + // after first being seen by the watcher means dismissal lost the race. + consentStuckGracePeriod = 30 * time.Second + // consentDismissCooldown is the minimum interval between watcher-triggered + // dismissal passes for one agent, so a stubborn screen can't spam + // keystroke goroutines (each dismissal pass itself polls for 60s). + consentDismissCooldown = 2 * time.Minute +) + +// clearConsentTracking resets the consent-stuck timer for an agent whose pane +// no longer shows a consent screen. +func (m *Manager) clearConsentTracking(name string) { + m.mu.Lock() + defer m.mu.Unlock() + if agent, ok := m.agents[name]; ok { + agent.consentSeenAt = time.Time{} + } +} + +// dismissConsentIfStuck re-runs dismissInferencePrompts for an inference agent +// whose pane has shown a consent screen for longer than the grace period, +// subject to a per-agent cooldown. Called from the watcher loop +// (CheckAndRestartCrashedAgents) so an agent that lands on a consent screen +// after launch — e.g. a crash-recovery restart whose launch-time dismissal +// timed out — recovers instead of sitting stuck while kicks appear to succeed. +func (m *Manager) dismissConsentIfStuck(name string) { + now := time.Now() + m.mu.Lock() + agent, ok := m.agents[name] + if !ok { + m.mu.Unlock() + return + } + if agent.consentSeenAt.IsZero() { + agent.consentSeenAt = now + m.mu.Unlock() + return + } + stuckFor := now.Sub(agent.consentSeenAt) + if stuckFor < consentStuckGracePeriod || now.Sub(agent.lastConsentDismiss) < consentDismissCooldown { + m.mu.Unlock() + return + } + agent.lastConsentDismiss = now + m.mu.Unlock() + + m.logger.Warn("inference agent stuck on consent screen, re-running prompt dismissal", + "name", name, "stuck_seconds", int(stuckFor.Seconds())) + go m.dismissInferencePrompts(agent) +} diff --git a/src/pkg/agent/manager_copilot_auth.go b/src/pkg/agent/manager_copilot_auth.go new file mode 100644 index 0000000000..f3279163b5 --- /dev/null +++ b/src/pkg/agent/manager_copilot_auth.go @@ -0,0 +1,48 @@ +package agent + +import ( + "time" +) + +const ( + sharedConfigDesiredMode = 0o660 + // agyDefaultEffort is the reasoning effort passed alongside agy's --model + // when the agent has no usable reasoning_effort configured (see + // agyLaunchEffort). agy requires --effort whenever --model is given and + // otherwise ignores the model entirely; "low" is the effort agy defaults + // to on its own, so this makes the configured model take effect without + // changing behaviour. + agyDefaultEffort = "low" + + tokenRestartCooldownSec = 60 // minimum seconds between token-triggered restarts per agent + // loginPromptTailLines bounds the pane region the login-prompt detector + // reads: a prompt the CLI is stuck at sits at the pane bottom, while + // echoed kick text and startup flashes live in scrollback (see the poller). + loginPromptTailLines = 15 + // loginStreakRestartMin is how many consecutive polls (~3s apart) must see + // the login prompt before a token-triggered restart may fire — filters the + // CLI's transient startup "/login" flash. + loginStreakRestartMin = 3 + // tokenRestartMaxAttempts bounds CONSECUTIVE token-triggered restarts that + // fail to clear the login prompt. + // + // The three guards above answer WHEN to restart; none of them answered HOW + // MANY TIMES, so a restart that could never work was retried forever at the + // cooldown interval. #4596 is precisely that shape: the shared credential is + // valid (so configHasTokens() is true) while $HOME/.claude.json has lost its + // oauthAccount (so the CLI shows the login menu regardless), and each + // restart re-launched a CLI that rewrote the same contended file and asked + // again. Restarts are not free — they destroy in-flight work, which is the + // failure the kick grace above was added for. + // + // Three is deliberately generous: one restart genuinely does fix the case + // this feature was built for (an operator authenticates in one agent's + // terminal and the others need a nudge), so the cap only engages on a + // theory that has now failed repeatedly. + tokenRestartMaxAttempts = 3 + // tokenRestartKickGrace suppresses token-triggered restarts after a kick + // delivery so the restart can never destroy just-delivered work. + tokenRestartKickGrace = 10 * time.Minute + expiredTokenHangTimeoutSec = 180 // blank pane after this many seconds triggers token purge + restart + tlsErrorRestartCooldownSec = 120 // minimum seconds between TLS-error-triggered restarts per agent +) diff --git a/src/pkg/agent/manager_env.go b/src/pkg/agent/manager_env.go new file mode 100644 index 0000000000..46af591b77 --- /dev/null +++ b/src/pkg/agent/manager_env.go @@ -0,0 +1,420 @@ +package agent + +import ( + "encoding/json" + "fmt" + "os" + "os/exec" + "path/filepath" + "regexp" + "strconv" + "strings" + + "github.com/hivecommons/hive/pkg/config" + ghpkg "github.com/hivecommons/hive/pkg/github" +) + +func (m *Manager) buildBootstrapPrompt(agent *AgentProcess) string { + // No boot prompt — the governor's first eval cycle (10s after startup) + // kicks all due agents via BuildKickMessages with fully substituted + // templates. Sending a boot prompt here caused unsubstituted ${ISSUE_LIST} + // and other vars to reach the agent. The policy-file path list this + // function used to assemble was dead code once the boot prompt was + // removed, so it is gone too. + _ = agent // signature kept for the call site; the arg is no longer read + return "" +} + +// metricsCachePath is a var (not const) so tests can point it at a temp file +// to exercise readCoveragePreamble without a real /data volume. Production +// value is unchanged. +var metricsCachePath = "/data/metrics/agent-metrics-cache.json" + +func (m *Manager) readCoveragePreamble() string { + data, err := os.ReadFile(metricsCachePath) + if err != nil { + return "" + } + var metrics map[string]map[string]json.Number + if err := json.Unmarshal(data, &metrics); err != nil { + return "" + } + ci, ok := metrics["ci-maintainer"] + if !ok { + return "" + } + cov, err := ci["coverage"].Int64() + if err != nil { + return "" + } + target, err := ci["coverageTarget"].Int64() + if err != nil { + target = 91 + } + return fmt.Sprintf("[COVERAGE] Current: %d%% | Target: %d%%.", cov, target) +} + +// shellEnvVar formats KEY='value' with single-quoting so values containing +// spaces, parentheses, or other shell metacharacters are safe in inline env +// var assignments sent to tmux via send-keys. +func shellEnvVar(key, value string) string { + quoted := strings.ReplaceAll(value, "'", "'\"'\"'") + return fmt.Sprintf("%s='%s'", key, quoted) +} + +// applySecretEnv pushes only the Secret pairs into the agent's tmux session via +// set-environment. Values are passed as exec args (never through a shell), so +// they are not word-split and never land in the pane or in bash history. +// Failures are ignored for the same reason ensureTmuxSession ignores them: a +// missing session is handled by the launch path, not here. +func (m *Manager) applySecretEnv(agent *AgentProcess) { + if agent == nil || agent.tmuxSession == "" { + return + } + for _, p := range m.agentEnvPairs(agent) { + if !p.Secret { + continue + } + _ = m.tmuxCmd(agent, "set-environment", "-t", agent.tmuxSession, p.Key, p.Value).Run() + } +} + +func (m *Manager) buildEnvPrefix(agent *AgentProcess) string { + pairs := m.agentEnvPairs(agent) + var parts []string + for _, p := range pairs { + if p.Secret { + continue + } + parts = append(parts, shellEnvVar(p.Key, p.Value)) + } + if len(parts) == 0 { + return "" + } + return strings.Join(parts, " ") + " " +} + +// embeddedTokenRe matches git remote URLs with embedded credentials: +// https://x-access-token:TOKEN@github.com/org/repo.git +var embeddedTokenRe = regexp.MustCompile(`^https://[^@]+@(github\.com/.+)$`) + +// sanitizeGitRemotes strips embedded tokens from git remote URLs in all repos +// under the agent's work directory. Copilot CLI embeds the GitHub App token +// directly in the remote URL when it clones, bypassing both the credential +// helper (Layer 1) and env var filtering (Layer 2). +func (m *Manager) sanitizeGitRemotes(agent *AgentProcess) { + if m.agentMode(agent).CanPush() { + return + } + agentDir := m.workDir + "/" + agent.Name + _ = filepath.WalkDir(agentDir, func(path string, d os.DirEntry, err error) error { + if err != nil || d.Name() != ".git" || !d.IsDir() { + return nil + } + repoDir := filepath.Dir(path) + out, err := exec.Command("git", "-C", repoDir, "remote", "get-url", "origin").Output() + if err != nil { + return filepath.SkipDir + } + url := strings.TrimSpace(string(out)) + if match := embeddedTokenRe.FindStringSubmatch(url); match != nil { + clean := "https://" + match[1] + _ = exec.Command("git", "-C", repoDir, "remote", "set-url", "origin", clean).Run() + m.logger.Info("stripped embedded token from git remote", + "agent", agent.Name, "repo", repoDir) + } + return filepath.SkipDir + }) +} + +// agentEnvPair is an unquoted key-value environment variable. +type agentEnvPair struct { + Key string + Value string + // Secret vars are set via tmux set-environment only, never on the command line. + Secret bool +} + +// inferenceQuietCLIEnv is the set of Claude CLI switches exported to +// inference-routed sessions so the CLI stops emitting non-inference traffic +// (telemetry, error reporting, nonessential lookups) to its Anthropic host. +var inferenceQuietCLIEnv = []string{ + "DISABLE_TELEMETRY", + "DISABLE_ERROR_REPORTING", + "CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC", +} + +func (m *Manager) agentEnvPairs(agent *AgentProcess) []agentEnvPair { + model := agent.Config.Model + if agent.ModelOverride != "" { + model = agent.ModelOverride + } + backend := agent.Config.Backend + if agent.BackendOverride != "" { + backend = agent.BackendOverride + } + displayName := agent.Config.DisplayName + if displayName == "" { + displayName = agent.Name + } + vars := []agentEnvPair{ + {"HIVE_AGENT", agent.Name, false}, + {"HIVE_AGENT_DISPLAY_NAME", displayName, false}, + {"HIVE_BACKEND", backend, false}, + {"HIVE_MODEL", model, false}, + } + if hiveID := os.Getenv("HIVE_ID"); hiveID != "" { + vars = append(vars, agentEnvPair{"HIVE_ID", hiveID, false}) + } + vars = append(vars, agentEnvPair{"HIVE_ACMM_LEVEL", fmt.Sprintf("%d", m.project.ACMMLevel), false}) + + mode := m.agentMode(agent) + if agent.Config.Tools != nil { + if effectiveMode := agent.Config.Tools.EffectiveMode(); effectiveMode != "" { + vars = append(vars, agentEnvPair{"HIVE_AGENT_MODE", effectiveMode, false}) + } else { + vars = append(vars, agentEnvPair{"HIVE_AGENT_MODE", mode.String(), false}) + } + } else { + vars = append(vars, agentEnvPair{"HIVE_AGENT_MODE", mode.String(), false}) + } + modeFile := filepath.Join(agentStateDir, ".hive-mode-"+agent.Name) + if err := writeAgentStateFile(modeFile, []byte(mode.String())); err != nil { + m.logger.Warn("agentBootstrapEnv: mode file write failed", "file", modeFile, "error", err) + } + m.writeAgentCapsFile(agent.Name, m.agentCapabilities(agent)) + // Plain proxy URL without userinfo — Claude Code's native binary fails + // to open a socket when the URL contains username:password@ (FailedToOpenSocket). + // Agent identification uses UID-based /proc/net/tcp lookup instead of + // Proxy-Authorization headers. GIT_TERMINAL_PROMPT=0 prevents git from + // prompting for proxy credentials. + proxyURL := fmt.Sprintf("http://127.0.0.1:%d", proxyListenPort) + vars = append(vars, agentEnvPair{"HTTPS_PROXY", proxyURL, false}) + vars = append(vars, agentEnvPair{"HTTP_PROXY", proxyURL, false}) + vars = append(vars, agentEnvPair{"HIVE_PROXY_AGENT", agent.Name, false}) + vars = append(vars, agentEnvPair{"GIT_TERMINAL_PROMPT", "0", false}) + vars = append(vars, agentEnvPair{"NODE_EXTRA_CA_CERTS", proxyCACertPath, false}) + if sha := os.Getenv("HIVE_SHA"); sha != "" { + vars = append(vars, agentEnvPair{"HIVE_SHA", sha, false}) + } + if advisory := os.Getenv("HIVE_ADVISORY_ISSUE"); advisory != "" { + vars = append(vars, agentEnvPair{"HIVE_ADVISORY_ISSUE", advisory, false}) + } + // HIVE_REPO / HIVE_REPOS: the shipped policy templates instruct agents to + // run `gh issue create --repo "$HIVE_REPO"`, but nothing ever exported it + // to hosted agents (only the OSS scheduler set a hardcoded "/hive"). + // Root-caused on a live hosted hive (2026-08-20): the sec-check agent saw + // HIVE_REPO unset, fell back to the git remote of its own workdir, and + // silently scanned only the primary repo — the other project repos were + // never touched. Export the primary repo and the full project repo list so + // templates and agents can target every configured repo. + if m.project.Org != "" && len(m.project.Repos) != 0 { + primary := m.project.PrimaryRepo() + if primary == "" { + primary = m.project.Repos[0] + } + vars = append(vars, agentEnvPair{"HIVE_REPO", m.project.Org + "/" + primary, false}) + full := make([]string, len(m.project.Repos)) + for i, r := range m.project.Repos { + full[i] = m.project.Org + "/" + r + } + vars = append(vars, agentEnvPair{"HIVE_REPOS", strings.Join(full, ","), false}) + } + // GH_HOST: point the gh CLI at the configured forge host for GHE spokes. + // See ProjectContext.GHHost. The gh wrapper pairs this with + // GH_ENTERPRISE_TOKEN so the per-agent scoped token authenticates there. + if m.project.GHHost != "" { + vars = append(vars, agentEnvPair{"GH_HOST", m.project.GHHost, false}) + } + if m.routableBackend(backend) { + const inferenceTranslatePort = 18444 + vars = append(vars, agentEnvPair{"ANTHROPIC_API_KEY", "sk-hive-" + agent.Name, false}) + baseURL := fmt.Sprintf("http://127.0.0.1:%d", inferenceTranslatePort) + vars = append(vars, agentEnvPair{"ANTHROPIC_BASE_URL", baseURL, false}) + vars = append(vars, agentEnvPair{"NO_PROXY", "127.0.0.1,localhost", false}) + // Cap the CLI output-token budget at a value every commercial model + // litellm may front will accept. A prior 128000 (chosen so verbose + // OSS models would not truncate) exceeds Azure GPT-4o's 16384 + // completion-token cap, so every request 400s with + // "max_tokens is too large: 128000. This model supports at most + // 16384 completion tokens". See inferenceMaxOutputTokensDefault. + // TODO: the gateway 400 body names the model's real cap ("supports + // at most N completion tokens"); a future enhancement could parse it + // to auto-adjust per-model instead of using a universal floor. + vars = append(vars, agentEnvPair{"CLAUDE_CODE_MAX_OUTPUT_TOKENS", strconv.Itoa(inferenceMaxOutputTokensDefault), false}) + // The Claude CLI sends telemetry batches, error reports, and other + // non-inference traffic to its configured Anthropic host. Routed at + // an OpenAI-compatible gateway that traffic has nowhere useful to go + // (the proxy now answers it locally rather than forwarding it — see + // classifyInferencePath), so switch it off at the source. Only for + // inference-routed sessions: subscription/Anthropic-direct sessions + // keep Anthropic's own telemetry. + for _, v := range inferenceQuietCLIEnv { + vars = append(vars, agentEnvPair{v, "1", false}) + } + } + if m.copilotAuthToken != "" { + vars = append(vars, agentEnvPair{copilotTokenEnvVar, m.copilotAuthToken, true}) + } + // Point the GitHub MCP server at the App installation token so PRs, issue + // comments, and merges are authored by the App bot ("[bot]") — NOT by + // the Copilot login user. COPILOT_GITHUB_TOKEN above stays as the Copilot + // OAuth token because it authenticates the AI model (a separate concern from + // GitHub write identity); leaving it untouched keeps the Copilot CLI login + // working. The Copilot CLI reads GH_TOKEN / GITHUB_TOKEN for GitHub API auth + // (per its README: GH_TOKEN or GITHUB_TOKEN, in that precedence), so setting + // GITHUB_TOKEN here makes the built-in GitHub MCP server act as the App bot. + // + // Gated on the opt-in flag first (default OFF → no behavior change on any + // hive that has not explicitly enabled App-bot authorship), then on CanPush(): + // advisory agents are deliberately kept GITHUB_TOKEN-less (see the -u + // GITHUB_TOKEN strip after the env loop) so they cannot write; only push- + // capable tiers — the ones that legitimately open/merge PRs — get the App + // token. m.appAuth != nil means an App is configured. The value is the + // per-agent tier-SCOPED App token, and refreshAgentTokens re-pushes it hourly + // so it never goes stale. + if m.project.AppAuthoredPRs && m.appAuth != nil && agent.UID > 0 && m.agentMode(agent).CanPush() { + if data, err := os.ReadFile(ghpkg.AgentTokenCachePath(agent.Name)); err == nil { + if tok := strings.TrimSpace(string(data)); tok != "" { + vars = append(vars, agentEnvPair{"GITHUB_TOKEN", tok, true}) + } + } + } + // Linear write credential for ISSUES_ONLY+ agents — see linearEnvPairs. + // Nil for advisory agents and for hives with no Linear credential, so a + // GitHub-only hive sees no change. + vars = append(vars, m.linearEnvPairs(agent)...) + // CLAUDE_CODE_OAUTH_TOKEN is a LAST RESORT, not the normal delivery path. + // + // Claude Code treats this variable as a static bearer token: when it is + // set the CLI uses it verbatim, never opens ~/.claude/.credentials.json, + // and therefore never refreshes. Measured in-container (2026-09-01): with + // the variable set to a bad value and a perfectly good credentials file + // beside it, the CLI answered "401 OAuth access token is invalid" — there + // is no fallback to the file. + // + // m.claudeAuthToken is a snapshot of the SHORT-LIVED access token, taken + // once at manager construction and refreshed only by ReloadClaudeToken() + // after a dashboard login. Injecting it therefore pinned every claude + // agent to the remaining life of whatever access token happened to be on + // disk when the container started — Claude access tokens live 8h, so the + // whole fleet 401'd within a day of every restart and the only recovery + // hive offered was an operator re-login, once per agent. That is the daily + // re-authentication treadmill of #5454. + // + // It is also unnecessary since per-agent homes (#4619): every agent's + // ~/.claude is a symlink to the shared /data/home/.claude, so the CLI can + // read the credential itself — and redeem its refresh grant on start, + // which is the one thing the env var makes impossible. + // + // So inject ONLY when the agent has no credential file it can read. That + // keeps the variable doing the job it was added for (#c5648bc9: deliver a + // dashboard-obtained token to an agent that cannot see the file) and stops + // it overriding a credential that can still refresh itself. + if m.claudeAuthToken != "" && backend == "claude" && !claudeCredentialReachable(agent, backend) { + vars = append(vars, agentEnvPair{"CLAUDE_CODE_OAUTH_TOKEN", m.claudeAuthToken, true}) + } + // bob reads its key from BOBSHELL_API_KEY. Secret: true keeps the value off + // the shell command line (out of `ps`, bash history, and pane scrollback); + // it reaches the CLI via tmux set-environment only. Gated on the backend so + // no other CLI's environment carries an IBM credential it has no use for. + if backend == bobBackend { + if key := m.bobAPIKey(); key != "" { + vars = append(vars, agentEnvPair{config.BobAPIKeyEnvVar, key, true}) + } + // BOBSHELL_DEFAULT_AUTH_TYPE is what actually selects API-key auth; + // without it bob defaults to W3ID SSO and parks at the interactive key + // prompt forever. Deliberately NOT Secret: the value is the literal + // non-credential string "api-key", and secret pairs only reach a + // freshly-created pane shell via tmux set-environment, whereas + // non-secret pairs are re-applied on EVERY launch through + // buildEnvPrefix. That asymmetry is exactly what caused the sibling + // bug fixed in #2228, so the auth type must ride the always-reapplied + // path or a relaunch into an existing session loses it. + vars = append(vars, agentEnvPair{config.BobAuthTypeEnvVar, config.BobAuthTypeAPIKey, false}) + } + // BD_DIR tells the `bd` CLI where to read/write beads. Without this, + // bd falls back to cwd (/data/agents/) instead of the configured + // beads_dir (/data/beads/), causing a path mismatch with the + // dashboard and advisory digest. + if agent.Config.BeadsDir != "" { + vars = append(vars, agentEnvPair{"BD_DIR", agent.Config.BeadsDir, false}) + } + if agent.Config.CavemanMode != "" { + vars = append(vars, agentEnvPair{"HIVE_CAVEMAN_MODE", agent.Config.CavemanMode, false}) + } + // Export the RESOLVED explain mode, not the raw config value, so an agent's + // skills and helper scripts see the same answer the kick suffix acted on + // (including inheritance from the hive-wide default and the off fallback for + // an invalid value). Always exported, off included, so a script can branch on + // it without having to re-derive the precedence rules itself. + vars = append(vars, agentEnvPair{config.ExplainModeEnvVar, resolveExplainMode(agent.Config, m.explainModeDefault()), false}) + // GIT_SSL_CAINFO only — NOT SSL_CERT_FILE (that breaks Copilot API TLS) + vars = append(vars, agentEnvPair{"GIT_SSL_CAINFO", proxyCACertPath, false}) + if agent.UID > 0 { + vars = append(vars, agentEnvPair{"HIVE_AGENT_TOKEN_CACHE", ghpkg.AgentTokenCachePath(agent.Name), false}) + } + if agent.UID > 0 { + // Per-UID agents get a per-agent HOME (#4596) — AgentHome is the single + // source of truth so the auth probe and this export can never diverge. + vars = append(vars, agentEnvPair{"HOME", AgentHome(agent.Name, agent.UID, backend), false}) + + // Per-agent XDG data/state roots (#6238), beneath the per-agent HOME. + // Every backend CLI keeps its session transcripts, run locks and + // caches under $XDG_DATA_HOME / $XDG_STATE_HOME; with the legacy + // shared /data/home/.local those were one contended tree owned by + // whichever agent wrote first. Exported explicitly (not left to the + // spec default under $HOME) so the answer cannot depend on how each + // CLI resolves XDG, and only when HOME itself is per-agent — the + // HIVE_SHARED_AGENT_HOME=1 escape hatch keeps the legacy layout whole. + // XDG_CONFIG_HOME is deliberately NOT set: ~/.config stays the shared + // credential/config bridge (gh hosts.yml, goose config.yaml). See + // setupAgentXDGDirs, which pre-creates these as the agent's own dirs. + if xdgHome, ok := perAgentXDGHome(agent.Name, agent.UID, backend); ok { + vars = append(vars, agentEnvPair{"XDG_DATA_HOME", agentXDGDataHome(xdgHome), false}) + vars = append(vars, agentEnvPair{"XDG_STATE_HOME", agentXDGStateHome(xdgHome), false}) + } + + // Under the per-agent-UID layout the global npm prefix is owned by the + // image's build user, so the Claude Code CLI's self-updater fails on + // every launch with "✘ Auto-update failed: no write permission to npm + // prefix" — a red line in every agent pane for an update the agent must + // not perform anyway (the CLI version is managed by the image, not by + // an in-pod npm write). Disabling the updater removes the failure at its + // source; a per-agent npm prefix would instead let an agent drift off + // the pinned image version. + vars = append(vars, agentEnvPair{"DISABLE_AUTOUPDATER", "1", false}) + } + + // Codex CLI 0.144.1's in-process app-server performs OWNER-gated operations + // on files under CODEX_HOME (helper-binary "PATH alias" symlinks under + // tmp/arg0, sqlite state). The shared /data/home/.codex is owned by dev + // (the entrypoint chowns it group-writable + setgid), which claude/copilot + // tolerate but Codex does not — every non-owner agent UID fails with + // "failed to start embedded app server: Operation not permitted (os error 1)". + // The manager launches the codex binary DIRECTLY (not via agent-launch.sh), + // so CODEX_HOME must be set here. Give each agent its own dir; codex will + // NOT create it (it errors "CODEX_HOME ... does not exist"), so it is + // pre-created AS the agent below in setupCodexHome. + if backend == codexBackend { + vars = append(vars, agentEnvPair{"CODEX_HOME", codexHomePath(agent.Name), false}) + } + + for _, conn := range agent.Config.Connections { + if conn.Type != "api" { + continue + } + envName := conn.EnvName + if envName == "" { + envName = "HIVE_CONN_" + strings.ToUpper(strings.ReplaceAll(conn.Name, "-", "_")) + "_URL" + } + vars = append(vars, agentEnvPair{envName, conn.URI, false}) + if conn.Auth != nil && conn.Auth.Type == "env" && conn.Auth.EnvVar != "" { + if tokenVal := os.Getenv(conn.Auth.EnvVar); tokenVal != "" { + vars = append(vars, agentEnvPair{conn.Auth.EnvVar, tokenVal, true}) + } + } + } + + return vars +} diff --git a/src/pkg/agent/manager_modes.go b/src/pkg/agent/manager_modes.go new file mode 100644 index 0000000000..029f18f7a4 --- /dev/null +++ b/src/pkg/agent/manager_modes.go @@ -0,0 +1,354 @@ +package agent + +import ( + "fmt" + "path/filepath" + "strings" +) + +// ClearModeOverrides clears Config.Mode for the NAMED agents so that +// DefaultAgentMode determines their mode from the ACMM level. Call it before +// SyncModeFiles when applying a pack, because a pack agent's Config.Mode may +// have been set by a previous level's pack and would otherwise override the +// new level's expected default. +// +// Scoped to a name list — the pack's roster — on purpose (#7503). The previous +// ClearAllModeOverrides wiped every agent in the process table, including +// agents no pack manages. Their Mode was never pack-seeded, so there is no +// stale pack value to clear: what got cleared was the OPERATOR's setting. On +// the projectbluefin spoke `reviewer` was `mode: ADVISORY` in every config +// layer and ran as ISSUES_AND_PRS, the L5 default — two rungs more authority +// than anything on disk granted it — because the startup pack apply cleared it +// here and SyncModeFiles then wrote the default. Names not in the process +// table are ignored. +func (m *Manager) ClearModeOverrides(names []string) { + m.mu.Lock() + defer m.mu.Unlock() + for _, name := range names { + if agent, ok := m.agents[name]; ok { + agent.Config.Mode = "" + } + } +} + +// SyncModeFiles rewrites /tmp/.hive-mode-* for all running agents to reflect the given ACMM level. +func (m *Manager) SyncModeFiles(level int) { + m.mu.RLock() + defer m.mu.RUnlock() + for name, agent := range m.agents { + if agent.Paused { + continue + } + mode := DefaultAgentMode(name, level) + // converseConfigured is logged on both branches below so one grep by + // agent name shows whether the capability came from config or fell + // back to the default alongside the mode decision (#7503). + converseConfigured := agent.Config.Converse != nil + if modeStr := agent.Config.Mode; modeStr != "" { + if parsed, ok := ParseAgentMode(modeStr); ok { + m.logger.Info("SyncModeFiles: Config.Mode override", + "agent", name, "level", level, + "default", DefaultAgentMode(name, level).String(), + "override", modeStr, + "converse_configured", converseConfigured) + mode = parsed + } + } else { + // Log the fallback too, not only the override (#7503). Before this, + // an agent whose configured mode had been dropped on the way to the + // process table was indistinguishable from one that never set a + // mode: the only signal was reading /tmp/.hive-mode- and + // comparing by hand. One line saying "config said nothing, using + // the level default" turns that into a grep. `overlay_file` names + // the per-agent file when the entry came from one, so an operator + // can check what it says against what is being used. + m.logger.Info("SyncModeFiles: no Config.Mode, using level default", + "agent", name, "level", level, + "default", mode.String(), + "converse_configured", converseConfigured, + "overlay_file", agent.Config.SourceFile()) + } + modeFile := filepath.Join(agentStateDir, ".hive-mode-"+name) + if err := writeAgentStateFile(modeFile, []byte(mode.String())); err != nil { + m.logger.Warn("SyncModeFiles: write failed", "file", modeFile, "error", err) + } + // The capability file rides the same sync (#4492). It is level-independent + // today, but writing it here is what makes a `converse` change take effect + // on the next reconcile instead of only at the next agent launch. + caps := DefaultCapabilities(mode, level) + if agent.Config.Converse != nil { + caps.Converse = *agent.Config.Converse + } + m.writeAgentCapsFile(name, caps) + } +} + +// agentCapabilities returns the ORTHOGONAL capabilities for a given agent +// (#4492). Unlike agentMode there is no per-level default table: `converse` is +// opt-in everywhere, so an agent whose config says nothing gets the zero value +// and behaves exactly as it did before capabilities existed. +func (m *Manager) agentCapabilities(agent *AgentProcess) AgentCapabilities { + caps := DefaultCapabilities(m.agentMode(agent), m.project.ACMMLevel) + if agent.Config.Converse != nil { + caps.Converse = *agent.Config.Converse + } + return caps +} + +// writeAgentCapsFile persists the capability set the proxy reads on the request +// path. It is written for EVERY agent, including those with no capabilities, so +// a cleared `converse` actually revokes: leaving a stale file behind would keep +// granting the capability after the operator turned it off. +func (m *Manager) writeAgentCapsFile(name string, caps AgentCapabilities) { + capsFile := filepath.Join(agentStateDir, ".hive-caps-"+name) + if err := writeAgentStateFile(capsFile, []byte(caps.String())); err != nil { + m.logger.Warn("caps file write failed", "file", capsFile, "error", err) + } +} + +// agentMode returns the GitHub interaction mode for a given agent at the current ACMM level. +// If the agent has an explicit Mode in its config (hive.yaml or pack YAML), that takes precedence. +// Otherwise, the default table by ACMM level is used. +func (m *Manager) agentMode(agent *AgentProcess) AgentMode { + if modeStr := agent.Config.Mode; modeStr != "" { + if parsed, ok := ParseAgentMode(modeStr); ok { + return parsed + } + } + return DefaultAgentMode(agent.Name, m.project.ACMMLevel) +} + +// DefaultAgentMode returns the default mode for a given agent name and ACMM level, +// ignoring any hive.yaml override. Used by the dashboard to show "(default)" indicators. +func DefaultAgentMode(agentName string, level int) AgentMode { + if agentName == "supervisor" { + return ModeAdvisory + } + switch level { + case 1: + return ModeAdvisory + case 2: + return ModeAdvisory + case 3: + if agentName == "quality" { + return ModeIssuesAndPRs + } + return ModeAdvisory + case 4: + switch agentName { + case "quality", "sec-check", "ci-maintainer": + return ModeIssuesAndPRs + case "scanner", "guide": + return ModeIssuesOnly + default: + return ModeAdvisory + } + case 5: + return ModeIssuesAndPRs + case 6: + if agentName == "scanner" { + return ModeIssuesPRsMerge + } + return ModeIssuesAndPRs + default: + return ModeAdvisory + } +} + +// AuthorizePROpen enforces the policy for the hive-opens-PR watcher: an agent +// may open a PR (by dropping a request file) only if BOTH hold: +// +// 1. Forge-resistance — the request file's owning UID (fileUID) maps to the +// agent it claims to be (via the uid-map). One agent cannot open a PR "as" +// another, and a non-agent process (unknown UID) is refused. When per-agent +// UIDs are not in play (fileUID <= 0, e.g. shared-dev-UID mode with no map), +// ownership is unverifiable, so we fall back to the ACMM check alone rather +// than hard-failing — the same posture the credential helper takes. +// 2. ACMM write-gate — the agent must be push-capable at the hive's current +// ACMM level, i.e. exactly the CanPush() check that governs `gh pr create`. +// +// Returns nil to authorize, or an error describing the denial. This mirrors the +// direct PR path's policy so the request-file route grants no extra privilege. +func (m *Manager) AuthorizePROpen(agentName string, fileUID int) error { + if strings.TrimSpace(agentName) == "" { + return fmt.Errorf("no agent named in the request") + } + // Forge check: when we have a UID map and a real owning UID, the file owner + // must BE this agent. + if m.uidMap != nil && fileUID > 0 { + owner := m.uidMap.LookupByUID(fileUID) + if owner == "" { + return fmt.Errorf("request file owned by unknown uid %d (not a registered agent)", fileUID) + } + if owner != agentName { + return fmt.Errorf("request claims agent %q but file is owned by agent %q (uid %d)", agentName, owner, fileUID) + } + } + // ACMM write-gate: resolve the agent and check CanPush. + m.mu.RLock() + agent := m.agents[agentName] + m.mu.RUnlock() + if agent == nil { + return fmt.Errorf("unknown agent %q", agentName) + } + if !m.agentMode(agent).CanPush() { + return fmt.Errorf("agent %q is not push-capable at this ACMM level (mode %s) — advisory agents may not open PRs", + agentName, m.agentMode(agent).String()) + } + return nil +} + +// AuthorizeIssueOpen enforces the policy for the issue-request watcher, +// mirroring AuthorizePROpen with the mode gates that govern the direct gh +// paths: "issue" requests need CanCreateIssues() (mode >= ISSUES_ONLY); +// "comment" and "claim" requests need the same (commenting and claiming an +// issue are both issue-writes under the same tier). The same UID +// forge-resistance applies: the request file's owner must BE the claimed +// agent. A nil manager or unknown agent is denied. +func (m *Manager) AuthorizeIssueOpen(agentName string, fileUID int, kind string) error { + if strings.TrimSpace(agentName) == "" { + return fmt.Errorf("no agent named in the request") + } + if m.uidMap != nil && fileUID > 0 { + owner := m.uidMap.LookupByUID(fileUID) + if owner == "" { + return fmt.Errorf("request file owned by unknown uid %d (not a registered agent)", fileUID) + } + if owner != agentName { + return fmt.Errorf("request claims agent %q but file is owned by agent %q (uid %d)", agentName, owner, fileUID) + } + } + m.mu.RLock() + agent := m.agents[agentName] + m.mu.RUnlock() + if agent == nil { + return fmt.Errorf("unknown agent %q", agentName) + } + if !m.agentMode(agent).CanCreateIssues() { + return fmt.Errorf("agent %q may not create issues or comments at this ACMM level (mode %s)", + agentName, m.agentMode(agent).String()) + } + return nil +} + +// AuthorizeMerge enforces the policy for the hive-merges-PR watcher, mirroring +// AuthorizePROpen but with the stricter CanMerge() gate: the request's agent +// must own the request file (forge-resistance) AND be merge-capable at the +// hive's current ACMM level (ModeIssuesPRsMerge). This keeps the file-based +// merge relay under the exact same authority as a direct merge would require — +// an issues/PRs agent that can open PRs still cannot merge them unless its mode +// grants merge. A nil manager or unknown agent is denied. +func (m *Manager) AuthorizeMerge(agentName string, fileUID int) error { + if strings.TrimSpace(agentName) == "" { + return fmt.Errorf("no agent named in the request") + } + // Forge check: when we have a UID map and a real owning UID, the file owner + // must BE this agent. + if m.uidMap != nil && fileUID > 0 { + owner := m.uidMap.LookupByUID(fileUID) + if owner == "" { + return fmt.Errorf("request file owned by unknown uid %d (not a registered agent)", fileUID) + } + if owner != agentName { + return fmt.Errorf("request claims agent %q but file is owned by agent %q (uid %d)", agentName, owner, fileUID) + } + } + // ACMM merge-gate: resolve the agent and check CanMerge. + m.mu.RLock() + agent := m.agents[agentName] + m.mu.RUnlock() + if agent == nil { + return fmt.Errorf("unknown agent %q", agentName) + } + if !m.agentMode(agent).CanMerge() { + return fmt.Errorf("agent %q is not merge-capable at this ACMM level (mode %s) — only ISSUES_PRS_MERGE agents may merge PRs", + agentName, m.agentMode(agent).String()) + } + return nil +} + +// AgentCapabilities reports whether the named agent is ABLE — at the hive's +// current ACMM level and the agent's effective mode — to create issues, open +// PRs, and merge PRs. These are the EXACT gates AuthorizePROpen (CanPush) and +// AuthorizeMerge (CanMerge) enforce, so a hub capability badge derived from +// these can never claim a capability the merge/PR relay would actually refuse. +// ok=false when the agent is unknown to the manager (the caller then reports +// "unknown", not a false negative). Read-only under RLock. +func (m *Manager) AgentCapabilities(agentName string) (canOpenIssue, canOpenPR, canMerge, ok bool) { + m.mu.RLock() + agent, exists := m.agents[agentName] + m.mu.RUnlock() + if !exists || agent == nil { + return false, false, false, false + } + mode := m.agentMode(agent) + return mode.CanCreateIssues(), mode.CanPush(), mode.CanMerge(), true +} + +// EffectiveBackend reports the named agent's effective backend, honoring any +// runtime BackendOverride (see effectiveBackend). ok=false when the agent is +// unknown. Read-only under RLock — a small exported wrapper so callers outside +// the package (the heartbeat builder) need not reach into unexported state. +func (m *Manager) EffectiveBackend(agentName string) (backend string, ok bool) { + m.mu.RLock() + agent, exists := m.agents[agentName] + m.mu.RUnlock() + if !exists || agent == nil { + return "", false + } + return effectiveBackend(agent), true +} + +// InvocationMetadata reports the effective backend, model, and reasoning effort +// the hive invokes for the named agent, accounting for runtime overrides — the +// launch-time truth the invocation-attribution trail records (see pkg/github/attribution +// .go). ok=false when the agent is unknown to the manager (the caller then +// falls back to static config). Read-only under RLock; called from the +// PR-request watcher goroutine, never from the launch path. +func (m *Manager) InvocationMetadata(agentName string) (backend, model, effort string, ok bool) { + m.mu.RLock() + defer m.mu.RUnlock() + agent, exists := m.agents[agentName] + if !exists { + return "", "", "", false + } + backend = effectiveBackend(agent) + model = agent.Config.Model + if agent.ModelOverride != "" { + model = agent.ModelOverride + } + return backend, model, ResolveReasoningEffort(backend, model, agent.Config.ReasoningEffort), true +} + +// ResolveReasoningEffort reports the reasoning effort the hive actually launches +// a given backend/model pair with, given the agent's configured reasoning_effort. +// Exported because the attribution trail is +// resolved in TWO places — Manager.InvocationMetadata above for a running agent, +// and cmd/hive's fallback that reads straight from config when the Manager does +// not know the agent — and both must give the same answer. +// +// Before this existed the fallback carried its own hardcoded "low", so changing +// agyDefaultEffort here would have left cmd/hive silently stamping PRs with an +// effort agy was no longer being launched with. An attribution trail that +// misreports is worse than one that says nothing. +// +// The rules mirror the launch path exactly: +// - agy REQUIRES --effort whenever --model is given, so with a model it runs +// at agyLaunchEffort(configured) and with no model at no effort at all. +// - codex is launched with `-c model_reasoning_effort` only when an effort +// is configured; unset means codex's own default, which the hive does not +// resolve, so the honest answer is the configured value verbatim. +// - every other backend takes its effort from its own config, which the +// hive does not resolve here, so the honest answer is "". +func ResolveReasoningEffort(backend, model, configured string) string { + switch backend { + case "agy": + if model != "" { + return agyLaunchEffort(configured) + } + return "" + case codexBackend: + return configured + } + return "" +} diff --git a/src/pkg/agent/manager_routing.go b/src/pkg/agent/manager_routing.go new file mode 100644 index 0000000000..9ae3cc46db --- /dev/null +++ b/src/pkg/agent/manager_routing.go @@ -0,0 +1,456 @@ +package agent + +import ( + "fmt" + "os/exec" + "strings" + + "github.com/hivecommons/hive/pkg/config" +) + +// effectiveBackend returns the agent's backend accounting for any override. +func effectiveBackend(agent *AgentProcess) string { + if agent.BackendOverride != "" { + return agent.BackendOverride + } + return agent.Config.Backend +} + +// IsInferenceBackend returns true if the backend is a self-hosted inference +// backend (vllm, llm-d, litellm) rather than a CLI tool. Delegates to the +// canonical list in the config package (shared with the proxy package, +// which cannot be imported from here without a cycle). +func IsInferenceBackend(backend string) bool { + return config.IsInferenceBackend(backend) +} + +// SetInferenceCallbacks registers callbacks that the manager uses to +// configure/clear inference routing on the proxy when launching agents. +func (m *Manager) SetInferenceCallbacks( + setRoute func(agentName, backend, model string), + clearRoute func(agentName string), +) { + m.mu.Lock() + defer m.mu.Unlock() + m.inferenceRouteCallback = setRoute + m.clearInferenceRouteCallback = clearRoute +} + +// SetGatewayBackendChecker injects a predicate that reports whether a backend +// string names a configured model gateway. This makes an agent whose backend is +// a gateway name inference-routable, so its route is resolved via the inference +// callback exactly like the built-in litellm/vllm/llm-d backends. +func (m *Manager) SetGatewayBackendChecker(fn func(backend string) bool) { + // Atomic store — no m.mu — so routableBackend can read it lock-free from the + // lock-holding launch path without deadlocking (see isGatewayBackend docs). + m.isGatewayBackend.Store(&fn) +} + +// routableBackend reports whether a backend should be routed through the +// inference proxy: either a built-in inference backend, or a configured gateway +// name. Safe to call while holding m.mu (isGatewayBackend is read atomically). +func (m *Manager) routableBackend(backend string) bool { + if IsInferenceBackend(backend) { + return true + } + // Lock-free atomic read: this is invoked from the launch path while m.mu is + // already held, so it MUST NOT take m.mu (non-reentrant RWMutex → deadlock). + fnp := m.isGatewayBackend.Load() + return fnp != nil && *fnp != nil && (*fnp)(backend) +} + +// validateBackendName reports whether backend is one the launcher can actually +// dispatch: an agentic CLI, a model-gateway backend, or a configured gateway +// name. An empty backend is valid (it means "the hive default"). +// +// This is the manager-side half of the accept-then-fail fix. It dispatches on +// the SAME canonical lists as config.ValidateBackend and backendBinary, so a +// backend accepted by any write path is one the launch path can start. +// Safe to call while holding m.mu (routableBackend reads atomically). +func (m *Manager) validateBackendName(backend string) error { + if backend == "" || config.IsCLIBackend(backend) || m.routableBackend(backend) { + return nil + } + return fmt.Errorf("unsupported backend %q (supported: %s; or the name of a configured model gateway)", + backend, strings.Join(config.SupportedBackends(), ", ")) +} + +// effectiveBackend is the backend this agent will actually launch with: the +// per-agent override when set, otherwise its configured backend. +func (a *AgentProcess) effectiveBackend() string { + if a.BackendOverride != "" { + return a.BackendOverride + } + return a.Config.Backend +} + +// effectiveModel is the model this agent will actually launch with: the +// per-agent override when set, otherwise its configured model. Returns the +// raw (un-normalized) name — the audit log should show what was ASKED for, +// since a bad model name is exactly the kind of misconfiguration being +// audited. +func (a *AgentProcess) effectiveModel() string { + if a.ModelOverride != "" { + return a.ModelOverride + } + return a.Config.Model +} + +// backendBinaryAliases names the backends whose binary is NOT simply the +// backend name. Only genuine aliases belong here: every other CLI backend is +// derived from config.CLIBackends by identity, and every routable model-gateway +// backend is resolved by Manager.backendBinaryName. Keeping this map to aliases +// only is what makes the accept-then-fail class of bug structurally impossible. +var backendBinaryAliases = map[string]string{ + // pi was previously aliased to "goose", which made every pi-configured + // agent exec the goose CLI instead of pi (the backend launch command + // switch now has a real pi case). pi is a first-class CLI backend + // (config.CLIBackends includes "pi"), so identity mapping applies. +} + +// backendBinaryName maps a config-independent agent backend to the NAME of the +// CLI binary that is exec'd for it, without touching the filesystem. Split out +// from backendBinary so the "every supported backend resolves" invariant can be +// tested without requiring each CLI to be installed on the test machine. +// +// Both canonical lists are derived rather than written out here: +// +// - config.CLIBackends (claude, copilot, goose, codex, pi, bob, aider, gemini) +// each launch a binary of the same name, except for the aliases above. +// - config.InferenceBackends (vllm, llm-d, litellm, watsonx) all launch the +// SAME claude CLI, pointed at hive's local OpenAI-compatible translator via +// ANTHROPIC_BASE_URL — the backend name selects the upstream route, not the +// binary. +// +// Deriving both means a backend added to either list can never again be +// accepted by config.ValidateBackend and then rejected hours later at kick time +// with "unknown backend". Previously only InferenceBackends was derived, so +// codex and aider were valid config values that failed at launch. +func backendBinaryName(backend string) (string, error) { + binaries := make(map[string]string, len(config.CLIBackends)+len(config.InferenceBackends)) + for _, b := range config.CLIBackends { + binaries[b] = b + } + for _, b := range config.InferenceBackends { + binaries[b] = "claude" + } + for backend, binary := range backendBinaryAliases { + binaries[backend] = binary + } + + binary, ok := binaries[backend] + if !ok { + return "", fmt.Errorf("unknown backend: %s", backend) + } + return binary, nil +} + +// backendBinaryName resolves both config-independent backends and live +// configured gateway names. A gateway name validates via Manager.routableBackend, +// so the launch path must use the same predicate and route it through claude. +func (m *Manager) backendBinaryName(backend string) (string, error) { + if binary, err := backendBinaryName(backend); err == nil { + return binary, nil + } + if m != nil && m.routableBackend(backend) { + return "claude", nil + } + return "", fmt.Errorf("unknown backend: %s", backend) +} + +// backendBinary resolves an agent backend to the absolute path of the CLI +// binary that is actually exec'd for it. +func backendBinary(backend string) (string, error) { + binary, err := backendBinaryName(backend) + if err != nil { + return "", err + } + + path, err := exec.LookPath(binary) + if err != nil { + return "", fmt.Errorf("backend %s not found in PATH: %w", backend, err) + } + + return path, nil +} + +func (m *Manager) backendBinary(backend string) (string, error) { + binary, err := m.backendBinaryName(backend) + if err != nil { + return "", err + } + + path, err := exec.LookPath(binary) + if err != nil { + return "", fmt.Errorf("backend %s binary %s not found in PATH: %w", backend, binary, err) + } + + return path, nil +} + +func (m *Manager) backendLaunchFailureMessage(backend string, err error) string { + binary, nameErr := m.backendBinaryName(backend) + if nameErr != nil { + return fmt.Sprintf( + "backend %s did not launch: %v. This backend is not a supported CLI, built-in inference backend, or configured model gateway; switch this agent to a supported backend or configure a matching model gateway.", + backend, err) + } + return fmt.Sprintf( + "backend %s did not launch: %v. The %s CLI required for this backend is not installed in this hive image — upgrade the hive image or switch this agent to a different backend.", + backend, err, binary) +} + +// codexBackend is the backend name for the OpenAI Codex CLI. +const codexBackend = "codex" + +// bobBackend is the backend name for the IBM bobshell ("bob") CLI. +const bobBackend = "bob" + +// normalizeModelName converts YAML-friendly model names to the format each +// CLI backend expects. Claude CLI uses hyphens (claude-opus-4-7), while +// gemini/goose/agy-style backends use dots (claude-opus-4.7). +// +// copilot does NOT take the blind trailing-digits dot-rewrite below: the +// Copilot CLI's --model nomenclature mixes separators per model family +// (claude-fable-5 is DASHED, claude-opus-4.6 is DOTTED), so the rewrite +// corrupted every dashed-family id — verified live, copilot CLI v1.0.78 +// rejected the rewritten `claude-fable.5` ("is not available") and fell back +// to a different model (#4262). copilot instead uses the alias-based +// CanonicalizeCopilotModel (copilot_models.go), which normalizes separator +// drift against the known CLI-accepted list in both directions and passes +// unknown ids through verbatim. Applied here — at launch time — so an +// already-stored bad id self-corrects on existing spokes without operator +// action. +// +// Self-hosted inference backends (vllm, llm-d, litellm) and configured gateway +// names are the outbound gateway model id verbatim — the string must match an +// entitled model on the gateway EXACTLY (prefixes like "Azure/", dots vs +// hyphens, case). Rewriting it (e.g. "Azure/gpt-4" -> "Azure/gpt.4", +// "gpt-4o-2024-08-06" -> "gpt-4o-2024-08.06") produces a model the team is not +// entitled to and the gateway 403s ("team not allowed to access model") even +// for entitled models. So never normalize inference model names — pass them +// through untouched. +// +// bob is likewise excluded. bobLaunchCmd passes no --model at all (bob +// auto-selects), so this is defense-in-depth rather than the fix: the value is +// still computed and logged on the bob launch path, and the dot-rewrite is +// what turned a configured `claude-sonnet-4-6` into the unknown +// `claude-sonnet-4.6` that made bob die with "Cannot read properties of +// undefined (reading 'maxTokens')". Leaving it unrewritten keeps logs honest +// about what was configured and stops the corrupted id from being handed to a +// future bob consumer. +func normalizeModelNameForBackend(model, backend string, inferenceRoutable bool) string { + if backend == "claude" || backend == bobBackend || inferenceRoutable { + return model + } + if backend == "copilot" { + return CanonicalizeCopilotModel(model) + } + idx := strings.LastIndex(model, "-") + if idx < 0 || idx == len(model)-1 { + return model + } + suffix := model[idx+1:] + allDigits := true + for _, c := range suffix { + if c < '0' || c > '9' { + allDigits = false + break + } + } + if allDigits { + return model[:idx] + "." + suffix + } + return model +} + +func (m *Manager) PinCLI(name, version string) error { + m.mu.Lock() + defer m.mu.Unlock() + + agent, ok := m.agents[name] + if !ok { + return fmt.Errorf("agent %s not found", name) + } + + agent.PinnedCLI = version + m.logger.Info("agent CLI pinned", "name", name, "version", version) + return nil +} + +func (m *Manager) UnpinCLI(name string) error { + m.mu.Lock() + defer m.mu.Unlock() + + agent, ok := m.agents[name] + if !ok { + return fmt.Errorf("agent %s not found", name) + } + + agent.PinnedCLI = "" + m.logger.Info("agent CLI unpinned", "name", name) + return nil +} + +func (m *Manager) PinModel(name, model string) error { + m.mu.Lock() + defer m.mu.Unlock() + + agent, ok := m.agents[name] + if !ok { + return fmt.Errorf("agent %s not found", name) + } + + prevModel := agent.effectiveModel() + agent.PinnedModel = model + agent.ModelOverride = model + m.logger.Info("agent model pinned", "name", name, "model", model) + if prevModel != model { + m.audit(AuditAgentModelSet, name, auditFields( + "outcome", "success", + "backend", agent.effectiveBackend(), + "model", model, + "previous_model", prevModel, + "trigger", "pin", + )) + } + return nil +} + +func (m *Manager) UnpinModel(name string) error { + m.mu.Lock() + defer m.mu.Unlock() + + agent, ok := m.agents[name] + if !ok { + return fmt.Errorf("agent %s not found", name) + } + + agent.PinnedModel = "" + m.logger.Info("agent model unpinned", "name", name) + return nil +} + +func (m *Manager) SetModelOverride(name, model string) error { + m.mu.Lock() + defer m.mu.Unlock() + + agent, ok := m.agents[name] + if !ok { + return fmt.Errorf("agent %s not found", name) + } + + // Store the CLI-accepted spelling for copilot so the persisted selection, + // the dropdown preselect, and auto-heal all agree on one canonical id + // (separator drift like claude-fable.5 vs claude-fable-5, #4262). Launch + // re-applies the same canonicalization, so even ids stored before this + // existed self-correct there. + if agent.effectiveBackend() == "copilot" { + model = CanonicalizeCopilotModel(model) + } + + // A pin blocks the governor's auto-selection, never a user's explicit + // switch: retarget the pin to the new model so the pin state is + // unchanged (still pinned) while the change takes effect. + if agent.PinnedModel != "" { + agent.PinnedModel = model + m.logger.Info("agent model pin retargeted by user switch", "name", name, "model", model) + } + + prevModel := agent.effectiveModel() + agent.ModelOverride = model + m.logger.Info("agent model override set", "name", name, "model", model) + // State CHANGES only — the governor re-asserts the current model on every + // evaluation cycle, so auditing unchanged writes would flood the ring. + if prevModel != model { + m.audit(AuditAgentModelSet, name, auditFields( + "outcome", "success", + "backend", agent.effectiveBackend(), + "model", model, + "previous_model", prevModel, + )) + } + + effectiveBackend := agent.Config.Backend + if agent.BackendOverride != "" { + effectiveBackend = agent.BackendOverride + } + if m.routableBackend(effectiveBackend) && m.inferenceRouteCallback != nil { + m.inferenceRouteCallback(name, effectiveBackend, model) + } + return nil +} + +func (m *Manager) SetBackendOverride(name, backend string) error { + m.mu.Lock() + defer m.mu.Unlock() + + agent, ok := m.agents[name] + if !ok { + return fmt.Errorf("agent %s not found", name) + } + + // Refuse a backend the launcher cannot dispatch, at SET time. Previously + // any string was accepted here and the agent was then restarted into it, + // failing only at launch with "unknown backend: " — the agent stops + // working and the operator gets no signal at the moment of the change. + // routableBackend covers configured gateway names (resolved live), so a + // gateway-named backend still passes. + if err := m.validateBackendName(backend); err != nil { + return err + } + + // Captured after validation so a rejected switch records no audit event: + // the override is only mutated below, once the backend is known routable. + prevBackend := agent.effectiveBackend() + agent.BackendOverride = backend + m.logger.Info("agent backend override set", "name", name, "backend", backend) + // Record only a real transition: /switch/{backend} is also re-applied on + // config reload with the value already in effect, and auditing those + // no-ops would bury the actual operator changes. + if prevBackend != backend { + m.audit(AuditAgentBackendSet, name, auditFields( + "outcome", "success", + "backend", backend, + "model", agent.effectiveModel(), + "previous_backend", prevBackend, + )) + } + + if m.routableBackend(backend) && m.inferenceRouteCallback != nil { + model := agent.ModelOverride + if model == "" { + model = agent.Config.Model + } + m.inferenceRouteCallback(name, backend, model) + } else if !IsInferenceBackend(backend) && m.clearInferenceRouteCallback != nil { + m.clearInferenceRouteCallback(name) + } + return nil +} + +// RefreshInferenceRoutes re-fires the inference route callback for every +// agent whose effective backend matches, so endpoint or credential changes +// (e.g. a governor LiteLLM config save) take effect on live agents without +// a restart. +func (m *Manager) RefreshInferenceRoutes(backend string) { + m.mu.Lock() + defer m.mu.Unlock() + if m.inferenceRouteCallback == nil || !m.routableBackend(backend) { + return + } + for name, agent := range m.agents { + effective := agent.Config.Backend + if agent.BackendOverride != "" { + effective = agent.BackendOverride + } + if effective != backend { + continue + } + model := agent.ModelOverride + if model == "" { + model = agent.Config.Model + } + m.inferenceRouteCallback(name, backend, model) + } +} diff --git a/src/pkg/agent/manager_thrash.go b/src/pkg/agent/manager_thrash.go new file mode 100644 index 0000000000..b8d1b94639 --- /dev/null +++ b/src/pkg/agent/manager_thrash.go @@ -0,0 +1,100 @@ +package agent + +import ( + "fmt" + "strings" + "time" +) + +// Blocked-action thrash breaker: an agent that keeps hammering a policy wall +// (e.g. a push with no per-agent token, blocked every ~3s by +// git-credential-hive, or a proxy hard-deny) burns model tokens indefinitely +// with zero possible output — observed live 2026-08-04 on a hosted L2 hive +// whose guide agent retried a blocked push every 3 seconds. (Since #4289, +// ADVISORY-mode pushes are no longer blocked by the credential helper — the +// read-only token is served and GitHub rejects the push with 403 — but the +// helper still emits "git push blocked:" for unknown-UID and missing-token +// failures, which this breaker continues to catch.) The hub, not the model, +// breaks the loop: thrashThreshold blocked-action lines within thrashWindow +// pauses the session (visible, reversible, stops governor kicks) with the +// reason spelled out. +const ( + thrashWindow = 60 * time.Second + thrashThreshold = 5 + thrashCooldown = 10 * time.Minute +) + +// blockedActionMarkers are the policy-wall stderr lines that can never +// succeed by retrying. Keep in sync with bin/git-credential-hive.sh and the +// proxy's hard-deny responses. +var blockedActionMarkers = []string{ + "git push blocked:", + "blocked by hive policy", +} + +type thrashState struct { + times []time.Time + lastTrip time.Time +} + +// checkBlockedThrash records a blocked-action output line for the agent and, +// past the threshold, pauses the agent asynchronously (never inline: this is +// called from the output-capture goroutine and Pause takes m.mu). +func (m *Manager) checkBlockedThrash(agent, line string) { + matched := false + for _, marker := range blockedActionMarkers { + if strings.Contains(line, marker) { + matched = true + break + } + } + if !matched { + return + } + now := time.Now() + m.thrashMu.Lock() + if m.thrash == nil { + m.thrash = map[string]*thrashState{} + } + st := m.thrash[agent] + if st == nil { + st = &thrashState{} + m.thrash[agent] = st + } + trip := recordBlockedAndCheck(st, now, thrashWindow, thrashThreshold, thrashCooldown) + m.thrashMu.Unlock() + if !trip { + return + } + reason := fmt.Sprintf("blocked-action loop: %d+ policy-blocked attempts in %s — the block is terminal in this mode; paused to stop token burn", thrashThreshold, thrashWindow) + m.logger.Warn("thrash breaker tripped", "agent", agent, "line", truncateStr(line, 160)) + go func() { + if err := m.Pause(agent, "thrash-breaker", reason); err != nil { + m.logger.Warn("thrash breaker pause failed", "agent", agent, "error", err) + } + }() +} + +// recordBlockedAndCheck is the pure sliding-window decision: append now, drop +// entries older than window, and report whether the threshold is crossed +// outside the cooldown. Split out for direct unit testing. +func recordBlockedAndCheck(st *thrashState, now time.Time, window time.Duration, threshold int, cooldown time.Duration) bool { + st.times = append(st.times, now) + cutoff := now.Add(-window) + kept := st.times[:0] + for _, t := range st.times { + if t.After(cutoff) { + kept = append(kept, t) + } + } + st.times = kept + if len(st.times) < threshold { + return false + } + if !st.lastTrip.IsZero() && now.Sub(st.lastTrip) < cooldown { + return false + } + st.lastTrip = now + st.times = nil + return true +} diff --git a/src/pkg/agent/ringbuffer.go b/src/pkg/agent/ringbuffer.go index 43270c5831..2f86f7251d 100644 --- a/src/pkg/agent/ringbuffer.go +++ b/src/pkg/agent/ringbuffer.go @@ -51,7 +51,6 @@ func (r *RingBuffer) Last(n int) []string { return result } - func (r *RingBuffer) Count() int { r.mu.RLock() defer r.mu.RUnlock() diff --git a/src/pkg/agent/routing_coverage_test.go b/src/pkg/agent/routing_coverage_test.go index bbfb3224ba..0d22c08b9b 100644 --- a/src/pkg/agent/routing_coverage_test.go +++ b/src/pkg/agent/routing_coverage_test.go @@ -303,14 +303,14 @@ func TestNormalizeModelName(t *testing.T) { // ids are canonicalized against the CLI-accepted list and UNKNOWN ids // pass through verbatim — the rewrite is what corrupted the known // claude-fable-5 into the CLI-rejected claude-fable.5. - {"gpt-4o-2024", "copilot", "gpt-4o-2024"}, // unknown id passthrough - {"claude-fable-5", "copilot", "claude-fable-5"}, // dashed family kept dashed - {"claude-fable.5", "copilot", "claude-fable-5"}, // stored bad id self-corrects + {"gpt-4o-2024", "copilot", "gpt-4o-2024"}, // unknown id passthrough + {"claude-fable-5", "copilot", "claude-fable-5"}, // dashed family kept dashed + {"claude-fable.5", "copilot", "claude-fable-5"}, // stored bad id self-corrects {"claude-opus-4-6", "copilot", "claude-opus-4.6"}, // known dotted family still mapped - {"gpt-4o-2024", "gemini", "gpt-4o.2024"}, // digit suffix -> dot (non-copilot) - {"gpt-4o-preview", "copilot", "gpt-4o-preview"}, // non-digit suffix unchanged - {"nohyphen", "copilot", "nohyphen"}, // no hyphen - {"trailing-", "copilot", "trailing-"}, // trailing hyphen + {"gpt-4o-2024", "gemini", "gpt-4o.2024"}, // digit suffix -> dot (non-copilot) + {"gpt-4o-preview", "copilot", "gpt-4o-preview"}, // non-digit suffix unchanged + {"nohyphen", "copilot", "nohyphen"}, // no hyphen + {"trailing-", "copilot", "trailing-"}, // trailing hyphen } for _, c := range cases { if got := normalizeModelName(c.model, c.backend); got != c.want { From 33200bdbc1a36607366fde700f40eaf7bed4cf5c Mon Sep 17 00:00:00 2001 From: hive-release-bot Date: Fri, 18 Sep 2026 03:49:40 +0000 Subject: [PATCH 13/17] =?UTF-8?q?=F0=9F=94=96=20release:=20v4.55.0?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Automated release commit. Compiles changelog.d/ fragments and moves the CHANGELOG.md Unreleased section into a dated v4.55.0 entry. See src/docs/releases.md. Signed-off-by: hive-release-bot --- CHANGELOG.md | 20 +++++++++++++++++++ changelog.d/added-review-breadth-first.md | 1 - .../added-reviewer-routes-to-humans.md | 1 - .../changed-7303-manager-domain-split.md | 1 - ...hanged-7515-blocked-pill-names-the-rule.md | 1 - ...7-contribute-ingress-auth-url-reconcile.md | 1 - .../security-6287-token-access-audit-log.md | 1 - 7 files changed, 20 insertions(+), 6 deletions(-) delete mode 100644 changelog.d/added-review-breadth-first.md delete mode 100644 changelog.d/added-reviewer-routes-to-humans.md delete mode 100644 changelog.d/changed-7303-manager-domain-split.md delete mode 100644 changelog.d/changed-7515-blocked-pill-names-the-rule.md delete mode 100644 changelog.d/fixed-7517-contribute-ingress-auth-url-reconcile.md delete mode 100644 changelog.d/security-6287-token-access-audit-log.md diff --git a/CHANGELOG.md b/CHANGELOG.md index f82b35005b..36e6e8412b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -11,6 +11,26 @@ Hive did not historically maintain a complete changelog. This file starts a prag ## Unreleased +## 2026-09-18 (v4.55.0) + +### Added + +- Added `review.max_perspectives_per_pr`, an opt-in cap on how many review perspectives one PR may be given per dispatch cycle. Without it the parallel review budget is spent in PR order, so the head of a deep queue absorbs every slot and adding reviewers buys more opinions on one PR instead of coverage across many. Capping spends the same budget breadth-first, and loses no coverage because skipped perspectives are still dispatched on later cycles. Zero (the default) keeps the existing fan-out behavior. +- Review comments now route: a `requires_human` or `reject` verdict opens with a pinned `**HUMAN DECISION NEEDED**` marker maintainers can filter on, mentions the PR author when that author is a person (never an app/bot account, which notifies nobody), and asks the reviewer to state plainly what it could not judge instead of approving around it. + +### Changed + +- Split six more domains out of `pkg/agent/manager.go` (consent, copilot auth, env, modes, routing, thrash) into per-domain files matching the v5 layout; manager.go shrinks from 4,321 to 1,793 lines. No behavior change — verbatim block moves. +- Hovering a blocked PR pill on the dashboard now names the branch-protection rule that is holding the PR, instead of GitHub's one-word `blocked` state ([#7515](https://github.com/hivecommons/hive/issues/7515)). GitHub folds every unsatisfied rule — a red required check, a required check that never ran, a missing approval, a reviewer who asked for changes — into that single word, so an operator had to open the PR on GitHub to learn which one, the trip the pill exists to save. The sweep now collects GitHub's own review decision (one GraphQL query per repository per cycle, not one per PR) and compares the base branch's required-check set against the check runs it already walks, and the tooltip reads `blocked — changes requested by @reviewer`, `blocked — required check "validate" has not reported`, or `blocked — required check "build" is failing`. When the facts on hand genuinely do not identify a rule the tooltip still says so plainly rather than guessing, and on a repository that reports its required contexts as commit statuses no "has not reported" claim is made at all. + +### Fixed + +- Hub: the `auth-url` / `auth-response-headers` annotations [#7457](https://github.com/hivecommons/hive/pull/7457) added to the `hive-contribute` Ingress now reach hosted spokes that were provisioned before that fix ([#7517](https://github.com/hivecommons/hive/issues/7517)). The provisioning template is applied only when a hive is created, so on every pre-existing spoke nginx never asked the hub who was calling and `/api/contribute/me` kept answering `401` to the hive's own signed-in owner — the #7453 symptom, on a spoke whose `served_sha` was well past the fix. A new 15-minute hub sweep (`contribute_ingress_reconcile.go`, alongside the NET_ADMIN and per-hive-env reconciles) reads each hosted spoke's live `hive-contribute` Ingress on nginx clusters and merge-patches the two annotations on when they are missing or stale; a converged Ingress is a no-op, the patch rolls no pod, and the expected values are pinned equal to what the template renders so the sweep and provisioning cannot disagree. OpenShift-Route and pull-only clusters are skipped. `src/docs/security-model.md` now records the general rule: a template change to an existing object needs a reconcile path or a re-provision note. + +### Security + +- The token-access audit log behind `GET /api/token-access` is no longer writable by the agents it audits ([#6287](https://github.com/hivecommons/hive/issues/6287)). The log records every gh CLI command and git credential lookup an agent makes and is gated at owner role for exactly that reason, yet it was fed by the audited parties: `bin/gh-wrapper.sh` and `bin/git-credential-hive.sh` run as the agent UID and appended straight into `/var/run/hive-metrics/token-access.jsonl`, which only works when the file is writable by every agent (dev:node 0664, and every agent's primary group is node). Append is indistinguishable from write at the permission level, so a compromised or prompt-injected agent could truncate the trail of its own token use, rewrite lines to blame a peer, or forge entries outright. The wrappers now drop one JSON event per call into `/var/run/hive-metrics/token-access-events`, a drop-box the agents can create files in but cannot list (0730, sticky, setgid), and the hive process ingests those events into the log, which is now hive-owned 0600 with no group or other bit at all. On ingest the hive replaces each event's self-reported `uid` with the uid that owns the event file, the same trust anchor the PR-request watcher uses, and keeps a disagreeing claim beside it as `claimed_uid`, so an entry forged in a peer's name lands attributed to the forger. Malformed or oversized events are rejected rather than appended. A 0664 log left by an older image is tightened on every boot and on every ingest pass instead of trusted. What remains open is narrower than before: an agent can still delete its own event during the two-second window before ingest, and can still flood the log with real calls; it can no longer alter anything already recorded. + ## 2026-09-18 (v4.54.2) ### Fixed diff --git a/changelog.d/added-review-breadth-first.md b/changelog.d/added-review-breadth-first.md deleted file mode 100644 index c50bfca39b..0000000000 --- a/changelog.d/added-review-breadth-first.md +++ /dev/null @@ -1 +0,0 @@ -- Added `review.max_perspectives_per_pr`, an opt-in cap on how many review perspectives one PR may be given per dispatch cycle. Without it the parallel review budget is spent in PR order, so the head of a deep queue absorbs every slot and adding reviewers buys more opinions on one PR instead of coverage across many. Capping spends the same budget breadth-first, and loses no coverage because skipped perspectives are still dispatched on later cycles. Zero (the default) keeps the existing fan-out behavior. diff --git a/changelog.d/added-reviewer-routes-to-humans.md b/changelog.d/added-reviewer-routes-to-humans.md deleted file mode 100644 index f1b1a9981a..0000000000 --- a/changelog.d/added-reviewer-routes-to-humans.md +++ /dev/null @@ -1 +0,0 @@ -- Review comments now route: a `requires_human` or `reject` verdict opens with a pinned `**HUMAN DECISION NEEDED**` marker maintainers can filter on, mentions the PR author when that author is a person (never an app/bot account, which notifies nobody), and asks the reviewer to state plainly what it could not judge instead of approving around it. diff --git a/changelog.d/changed-7303-manager-domain-split.md b/changelog.d/changed-7303-manager-domain-split.md deleted file mode 100644 index bba77b803d..0000000000 --- a/changelog.d/changed-7303-manager-domain-split.md +++ /dev/null @@ -1 +0,0 @@ -- Split six more domains out of `pkg/agent/manager.go` (consent, copilot auth, env, modes, routing, thrash) into per-domain files matching the v5 layout; manager.go shrinks from 4,321 to 1,793 lines. No behavior change — verbatim block moves. diff --git a/changelog.d/changed-7515-blocked-pill-names-the-rule.md b/changelog.d/changed-7515-blocked-pill-names-the-rule.md deleted file mode 100644 index 5da4836afe..0000000000 --- a/changelog.d/changed-7515-blocked-pill-names-the-rule.md +++ /dev/null @@ -1 +0,0 @@ -- Hovering a blocked PR pill on the dashboard now names the branch-protection rule that is holding the PR, instead of GitHub's one-word `blocked` state ([#7515](https://github.com/hivecommons/hive/issues/7515)). GitHub folds every unsatisfied rule — a red required check, a required check that never ran, a missing approval, a reviewer who asked for changes — into that single word, so an operator had to open the PR on GitHub to learn which one, the trip the pill exists to save. The sweep now collects GitHub's own review decision (one GraphQL query per repository per cycle, not one per PR) and compares the base branch's required-check set against the check runs it already walks, and the tooltip reads `blocked — changes requested by @reviewer`, `blocked — required check "validate" has not reported`, or `blocked — required check "build" is failing`. When the facts on hand genuinely do not identify a rule the tooltip still says so plainly rather than guessing, and on a repository that reports its required contexts as commit statuses no "has not reported" claim is made at all. diff --git a/changelog.d/fixed-7517-contribute-ingress-auth-url-reconcile.md b/changelog.d/fixed-7517-contribute-ingress-auth-url-reconcile.md deleted file mode 100644 index 8e5f59ed02..0000000000 --- a/changelog.d/fixed-7517-contribute-ingress-auth-url-reconcile.md +++ /dev/null @@ -1 +0,0 @@ -- Hub: the `auth-url` / `auth-response-headers` annotations [#7457](https://github.com/hivecommons/hive/pull/7457) added to the `hive-contribute` Ingress now reach hosted spokes that were provisioned before that fix ([#7517](https://github.com/hivecommons/hive/issues/7517)). The provisioning template is applied only when a hive is created, so on every pre-existing spoke nginx never asked the hub who was calling and `/api/contribute/me` kept answering `401` to the hive's own signed-in owner — the #7453 symptom, on a spoke whose `served_sha` was well past the fix. A new 15-minute hub sweep (`contribute_ingress_reconcile.go`, alongside the NET_ADMIN and per-hive-env reconciles) reads each hosted spoke's live `hive-contribute` Ingress on nginx clusters and merge-patches the two annotations on when they are missing or stale; a converged Ingress is a no-op, the patch rolls no pod, and the expected values are pinned equal to what the template renders so the sweep and provisioning cannot disagree. OpenShift-Route and pull-only clusters are skipped. `src/docs/security-model.md` now records the general rule: a template change to an existing object needs a reconcile path or a re-provision note. diff --git a/changelog.d/security-6287-token-access-audit-log.md b/changelog.d/security-6287-token-access-audit-log.md deleted file mode 100644 index 38276d1eed..0000000000 --- a/changelog.d/security-6287-token-access-audit-log.md +++ /dev/null @@ -1 +0,0 @@ -- The token-access audit log behind `GET /api/token-access` is no longer writable by the agents it audits ([#6287](https://github.com/hivecommons/hive/issues/6287)). The log records every gh CLI command and git credential lookup an agent makes and is gated at owner role for exactly that reason, yet it was fed by the audited parties: `bin/gh-wrapper.sh` and `bin/git-credential-hive.sh` run as the agent UID and appended straight into `/var/run/hive-metrics/token-access.jsonl`, which only works when the file is writable by every agent (dev:node 0664, and every agent's primary group is node). Append is indistinguishable from write at the permission level, so a compromised or prompt-injected agent could truncate the trail of its own token use, rewrite lines to blame a peer, or forge entries outright. The wrappers now drop one JSON event per call into `/var/run/hive-metrics/token-access-events`, a drop-box the agents can create files in but cannot list (0730, sticky, setgid), and the hive process ingests those events into the log, which is now hive-owned 0600 with no group or other bit at all. On ingest the hive replaces each event's self-reported `uid` with the uid that owns the event file, the same trust anchor the PR-request watcher uses, and keeps a disagreeing claim beside it as `claimed_uid`, so an entry forged in a peer's name lands attributed to the forger. Malformed or oversized events are rejected rather than appended. A 0664 log left by an older image is tightened on every boot and on every ingest pass instead of trusted. What remains open is narrower than before: an agent can still delete its own event during the two-second window before ingest, and can still flood the log with real calls; it can no longer alter anything already recorded. From 24a67b07f25a5d751db3f339fe9cf56b9cb323fe Mon Sep 17 00:00:00 2001 From: "kubestellar-hive[bot]" <280983584+kubestellar-hive[bot]@users.noreply.github.com> Date: Thu, 17 Sep 2026 23:51:28 -0400 Subject: [PATCH 14/17] [quality] make copilot SDK-helper probe tests hermetic (#7535) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit TestProbeCopilotModelsSDK_AbsentHelperYieldsSentinel and TestProbeCopilotModelsSDK_HelperNotInstalled exec'd whatever lived at the production helper path (/usr/local/bin/copilot-models.mjs). They handled two states — helper absent (assert sentinel) or helper works (skip) — but not the third: helper installed without stored copilot auth, the normal state on live agent hosts, where the helper exits 1 ('Not authenticated'), the sentinel check fatals, and the entire pkg/dashboard suite goes red. Fix: copilotSDKHelperPath becomes a var with an in-package test seam (setCopilotSDKHelperPathForTest, mirroring knowledge.SetBaseDirForTest). Both tests repoint it at a nonexistent temp path so the absence assertion holds on every host, and a new TestProbeCopilotModelsSDK_FailingHelperIsNotAbsent covers the present-but-failing exec path (#7365's signature) with a stub script, hermetically. No production behavior change. Signed-off-by: hive-quality[bot] Co-authored-by: hive-quality[bot] Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../fixed-quality-sdk-helper-test-hermetic.md | 1 + src/pkg/dashboard/cli_models.go | 36 +++++++++----- .../dashboard/cli_models_sdk_severity_test.go | 47 ++++++++++++++++--- src/pkg/dashboard/cli_models_sdk_test.go | 11 +++-- 4 files changed, 73 insertions(+), 22 deletions(-) create mode 100644 changelog.d/fixed-quality-sdk-helper-test-hermetic.md diff --git a/changelog.d/fixed-quality-sdk-helper-test-hermetic.md b/changelog.d/fixed-quality-sdk-helper-test-hermetic.md new file mode 100644 index 0000000000..c4a5d1ff2c --- /dev/null +++ b/changelog.d/fixed-quality-sdk-helper-test-hermetic.md @@ -0,0 +1 @@ +- Tests: the copilot SDK-helper probe tests in `pkg/dashboard` are now hermetic. `TestProbeCopilotModelsSDK_AbsentHelperYieldsSentinel` and `TestProbeCopilotModelsSDK_HelperNotInstalled` used to run whatever was installed at the production helper path, so on any host that ships `copilot-models.mjs` without stored copilot auth (every live agent host) the "absent helper" sentinel never fired and the whole `pkg/dashboard` suite went red with `Not authenticated`. The helper path is now a test-seam var (`setCopilotSDKHelperPathForTest`, mirroring the `knowledge.SetBaseDirForTest` convention) pointed at a temp path, and a new `TestProbeCopilotModelsSDK_FailingHelperIsNotAbsent` covers the previously untestable-on-CI third state — helper present but exiting nonzero (the #7365 case) — via a stub script instead of the real helper. diff --git a/src/pkg/dashboard/cli_models.go b/src/pkg/dashboard/cli_models.go index 300d6105e3..6ace0c6197 100644 --- a/src/pkg/dashboard/cli_models.go +++ b/src/pkg/dashboard/cli_models.go @@ -114,17 +114,6 @@ const ( // require a plausible editor identifier. copilotEditorVersion = "vscode/1.99.0" - // copilotSDKHelperPath is where the image installs the Node helper that - // lists Copilot models through the official @github/copilot-sdk (source: - // bin/copilot-models.mjs, COPYed by src/Dockerfile). The SDK spawns the - // pinned copilot CLI as a JSON-RPC server, so the probe rides the CLI's - // own stored auth and TLS handling — auth configurations the raw-HTTP - // probe below cannot reach (verified live against copilot CLI 1.0.59 in a - // hive pod: 25 models via stored device-flow auth behind the egress - // proxy). Absent outside the image, in which case the SDK probe is - // skipped instantly. - copilotSDKHelperPath = "/usr/local/bin/copilot-models.mjs" - // copilotSDKNodeBinary runs the helper. The helper is plain-Node ESM; it // is invoked explicitly (not via shebang) so no exec bit is needed. copilotSDKNodeBinary = "node" @@ -269,6 +258,31 @@ const ( // it at hermetic servers and exercise fallback branches without network. var copilotUserEndpointURL = "https://api.github.com/copilot_internal/user" +// copilotSDKHelperPath is where the image installs the Node helper that +// lists Copilot models through the official @github/copilot-sdk (source: +// bin/copilot-models.mjs, COPYed by src/Dockerfile). The SDK spawns the +// pinned copilot CLI as a JSON-RPC server, so the probe rides the CLI's +// own stored auth and TLS handling — auth configurations the raw-HTTP +// probe below cannot reach (verified live against copilot CLI 1.0.59 in a +// hive pod: 25 models via stored device-flow auth behind the egress +// proxy). Absent outside the image, in which case the SDK probe is +// skipped instantly. A var (not const) only so tests can point it at a +// hermetic path regardless of what the host has installed; production +// never reassigns it. +var copilotSDKHelperPath = "/usr/local/bin/copilot-models.mjs" + +// setCopilotSDKHelperPathForTest repoints the SDK helper script path for the +// lifetime of t, mirroring the knowledge.SetBaseDirForTest convention. +func setCopilotSDKHelperPathForTest(t interface { + Helper() + Cleanup(func()) +}, path string) { + t.Helper() + old := copilotSDKHelperPath + copilotSDKHelperPath = path + t.Cleanup(func() { copilotSDKHelperPath = old }) +} + // --- Static fallback lists (kept CURRENT — July 2026) --- // claudeStaticModels is the fallback offered when the Claude models probe diff --git a/src/pkg/dashboard/cli_models_sdk_severity_test.go b/src/pkg/dashboard/cli_models_sdk_severity_test.go index bfd73a33d4..cafc565c52 100644 --- a/src/pkg/dashboard/cli_models_sdk_severity_test.go +++ b/src/pkg/dashboard/cli_models_sdk_severity_test.go @@ -15,6 +15,9 @@ import ( "context" "errors" "fmt" + "os" + "os/exec" + "path/filepath" "strings" "testing" ) @@ -39,15 +42,45 @@ func TestCopilotSDKHelperAbsentIsASentinel(t *testing.T) { // The real exec path must produce the sentinel when the helper script is not // installed, otherwise the sentinel is dead code and every dev machine starts -// emitting a WARN. +// emitting a WARN. The helper path is repointed at a path that does not exist +// so the assertion holds regardless of whether the host image ships the real +// helper (on live agent hosts it exists but is unauthenticated, which used to +// flip this test to a hard FAIL — the third state the old version, which ran +// whatever was at the production path, never accounted for). func TestProbeCopilotModelsSDK_AbsentHelperYieldsSentinel(t *testing.T) { - if _, err := execCopilotSDKHelper(context.Background(), ""); err != nil { - if !errors.Is(err, errCopilotSDKHelperAbsent) { - t.Fatalf("helper absence did not yield the sentinel: %v", err) - } - return + setCopilotSDKHelperPathForTest(t, filepath.Join(t.TempDir(), "copilot-models.mjs")) + _, err := execCopilotSDKHelper(context.Background(), "") + if err == nil { + t.Fatal("exec of a nonexistent helper unexpectedly succeeded") + } + if !errors.Is(err, errCopilotSDKHelperAbsent) { + t.Fatalf("helper absence did not yield the sentinel: %v", err) + } +} + +// The converse on the same exec path: a helper that EXISTS and fails (#7365 — +// exits nonzero) must NOT match the absence sentinel, or the failure would be +// logged at INFO and stay invisible. Uses a stub script so the assertion never +// depends on the host's real helper or its auth state. +func TestProbeCopilotModelsSDK_FailingHelperIsNotAbsent(t *testing.T) { + if _, err := exec.LookPath("node"); err != nil { + t.Skip("node not on PATH; cannot exercise the helper exec path") + } + stub := filepath.Join(t.TempDir(), "copilot-models.mjs") + if err := os.WriteFile(stub, []byte("console.error('stub failure'); process.exit(1);\n"), 0o644); err != nil { + t.Fatal(err) + } + setCopilotSDKHelperPathForTest(t, stub) + _, err := execCopilotSDKHelper(context.Background(), "") + if err == nil { + t.Fatal("failing stub helper unexpectedly succeeded") + } + if errors.Is(err, errCopilotSDKHelperAbsent) { + t.Fatalf("a real helper failure matched the absence sentinel: %v", err) + } + if !strings.Contains(err.Error(), "stub failure") { + t.Errorf("helper stderr not folded into the error: %v", err) } - t.Skip("copilot SDK helper present on this machine; skipping absence check") } // The severity split itself, asserted on real emitted log records. diff --git a/src/pkg/dashboard/cli_models_sdk_test.go b/src/pkg/dashboard/cli_models_sdk_test.go index b2138af1bc..99d7df3af3 100644 --- a/src/pkg/dashboard/cli_models_sdk_test.go +++ b/src/pkg/dashboard/cli_models_sdk_test.go @@ -3,6 +3,7 @@ package dashboard import ( "context" "errors" + "path/filepath" "testing" ) @@ -145,13 +146,15 @@ func TestQueryCLIModels_SDKResultsFeedRetention(t *testing.T) { // TestProbeCopilotModelsSDK_HelperNotInstalled verifies the real exec path // degrades instantly (no hang, no panic) when the helper script is absent — -// the situation on dev machines and CI runners. +// the situation on dev machines and CI runners. The helper path is repointed +// at a nonexistent file so the check is hermetic on hosts that DO ship the +// helper (where the old version exec'd the real helper — a network-dependent +// probe whose outcome tracked the host's copilot auth state). func TestProbeCopilotModelsSDK_HelperNotInstalled(t *testing.T) { + setCopilotSDKHelperPathForTest(t, filepath.Join(t.TempDir(), "copilot-models.mjs")) s := &Server{logger: testLogger()} if _, err := s.probeCopilotModelsSDK(""); err == nil { - // The helper is installed only inside the hive image; if this machine - // actually has it, the probe may legitimately succeed — skip then. - t.Skip("copilot SDK helper present on this machine; skipping absence check") + t.Fatal("probe with an absent helper unexpectedly succeeded") } } From 99f25504328a7a3bab10f196a2156ac8f270daf7 Mon Sep 17 00:00:00 2001 From: Andy Anderson Date: Fri, 18 Sep 2026 00:03:00 -0400 Subject: [PATCH 15/17] =?UTF-8?q?=F0=9F=8C=B1=20cmd/hive:=20extract=20five?= =?UTF-8?q?=20business-logic=20domains=20from=20main.go=20(9,795=20?= =?UTF-8?q?=E2=86=92=208,072=20lines)=20(#7510)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * 🌱 cmd/hive: extract five domains from main.go into their own files main.go was 9,795 lines of package main, mixing unrelated business logic with the process entrypoint. The file this issue is named after, main_helpers.go, is already gone (dissolved across #7241..#7308), but the other half of #7238 — "package main concentrates business logic" — still held: main.go alone was 9,795 of the package's 12,059 non-test lines. This is a stage. Five self-contained domains move out, each into one file in the same package main: appkeyfile.go GitHub App private-key file management: where per-App-ID and per-hive key files live, how a key is written, resolved, fingerprinted, and how a missing key is described. selfupgrade.go self-upgrade bookkeeping: the upgrade marker and last-outcome files, the retry/backoff budget, and the boot-time reconciliation of the two. login_scan.go login-required scanning: detecting a backend dropping to an interactive login prompt and debouncing that across sightings so one noisy line cannot pause an agent. merge_eligibility.go merge eligibility: hold gates, required-check state, bucket classification, the trusted-merger authz binding, and the merge-eligible report. intent_verdicts.go intent verdicts: PR/issue evidence gathering, alignment summary, the advisory record, and the intent-verdicts report. main.go: 9,795 -> 8,072 lines. The diff on main.go is 1,723 deletions and 0 additions: nothing was edited in place. Pure code motion. Declarations were moved byte-identically, with their doc comments; nothing was renamed, reformatted, or "improved". Verified with a structural declaration diff that parses every top-level func (including methods), type, const and var — including members of grouped const(...) and var(...) blocks — from the original main.go and from the union of the six resulting files, keyed by name and compared byte-exactly: BEFORE files: 1 declarations: 242 AFTER files: 6 declarations: 242 LOST: 0 ADDED: 0 CHANGED: 0 DUPLICATE: 0 RESULT: PURE CODE MOTION Cross-checked against go/ast: 241 unique declaration names before and after, identical sets. One grouped var block (ciFailingPath, intentVerdictsPath) is deliberately left in main.go: its two members belong to different domains and splitting the group would have meant rewriting a declaration rather than moving it. Design-doc file:line citations that this motion shifted are updated by content anchoring — each cited line was located by its exact text in the new tree (see #7493 for why this drift keeps recurring). The App-key and trustedMergerFunc citations now name their new files. Pre-existing drift in citations unrelated to this move is left alone rather than silently rewritten. main.go remains 8,072 lines and #7238 stays open; the remaining bulk is main() and runEvalCycle, tracked under #7232. Refs #7238 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Andrew Anderson * 🌱 test(github): let the F3 trusted-merger guard follow trustedMergerFunc out of main.go TestF3AuthorizerIsWiredInMain read cmd/hive/main.go and looked for trustedMergerFunc in it. This PR moves that declaration to cmd/hive/merge_eligibility.go, so the guard failed with "trustedMergerFunc not found -- it was renamed or removed", which reads as a lost fix when this is pure code motion. The func-body assertions now scan every non-test .go file in cmd/hive via a new f3ReadPackage helper, so the guard follows the declaration wherever the ongoing cmd/hive split puts it next. The package, not the filename, is the unit that matters for "does this declaration still exist". The wiring assertion stays pinned to cmd/hive/main.go on purpose: that one is about where startup installs the authorizer, and it should fail if the call moves out of startup. Teeth verified by mutation, each a genuine assertion failure: - lowering the floor to config.RoleRead -> "no longer requires at least config.RoleMerger" - deleting the SetMergerAuthorizer call from main.go -> "does not install the trusted-merger authorizer ... INERT" - renaming trustedMergerFunc -> "not found -- it was renamed or removed" Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Andrew Anderson * 📖 docs: correct five cmd/hive file:line citations this split left stale Two shifts landed after the citations in this PR were first computed, and each moved a cited line without moving the citation. The new files' header comments were relocated below the `package main` clause, which shifted every line in appkeyfile.go and merge_eligibility.go up by one. And the v4 merge grew main.go by ~570 lines via #7512, which moved runHub. Re-anchored by content, not arithmetic: each citation's target was read out of v4's main.go and located by its exact text in the merged tree. Verified by reading every cited line back and matching it against the symbol the prose names: appkeyfile.go:47-48 -> "Vars rather than consts so tests can point ..." appkeyfile.go:50 -> spokeProvisionedAppKeyPath = "/secrets/..." appkeyfile.go:51 -> spokeAppKeyPath = "/data/..." merge_eligibility.go:50 -> func trustedMergerFunc( main.go:8104 -> func runHub( The thirteen main.go citations this PR already rewrote were re-checked the same way and are correct as they stand; they are untouched here. src/scripts/check-api-reference-citations.sh still exits 0. Refs #7238 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Andrew Anderson --------- Signed-off-by: Andrew Anderson Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../changed-7238-cmd-hive-domain-split.md | 1 + src/cmd/hive/appkeyfile.go | 388 ++++ src/cmd/hive/intent_verdicts.go | 365 ++++ src/cmd/hive/login_scan.go | 365 ++++ src/cmd/hive/main.go | 1785 +---------------- src/cmd/hive/merge_eligibility.go | 639 ++++++ src/cmd/hive/selfupgrade.go | 211 ++ src/docs/design/agent-turn-model.md | 14 +- src/docs/design/copilot-cost-capture.md | 2 +- src/docs/design/github-mention-triggers.md | 4 +- src/docs/design/master-delivery-wrapped.md | 8 +- src/docs/design/master-key-rotation.md | 2 +- src/docs/knowledge-curator.md | 4 +- .../github/f3_trusted_merger_source_test.go | 35 +- 14 files changed, 2021 insertions(+), 1802 deletions(-) create mode 100644 changelog.d/changed-7238-cmd-hive-domain-split.md create mode 100644 src/cmd/hive/appkeyfile.go create mode 100644 src/cmd/hive/intent_verdicts.go create mode 100644 src/cmd/hive/login_scan.go create mode 100644 src/cmd/hive/merge_eligibility.go create mode 100644 src/cmd/hive/selfupgrade.go diff --git a/changelog.d/changed-7238-cmd-hive-domain-split.md b/changelog.d/changed-7238-cmd-hive-domain-split.md new file mode 100644 index 0000000000..b21783324a --- /dev/null +++ b/changelog.d/changed-7238-cmd-hive-domain-split.md @@ -0,0 +1 @@ +- `cmd/hive`'s `main.go` shed five self-contained domains into their own files — GitHub App key-file resolution (`appkeyfile.go`), self-upgrade marker/outcome bookkeeping (`selfupgrade.go`), login-required scanning and its sighting debounce (`login_scan.go`), auto-merge eligibility classification (`merge_eligibility.go`), and PR/issue intent verdicts (`intent_verdicts.go`) ([#7238](https://github.com/hivecommons/hive/issues/7238)). `main.go` drops from 9,795 to 8,072 lines. This is pure code motion — every one of the 242 top-level declarations was moved byte-identically, verified by a structural declaration diff, so no operator-visible behaviour changes; it makes the merge, upgrade and App-key paths reviewable on their own instead of only as a slice of a 9.8K-line file. diff --git a/src/cmd/hive/appkeyfile.go b/src/cmd/hive/appkeyfile.go new file mode 100644 index 0000000000..9379b64369 --- /dev/null +++ b/src/cmd/hive/appkeyfile.go @@ -0,0 +1,388 @@ +package main + +// GitHub App private-key file management: where per-App-ID and per-hive key +// files live on disk, how a key is written, resolved, fingerprinted and how a +// failure to find one is described. + +import ( + "fmt" + "os" + "path/filepath" + "strconv" + "strings" + + // automaxprocs sets GOMAXPROCS to match the container's CPU quota (Linux + // CFS) at init. Without it the Go runtime sizes its P count to the whole + // NODE's core count, so on a many-core IKS worker a pod limited to a few + // CPUs spawns far more runnable Ps than its CFS quota can service; when the + // quota is exhausted mid-period EVERY goroutine — including the netpoller + // that answers the :3002 liveness probe and the heartbeat loop — is + // throttled until the next CFS period, which stacks on top of the NFS + // stalls to push probe latency past the kubelet timeout. Matching GOMAXPROCS + // to the quota removes that self-inflicted throttling. + // + // This is called explicitly rather than via the package's blank import + // because that import's init writes a line to the default logger (stderr) + // unconditionally. `hive` re-execs itself as a Git transport shim, and the + // setup path captures a child's stdout and stderr into a single buffer to + // parse (e.g. `symbolic-ref --short origin/HEAD`), so an init-time banner + // is indistinguishable from Git's answer and corrupts the parsed branch + // name. Setting it with a no-op logger keeps the GOMAXPROCS behaviour and + // drops the banner. + + "github.com/hivecommons/hive/pkg/apphealth" + "github.com/hivecommons/hive/pkg/config" +) + +// GitHub App private-key locations on a spoke, and how the two differ. +// +// - spokeProvisionedAppKeyPath is a read-only Kubernetes Secret mount, written +// at PROVISIONING time from a key an operator supplied for THIS hive +// specifically. Its presence is the marker of a deliberate per-hive +// credential, which the hub's cluster-wide reconcile must never overwrite. +// - spokeAppKeyPath is on the PVC and is where a hub-delivered (cluster +// default) key lands. It is also what cfg.GitHub.KeyFile is repointed at +// once the hub delivers one, so it takes effect over the provisioned mount. +// +// Vars rather than consts so tests can point them at a temp dir and exercise +// the real resolution order; production never reassigns them. +var ( + spokeProvisionedAppKeyPath = "/secrets/gh-app-key.pem" + spokeAppKeyPath = "/data/gh-app-key.pem" + // spokeAppKeyDir is where per-app-id keys the hub delivers land, one file per + // App the fleet knows: gh-app-key-.pem. It is the PVC directory that + // already holds spokeAppKeyPath, so both survive restarts. A var so tests can + // redirect it; production never reassigns it. + spokeAppKeyDir = "/data" + // spokeProvisionedAppKeyDir is the read-only projected-Secret mount where + // PROVISIONING places per-app-id keys (gh-app-key-.pem), mirroring + // spokeAppKeyDir on the PVC. A hive provisioned with the fleet's full key set + // holds them here from its very first boot — before any heartbeat has run — so + // a forge switch never has to wait a beat for the target forge's key. The + // mount is readOnly, so nothing ever writes here; it is a lookup source only. + spokeProvisionedAppKeyDir = "/secrets" +) + +// spokeAppKeyFileMode is rw------- : signing material must never be readable by +// anything else sharing the PVC or the pod. +const spokeAppKeyFileMode = 0o600 + +func perAppIDKeyPath(appID int64) string { + if appID <= 0 { + return "" + } + return filepath.Join(spokeAppKeyDir, fmt.Sprintf("gh-app-key-%d.pem", appID)) +} + +// deliveredKeyPath is where a hub-delivered private key for appID is stored. +// +// The filename NAMES the App, so a key can only ever be found under the App it +// was delivered for. The generic /data/gh-app-key.pem carries no such evidence: +// a key written there for one App silently becomes "the key" for whatever +// app_id the config later claims, which is how all 33 heartbeat-only-cluster spokes ended up +// signing as the public App with the GHE key and getting +// 404 Integration not found. +// +// Falls back to the generic path only when the delivery names no App, so a key +// is never dropped on the floor. +func deliveredKeyPath(appID int64) string { + if p := perAppIDKeyPath(appID); p != "" { + return p + } + return spokeAppKeyPath +} + +// perAppIDProvisionedKeyPath is perAppIDKeyPath's read-only twin: the same +// per-app-id filename under the provisioning Secret mount. It is consulted only +// when the PVC has no usable key for the app_id, so a heartbeat-delivered key +// (which can be rotated) always wins over the one baked in at provision time. +func perAppIDProvisionedKeyPath(appID int64) string { + if appID <= 0 { + return "" + } + return filepath.Join(spokeProvisionedAppKeyDir, fmt.Sprintf("gh-app-key-%d.pem", appID)) +} + +// perAppIDKeyFilePrefix / Suffix bracket the per-app-id key filename so a scan +// can recover the app_id from the name. Named so the format lives in exactly one +// place alongside perAppIDKeyPath. +const ( + perAppIDKeyFilePrefix = "gh-app-key-" + perAppIDKeyFileSuffix = ".pem" +) + +// heldPerAppIDKeyFingerprints scans the PVC for per-app-id key files +// (gh-app-key-.pem) and returns app_id (decimal string) → fingerprint for +// every one that holds a usable key. It is what the spoke reports so the hub +// delivers the fleet's additional keys idempotently: a key already present with +// the right fingerprint is not re-sent. +// +// It never returns key material — only fingerprints. A missing directory, +// unreadable file, or unparseable key is silently skipped: the worst case is the +// hub re-delivers a key the spoke already writes idempotently, never a crash. +func heldPerAppIDKeyFingerprints() map[string]string { + entries, err := os.ReadDir(spokeAppKeyDir) + if err != nil { + return nil + } + var held map[string]string + for _, e := range entries { + if e.IsDir() { + continue + } + name := e.Name() + if !strings.HasPrefix(name, perAppIDKeyFilePrefix) || !strings.HasSuffix(name, perAppIDKeyFileSuffix) { + continue + } + idStr := strings.TrimSuffix(strings.TrimPrefix(name, perAppIDKeyFilePrefix), perAppIDKeyFileSuffix) + id, convErr := strconv.ParseInt(idStr, 10, 64) + if convErr != nil || id <= 0 { + continue + } + fp, fpErr := config.AppKeyFingerprintFromFile(filepath.Join(spokeAppKeyDir, name)) + if fpErr != nil || fp == "" { + continue + } + if held == nil { + held = make(map[string]string) + } + held[idStr] = fp + } + return held +} + +// writePerAppIDKey persists a hub-delivered per-app-id key to its PVC file +// atomically (temp file in the same dir, then rename) with a restrictive 0600 +// mode from creation, so a spoke can never sign with a half-written key. Returns +// the resulting fingerprint (never the key) for auditable logging, or an error. +func writePerAppIDKey(appID int64, pemData string) (string, error) { + path := perAppIDKeyPath(appID) + if path == "" { + return "", fmt.Errorf("refusing to write key for non-positive app_id %d", appID) + } + trimmed := strings.TrimSpace(pemData) + if !strings.HasPrefix(trimmed, "-----BEGIN") { + return "", fmt.Errorf("app key for app_id %d is not PEM", appID) + } + fp, err := config.AppKeyFingerprint(trimmed) + if err != nil { + return "", fmt.Errorf("app key for app_id %d is unusable: %w", appID, err) + } + if err := os.MkdirAll(spokeAppKeyDir, 0o700); err != nil { + return "", fmt.Errorf("create app key dir: %w", err) + } + tmp, err := os.CreateTemp(spokeAppKeyDir, "."+filepath.Base(path)+".tmp*") + if err != nil { + return "", fmt.Errorf("create temp app key file: %w", err) + } + tmpName := tmp.Name() + defer func() { _ = os.Remove(tmpName) }() // no-op once the rename below succeeds + if err := tmp.Chmod(spokeAppKeyFileMode); err != nil { + _ = tmp.Close() // best-effort cleanup; the chmod error is what's returned + return "", fmt.Errorf("chmod temp app key file: %w", err) + } + if _, err := tmp.WriteString(trimmed + "\n"); err != nil { + _ = tmp.Close() // best-effort cleanup; the write error is what's returned + return "", fmt.Errorf("write temp app key file: %w", err) + } + if err := tmp.Sync(); err != nil { + _ = tmp.Close() // best-effort cleanup; the sync error is what's returned + return "", fmt.Errorf("sync temp app key file: %w", err) + } + if err := tmp.Close(); err != nil { + return "", fmt.Errorf("close temp app key file: %w", err) + } + if err := os.Rename(tmpName, path); err != nil { + return "", fmt.Errorf("rename app key into place: %w", err) + } + return fp, nil +} + +// reportedAppKeyFingerprint returns the non-secret fingerprint of the App key +// this spoke is ACTUALLY using, for the heartbeat payload. It fingerprints the +// resolved key file rather than a hard-coded path so the hub compares against +// the key that would really sign a JWT. +// +// Returns "" whenever there is no usable key — no file, empty file, or +// unparseable contents. All three mean the same thing to the hub ("this spoke +// cannot authenticate") and are repaired identically. The private key itself is +// never returned, and never enters the payload. +func reportedAppKeyFingerprint(keyFile string, appID int64) string { + // Lead with the same path resolveAppKeyFile would sign with, so the hub is + // told about the key actually in effect and never about a shadowed one. + candidates := []string{ + resolveAppKeyFile(keyFile, os.Getenv("GH_APP_KEY_FILE"), appID), + perAppIDKeyPath(appID), + keyFile, spokeAppKeyPath, spokeProvisionedAppKeyPath, + } + for _, p := range candidates { + if strings.TrimSpace(p) == "" { + continue + } + if fp, err := config.AppKeyFingerprintFromFile(p); err == nil && fp != "" { + return fp + } + } + return "" +} + +// hasPerHiveAppKey reports whether this spoke's key came from a per-hive +// provisioning secret rather than the cluster default. The provisioned mount is +// read-only, so its mere existence with real PEM content is the signal — a +// hub-delivered key can never create or alter it. +// +// When BOTH exist the hub-delivered PVC key is the one in effect (the callback +// repoints cfg.GitHub.KeyFile at it), so this only claims an override while the +// provisioned key is genuinely the one being used. +func hasPerHiveAppKey(keyFile string, appID int64) bool { + fp, err := config.AppKeyFingerprintFromFile(spokeProvisionedAppKeyPath) + if err != nil || fp == "" { + return false + } + // The provisioned key exists. It is the effective credential only if the + // resolved key file still points at it — resolveAppKeyFile is the single + // authority on that, so an unconfigured hive that has already taken delivery + // of a /data key (or a per-app-id key) correctly stops claiming a per-hive + // override. + return resolveAppKeyFile(keyFile, os.Getenv("GH_APP_KEY_FILE"), appID) == spokeProvisionedAppKeyPath +} + +// resolveAppKeyFile picks which App private key this process will actually sign +// with, given the configured key_file and the GH_APP_KEY_FILE env override. +// +// WHY THE /data PREFERENCE MATTERS +// +// A hub-delivered key lands at spokeAppKeyPath (/data, on the PVC) and the +// heartbeat callback repoints cfg.GitHub.KeyFile at it — but only in memory, for +// the life of that process. A hive whose config carries NO key_file (which is +// the state of the three live GHE hives this repairs) used to fall straight +// through to the read-only /secrets provisioning mount. That mount holds the +// stale, wrong key, and the spoke cannot write to it. So on every restart the +// hive would silently go back to signing with the key that cannot work, and the +// hub — seeing the wrong fingerprint reported again — would redeliver forever. +// The key would be delivered and never used: a fault that reads as fixed. +// +// So when nothing is explicitly configured, a key already present on the PVC is +// preferred over the provisioning mount. An EXPLICIT key_file or env override +// still wins outright: those are deliberate, and this must not silently redirect +// an operator who named a path. +// +// PER-APP-ID SELECTION (the both-keys fix) +// +// appID is the App this process is configured to authenticate AS +// (cfg.GitHub.AppID). When the hub has delivered a per-app-id key for exactly +// that App — /data/gh-app-key-.pem — it is preferred over the generic +// single-file paths, because it is provably the RIGHT key for the app_id we +// claim. This is what lets a github.com hive on a GitHub-Enterprise cluster sign +// with the github.com key even though its cluster default (and its single +// /data/gh-app-key.pem) is the GHE key. It sits just below an explicit +// key_file/env override — an operator who named a path still wins — and above +// the generic fallbacks. appID <= 0 disables it entirely, so nothing changes for +// a hive that reports no app_id. +func resolveAppKeyFile(configured, envOverride string, appID int64) string { + if v := strings.TrimSpace(envOverride); v != "" { + return v + } + if v := strings.TrimSpace(configured); v != "" { + // MIGRATION. A configured key_file that is the GENERIC path is not an + // operator's choice — it is a value older builds wrote automatically on + // every key delivery, and it does not name the App it holds. When we can + // see a per-app-id key for the app_id we actually claim, that key is + // correct by construction and the generic pin is stale, so ignore it. + // + // Without this, the ~33 spokes already carrying + // key_file: /data/gh-app-key.pem keep signing with whichever App's key + // happens to sit there — the live 404 Integration not found — because an + // explicit value short-circuits the per-app-id lookup below. + // + // Deliberately narrow: only the exact generic path is overridden, and + // only when a usable per-app-id key exists. Any other path is a genuine + // operator override (a hive on a third App with a bespoke key location) + // and still wins outright. + // + // BOTH generic paths qualify. /data/gh-app-key.pem is what older builds + // wrote on every key delivery; /secrets/gh-app-key.pem is what the + // PROVISIONING TEMPLATE hardcodes for every App-using hive + // (saas_provision.go). Neither names the App it holds, and neither was + // typed by an operator. Until /secrets was included here, a provisioned + // hive could never change forges: the hub could correct app_id all it + // liked and the spoke kept signing with the provisioned key, which on + // the spoke-cluster pool was a placeholder matching NEITHER real App. + if v == spokeAppKeyPath || v == spokeProvisionedAppKeyPath { + if p := perAppIDKeyPath(appID); p != "" { + if fp, err := config.AppKeyFingerprintFromFile(p); err == nil && fp != "" { + return p + } + } + } + return v + } + // Nothing explicitly configured. Prefer a per-app-id key matching the App we + // claim — the only key that is CORRECT-by-construction for this app_id — over + // the generic cluster/provisioned files. The fingerprint check (not mere + // existence) keeps an empty or truncated per-app file from shadowing a good + // generic key. + if p := perAppIDKeyPath(appID); p != "" { + if fp, err := config.AppKeyFingerprintFromFile(p); err == nil && fp != "" { + return p + } + } + // Same idea, but from the read-only provisioning mount: a hive provisioned + // with the fleet's full key set can sign as its configured App on its very + // first boot, before any heartbeat has delivered anything to the PVC. Ranked + // BELOW the PVC copy so a rotated key delivered by heartbeat always wins over + // the one frozen into the Secret at provision time. + if p := perAppIDProvisionedKeyPath(appID); p != "" { + if fp, err := config.AppKeyFingerprintFromFile(p); err == nil && fp != "" { + return p + } + } + // Prefer a usable hub-delivered key on the PVC; fall back to the provisioning + // mount only when /data has no parseable key. + if fp, err := config.AppKeyFingerprintFromFile(spokeAppKeyPath); err == nil && fp != "" { + return spokeAppKeyPath + } + return spokeProvisionedAppKeyPath +} + +// describeAppKeyFailure turns a bare wrapped os error from github.NewAppAuth +// into a message an operator can act on without reading the source: it names +// the path actually tried, the full resolution order that produced it, and the +// underlying cause. +// +// The generic "reading app key /secrets/gh-app-key.pem: no such file" that this +// replaces gave no hint that key_file, $GH_APP_KEY_FILE, the PVC path and the +// provisioning mount are all consulted in a fixed order — so the usual response +// was to put the key in the wrong one of the four. +func describeAppKeyFailure(configured, envOverride, resolved string, err error) string { + order := []string{ + fmt.Sprintf("$GH_APP_KEY_FILE=%s", describeKeySource(envOverride)), + fmt.Sprintf("github.key_file=%s", describeKeySource(configured)), + fmt.Sprintf("per-app-id PVC key %s/gh-app-key-.pem", spokeAppKeyDir), + fmt.Sprintf("per-app-id provisioning key %s/gh-app-key-.pem", spokeProvisionedAppKeyDir), + fmt.Sprintf("PVC fallback %s", spokeAppKeyPath), + fmt.Sprintf("provisioning mount %s", spokeProvisionedAppKeyPath), + } + return fmt.Sprintf( + "GitHub App private key could not be loaded from %q: %v. "+ + "Resolution order (first non-empty wins): %s. "+ + "Write a PEM-encoded RSA private key to that path, or point github.key_file at one.", + resolved, err, strings.Join(order, " → "), + ) +} + +// describeKeySource renders an unset key-file source as "(unset)" so the +// resolution order in describeAppKeyFailure reads unambiguously. +func describeKeySource(v string) string { + if strings.TrimSpace(v) == "" { + return "(unset)" + } + return v +} + +// appKeyPaths snapshots the two App key path vars for a pkg/apphealth call. +// Read at call time on purpose: tests repoint these vars, and capturing them +// once would silently ignore that. +func appKeyPaths() apphealth.KeyPaths { + return apphealth.KeyPaths{Spoke: spokeAppKeyPath, Provisioned: spokeProvisionedAppKeyPath} +} diff --git a/src/cmd/hive/intent_verdicts.go b/src/cmd/hive/intent_verdicts.go new file mode 100644 index 0000000000..521d728c72 --- /dev/null +++ b/src/cmd/hive/intent_verdicts.go @@ -0,0 +1,365 @@ +package main + +// Intent verdicts: judging whether an agent's pull request actually matches the +// issue it claims to resolve -- gathering PR and issue evidence, summarising the +// alignment, recording the advisory, and writing the intent-verdicts report. + +import ( + "context" + "encoding/json" + "fmt" + "log/slog" + "os" + "strings" + "time" + + // automaxprocs sets GOMAXPROCS to match the container's CPU quota (Linux + // CFS) at init. Without it the Go runtime sizes its P count to the whole + // NODE's core count, so on a many-core IKS worker a pod limited to a few + // CPUs spawns far more runnable Ps than its CFS quota can service; when the + // quota is exhausted mid-period EVERY goroutine — including the netpoller + // that answers the :3002 liveness probe and the heartbeat loop — is + // throttled until the next CFS period, which stacks on top of the NFS + // stalls to push probe latency past the kubelet timeout. Matching GOMAXPROCS + // to the quota removes that self-inflicted throttling. + // + // This is called explicitly rather than via the package's blank import + // because that import's init writes a line to the default logger (stderr) + // unconditionally. `hive` re-execs itself as a Git transport shim, and the + // setup path captures a child's stdout and stderr into a single buffer to + // parse (e.g. `symbolic-ref --short origin/HEAD`), so an init-time banner + // is indistinguishable from Git's answer and corrupts the parsed branch + // name. Setting it with a no-op logger keeps the GOMAXPROCS behaviour and + // drops the banner. + + gh "github.com/google/go-github/v72/github" + + "github.com/hivecommons/hive/pkg/beads" + "github.com/hivecommons/hive/pkg/config" + "github.com/hivecommons/hive/pkg/github" + "github.com/hivecommons/hive/pkg/intent" +) + +func writeIntentVerdicts( + ctx context.Context, + cfg *config.Config, + ghClient *github.Client, + actionable *github.ActionableResult, + beadStores map[string]*beads.Store, + logger *slog.Logger, +) map[string]intent.Verdict { + verdicts := make(map[string]intent.Verdict) + if cfg == nil || actionable == nil { + return verdicts + } + _ = os.MkdirAll("/var/run/hive-metrics", 0o755) + aiAuthor := strings.TrimSpace(cfg.EffectiveAIAuthor()) + intentCfg := intent.Config{ + TestPathPatterns: cfg.Intent.TestPathPatterns, + DocsPathPatterns: cfg.Intent.DocsPathPatterns, + GuardrailPathPatterns: cfg.Intent.GuardrailPathPatterns, + FeatureSignals: cfg.Intent.FeatureSignals, + } + var alignmentReviewer *intent.AlignmentReviewer + if strings.TrimSpace(cfg.Intent.AlignmentModel) != "" { + endpoint, apiKey, _ := cfg.Governor.ResolveReviewer() + var err error + alignmentReviewer, err = intent.NewAlignmentReviewer(intent.AlignmentReviewerConfig{ + Endpoint: endpoint, + APIKey: apiKey, + Model: cfg.Intent.AlignmentModel, + }) + if err != nil { + logger.Warn("intent alignment reviewer disabled", "error", err) + } + } + type verdictRecord struct { + Repo string `json:"repo"` + Number int `json:"number"` + Title string `json:"title"` + Author string `json:"author"` + Enforced bool `json:"enforced"` + Verdict intent.Verdict `json:"verdict"` + Classify string `json:"classification_reason"` + FetchError string `json:"fetch_error,omitempty"` + } + records := make([]verdictRecord, 0, len(actionable.PRs.Items)) + for _, pr := range actionable.PRs.Items { + fullRepo := fullRepoName(pr.Repo, cfg.Project.Org) + key := fmt.Sprintf("%s/%d", fullRepo, pr.Number) + agentPR := aiAuthor != "" && strings.EqualFold(pr.Author, aiAuthor) + record := verdictRecord{ + Repo: fullRepo, + Number: pr.Number, + Title: pr.Title, + Author: pr.Author, + Enforced: cfg.Intent.Enforce, + } + if !agentPR { + class := intent.Classify(intent.PR{Title: pr.Title, Labels: pr.Labels, Author: pr.Author, AgentAuthor: false}, intentCfg) + verdict := intent.Evaluate(class, intent.Evidence{}) + verdicts[key] = verdict + record.Verdict = verdict + record.Classify = class.Reason + records = append(records, record) + continue + } + body, files, approved, err := fetchIntentPREvidence(ctx, ghClient, fullRepo, pr.Number) + if err != nil { + verdict := intent.Verdict{ + Tier: intent.Tier1, + Authorized: false, + Reason: "intent evidence unavailable: " + err.Error(), + AgentPR: true, + } + verdicts[key] = verdict + record.Verdict = verdict + record.FetchError = err.Error() + records = append(records, record) + logger.Warn("intent verification evidence fetch failed", "repo", fullRepo, "number", pr.Number, "error", err) + continue + } + class := intent.Classify(intent.PR{ + Title: pr.Title, + Body: body, + Labels: pr.Labels, + Files: files, + Author: pr.Author, + AgentAuthor: true, + }, intentCfg) + evidence := intent.BuildEvidenceForRepo(body, fullRepo, beadStores, approved) + verdict := intent.Evaluate(class, evidence) + issueTexts, issueErr := fetchIntentIssueTexts(ctx, ghClient, fullRepo, body) + if issueErr != nil { + logger.Warn("intent alignment issue evidence fetch failed", "repo", fullRepo, "number", pr.Number, "error", issueErr) + } + refs := intent.LinkedIssueRefs(body, fullRepo) + alignCtx := intent.BuildAlignmentContext(intent.PR{ + Title: pr.Title, + Body: body, + Labels: pr.Labels, + Files: files, + Author: pr.Author, + AgentAuthor: true, + }, issueTexts, beadStores, refs) + alignment := intent.EvaluateAlignment(alignCtx, class.Tier, intentCfg) + if alignmentReviewer != nil { + modelVerdict, err := alignmentReviewer.Review(ctx, alignCtx) + if err != nil { + logger.Warn("intent alignment model review failed open", "repo", fullRepo, "number", pr.Number, "error", err) + alignment = intent.MergeAlignment(alignment, nil, err) + } else { + alignment = intent.MergeAlignment(alignment, &modelVerdict, nil) + } + } + verdict.Alignment = &alignment + verdicts[key] = verdict + record.Verdict = verdict + record.Classify = class.Reason + records = append(records, record) + if !verdict.Authorized { + logger.Info("intent authorization denied", "repo", fullRepo, "number", pr.Number, "tier", verdict.Tier, "reason", verdict.Reason, "enforce", cfg.Intent.Enforce) + } + if alignment.Misaligned() { + logger.Info("intent alignment denied", "repo", fullRepo, "number", pr.Number, "reason", alignment.Rationale, "enforce", cfg.Intent.Enforce) + recordIntentAlignmentAdvisory(beadStores, fullRepo, pr.Number, alignment, logger) + } + } + payload := map[string]any{ + "generated_at": time.Now().UTC().Format(time.RFC3339), + "enforced": cfg.Intent.Enforce, + "verdicts": records, + } + if data, err := json.Marshal(payload); err == nil { + atomicWrite(intentVerdictsPath, data) + } else { + logger.Warn("failed to marshal intent verdicts", "error", err) + } + return verdicts +} + +func fetchIntentPREvidence(ctx context.Context, ghClient *github.Client, repo string, number int) (string, []intent.ChangedFile, bool, error) { + if ghClient == nil || ghClient.GoGitHub() == nil { + return "", nil, false, github.ErrNoGitHubClient + } + owner, repoName, ok := strings.Cut(repo, "/") + if !ok || owner == "" || repoName == "" { + return "", nil, false, fmt.Errorf("invalid repo %q", repo) + } + client := ghClient.GoGitHub() + pr, _, err := client.PullRequests.Get(ctx, owner, repoName, number) + if err != nil { + return "", nil, false, fmt.Errorf("getting PR: %w", err) + } + var files []intent.ChangedFile + fileOpts := &gh.ListOptions{PerPage: 100} + for { + page, resp, err := client.PullRequests.ListFiles(ctx, owner, repoName, number, fileOpts) + if err != nil { + return "", nil, false, fmt.Errorf("listing PR files: %w", err) + } + for _, f := range page { + files = append(files, intent.ChangedFile{ + Filename: f.GetFilename(), + Status: f.GetStatus(), + Additions: f.GetAdditions(), + Deletions: f.GetDeletions(), + }) + } + if resp == nil || resp.NextPage == 0 { + break + } + fileOpts.Page = resp.NextPage + } + if reported := pr.GetChangedFiles(); reported > len(files) { + return "", nil, false, fmt.Errorf("incomplete PR file list: GitHub reported %d changed files but API returned %d; intent alignment requires the complete changed-file list", reported, len(files)) + } + approved, err := hasMaintainerApproval(ctx, client, owner, repoName, number) + if err != nil { + return "", nil, false, err + } + return pr.GetBody(), files, approved, nil +} + +func fetchIntentIssueTexts(ctx context.Context, ghClient *github.Client, defaultRepo, body string) ([]intent.TextEvidence, error) { + if ghClient == nil || ghClient.GoGitHub() == nil { + return nil, github.ErrNoGitHubClient + } + client := ghClient.GoGitHub() + refs := intent.LinkedIssueRefs(body, defaultRepo) + out := make([]intent.TextEvidence, 0, len(refs)) + for _, ref := range refs { + repo := ref.Repo + if repo == "" { + repo = defaultRepo + } + owner, repoName, ok := strings.Cut(repo, "/") + if !ok || owner == "" || repoName == "" { + continue + } + issue, _, err := client.Issues.Get(ctx, owner, repoName, ref.Number) + if err != nil { + return out, fmt.Errorf("getting linked issue %s#%d: %w", repo, ref.Number, err) + } + out = append(out, intent.TextEvidence{ + Source: fmt.Sprintf("issue %s#%d", repo, ref.Number), + Title: issue.GetTitle(), + Body: issue.GetBody(), + }) + } + return out, nil +} + +func recordIntentAlignmentAdvisory(stores map[string]*beads.Store, repo string, number int, alignment intent.AlignmentVerdict, logger *slog.Logger) { + store := stores["intent"] + if store == nil { + store = stores["quality"] + } + if store == nil { + for _, candidate := range stores { + if candidate != nil { + store = candidate + break + } + } + } + if store == nil { + return + } + title := fmt.Sprintf("Intent alignment drift in %s#%d", repo, number) + // "/#", NOT "gh-/#". The old form fused the + // source prefix into the org when the digest built its URL, so every one of + // these rendered a link to a github.com/gh- that does not exist + // (#6080). The renderer strips the prefix defensively for beads already + // written this way; this stops writing new ones. + ref := fmt.Sprintf("%s#%d", repo, number) + // Beads created before that change carry the prefixed form. Matching both + // keeps this idempotent across the change: without it the first run after + // upgrading would fail to recognise the existing bead and open a duplicate. + legacyRef := "gh-" + ref + for _, b := range store.List(beads.ListFilter{}) { + if b.Type == beads.TypeAdvisory && b.Title == title && + (b.ExternalRef == ref || b.ExternalRef == legacyRef) && + b.Status != beads.StatusClosed && b.Status != beads.StatusDone { + return + } + } + b, err := store.Create(title, beads.TypeAdvisory, beads.PriorityHigh, "intent", ref) + if err != nil { + logger.Warn("failed to record intent alignment advisory", "repo", repo, "number", number, "error", err) + return + } + _ = store.Update(b.ID, func(bead *beads.Bead) { + bead.Notes = alignmentSummary(alignment) + }) +} + +func alignmentSummary(alignment intent.AlignmentVerdict) string { + var parts []string + if alignment.Rationale != "" { + parts = append(parts, alignment.Rationale) + } + for _, f := range alignment.DeterministicFindings { + if f.Status == intent.AlignmentStatusMisaligned { + parts = append(parts, f.Code+": "+f.Reason+" ("+strings.Join(f.Files, ", ")+")") + } + } + if alignment.Model != nil && alignment.Model.Status == intent.AlignmentStatusMisaligned { + parts = append(parts, "model: "+alignment.Model.Rationale) + } + if len(parts) == 0 { + return "intent alignment check reported misalignment" + } + return strings.Join(parts, "\n") +} + +func hasMaintainerApproval(ctx context.Context, client *gh.Client, owner, repo string, number int) (bool, error) { + opts := &gh.ListOptions{PerPage: 100} + latest := make(map[string]string) + maintainer := make(map[string]bool) + for { + reviews, resp, err := client.PullRequests.ListReviews(ctx, owner, repo, number, opts) + if err != nil { + return false, fmt.Errorf("listing PR reviews: %w", err) + } + for _, review := range reviews { + login := review.GetUser().GetLogin() + if login == "" { + continue + } + if maintainerAssociation(review.GetAuthorAssociation()) { + switch review.GetState() { + case "APPROVED", "CHANGES_REQUESTED", "DISMISSED": + latest[login] = review.GetState() + } + maintainer[login] = true + } + } + if resp == nil || resp.NextPage == 0 { + break + } + opts.Page = resp.NextPage + } + approved := false + for login, state := range latest { + if !maintainer[login] { + continue + } + switch state { + case "CHANGES_REQUESTED": + return false, nil + case "APPROVED": + approved = true + } + } + return approved, nil +} + +func maintainerAssociation(association string) bool { + switch strings.ToUpper(strings.TrimSpace(association)) { + case "OWNER", "MEMBER", "COLLABORATOR": + return true + default: + return false + } +} diff --git a/src/cmd/hive/login_scan.go b/src/cmd/hive/login_scan.go new file mode 100644 index 0000000000..f10397b6cd --- /dev/null +++ b/src/cmd/hive/login_scan.go @@ -0,0 +1,365 @@ +package main + +// Login-required scanning: detecting that a coding backend has dropped to an +// interactive login prompt, debouncing that signal across sightings so a +// single noisy line cannot pause an agent, and acting on the verdict. + +import ( + "context" + "fmt" + "log/slog" + "regexp" + "strings" + "sync" + + // automaxprocs sets GOMAXPROCS to match the container's CPU quota (Linux + // CFS) at init. Without it the Go runtime sizes its P count to the whole + // NODE's core count, so on a many-core IKS worker a pod limited to a few + // CPUs spawns far more runnable Ps than its CFS quota can service; when the + // quota is exhausted mid-period EVERY goroutine — including the netpoller + // that answers the :3002 liveness probe and the heartbeat loop — is + // throttled until the next CFS period, which stacks on top of the NFS + // stalls to push probe latency past the kubelet timeout. Matching GOMAXPROCS + // to the quota removes that self-inflicted throttling. + // + // This is called explicitly rather than via the package's blank import + // because that import's init writes a line to the default logger (stderr) + // unconditionally. `hive` re-execs itself as a Git transport shim, and the + // setup path captures a child's stdout and stderr into a single buffer to + // parse (e.g. `symbolic-ref --short origin/HEAD`), so an init-time banner + // is indistinguishable from Git's answer and corrupts the parsed branch + // name. Setting it with a no-op logger keeps the GOMAXPROCS behaviour and + // drops the banner. + + "github.com/hivecommons/hive/pkg/agent" + "github.com/hivecommons/hive/pkg/config" + "github.com/hivecommons/hive/pkg/notify" +) + +// loginCommandForBackend returns the login instruction for a given CLI backend. +func loginCommandForBackend(backend string) string { + switch backend { + case "claude": + return "Run: claude login" + case "copilot": + return "Run: copilot auth login" + case "gemini": + return "Run: gemini auth login" + case "goose": + return "Run: goose auth login" + default: + return "Run the login command for " + backend + } +} + +// loginScanAction is what the detector should do about one agent this cycle. +type loginScanAction int + +const ( + // loginScanIgnore: nothing that looks like a login problem, or a startup + // modal is on screen. Any sighting streak is cleared. + loginScanIgnore loginScanAction = iota + // loginScanDeferAuthenticated: the pane matched, but the backend credential + // is demonstrably valid, so this is residue or a stuck CLI — the manager's + // token-restart heal's case, not an operator's (kubestellar/hive#5291). + loginScanDeferAuthenticated + // loginScanDeferStreak: the pane matched and the credential is not provably + // good, but this is the first consecutive cycle to see it. + loginScanDeferStreak + // loginScanPause: pause the agent and page the operator. + loginScanPause +) + +// loginPauseMinSightings is how many CONSECUTIVE governor cycles must see a +// login pattern before the detector pauses (kubestellar/hive#5291). +// +// The manager's own pane poller learned this at its ~3s cadence, where a single +// sighting restarted healthy agents; it now requires loginStreakRestartMin = 3. +// The detector had no equivalent, and a pause is far more expensive than a +// restart — it is sticky, it needs a human to undo, and it cancels the agent +// context that hosts the heal. Two is deliberate rather than three: a governor +// cycle is minutes, not seconds, so each extra cycle is real delay for a +// genuine logout, and the credential gate above already covers the case this +// backstops. It matters most for backends with no credential file this process +// can check, where it is the only new protection. +const loginPauseMinSightings = 2 + +// loginSightingTracker counts CONSECUTIVE cycles in which each agent's pane +// matched a login pattern. A clean cycle resets the count to zero, so a match +// has to persist to accumulate — a single flicker never reaches the threshold. +type loginSightingTracker struct { + mu sync.Mutex + streak map[string]int +} + +func newLoginSightingTracker() *loginSightingTracker { + return &loginSightingTracker{streak: map[string]int{}} +} + +// loginSightings is the detector's process-scoped state. The governor cycle is +// a function rather than an object, so the consecutive-sighting counts have to +// outlive a single call; tests build their own tracker and pass it explicitly. +var loginSightings = newLoginSightingTracker() + +// observe records this cycle's reading for one agent and returns the resulting +// consecutive-sighting count (1 on the first sighting). +func (t *loginSightingTracker) observe(agent string, matched bool) int { + if t != nil { + t.mu.Lock() + defer t.mu.Unlock() + } + if t == nil { + // No tracker wired: behave as if every sighting is its own streak, which + // is exactly the pre-#5291 single-observation behaviour. + if matched { + return loginPauseMinSightings + } + return 0 + } + if !matched { + delete(t.streak, agent) + return 0 + } + t.streak[agent]++ + return t.streak[agent] +} + +// forget drops an agent's streak — on pause (it stops being scanned) and for +// agents that are no longer present, so the map cannot grow without bound +// across a long-lived process. +func (t *loginSightingTracker) forget(agent string) { + if t == nil { + return + } + t.mu.Lock() + defer t.mu.Unlock() + delete(t.streak, agent) +} + +// retain drops every agent not in the given set. +func (t *loginSightingTracker) retain(present map[string]bool) { + if t == nil { + return + } + t.mu.Lock() + defer t.mu.Unlock() + for name := range t.streak { + if !present[name] { + delete(t.streak, name) + } + } +} + +// loginScanDecision is the detector's whole judgement about one agent, as a +// pure function of what was observed. It exists apart from scanForLoginRequired +// so the decision can be tested against real pane text without a tmux session, +// a manager, or a governor cycle. +// +// sightings is the consecutive-cycle count INCLUDING this one. +// +// The credential gate is the fix for kubestellar/hive#5291: the detector used +// to pause on pane text alone, and the pane during and just after an +// interactive /login necessarily contains login-screen chrome — so it fired on +// the evidence the operator's own fix had just produced, seven minutes after +// the credential was already valid. Worse, Pause() cancels the agent context +// and tears down the poller that hosts the token-restart heal (#4606), which is +// the mechanism built for exactly "login prompt on screen, credential valid". +// Pausing first therefore disabled the machinery that would have fixed the pane +// it misread. +// +// Text matching cannot be narrowed out of this: two earlier fixes tried +// (tail-only matching, then a tighter copilot pattern) and this incident is the +// third false positive. The pane legitimately contains login text at the moment +// the credential is freshest, so the credential has to be consulted. +func loginScanDecision( + backend, paneText string, + compiled []*regexp.Regexp, + credentialValid bool, + sightings int, +) (loginScanAction, *regexp.Regexp) { + matched := loginScanMatch(backend, paneText, compiled) + return loginScanVerdict(matched != nil, credentialValid, sightings), matched +} + +// loginScanMatch reports which login pattern this pane trips, or nil for none. +// Separate from the verdict so the scan loop can match ONCE and use the answer +// both to advance the sighting streak and to decide. +func loginScanMatch(backend, paneText string, compiled []*regexp.Regexp) *regexp.Regexp { + // Stand down while a startup-blocking modal (folder trust, codex update, …) + // is on screen: that is not a login problem, and pausing the agent for it + // cancels the trust-prompt watcher that would answer it — the deadlock that + // kept copilot agents "sitting at login prompt" through every operator + // re-login (hivecommons/hive, 2026-08-22). The watcher answers the modal + // within seconds; if a REAL login prompt follows, the next detector tick + // sees it on a clean pane. + if agent.PaneShowsBlockingPrompt(backend, paneText) { + return nil + } + for _, re := range compiled { + if re.MatchString(paneText) { + return re + } + } + return nil +} + +// loginScanVerdict turns "what the pane showed" into "what to do". It returns +// loginScanIgnore whenever matched is false, which is what lets the scan loop +// rely on a non-Ignore verdict implying a non-nil pattern to log. +func loginScanVerdict(matched, credentialValid bool, sightings int) loginScanAction { + if !matched { + return loginScanIgnore + } + if credentialValid { + return loginScanDeferAuthenticated + } + if sightings < loginPauseMinSightings { + return loginScanDeferStreak + } + return loginScanPause +} + +type loginScanAgentManager interface { + AllStatuses() map[string]*agent.AgentProcess + GetOutput(name string, lines int) ([]string, error) + AgentHasValidCredential(agentName string) bool + RefreshAgentTokenFor(ctx context.Context, name string) error + Pause(name, trigger, reason string) error +} + +type loginScanNotifier interface { + Send(title, message string, priority notify.Priority) +} + +type loginScanAuditor interface { + AuditLog(user, action, detail, agent string) +} + +// scanForLoginRequired checks each running agent's tmux pane output for login-required +// patterns. When a match is found, the agent is paused and a notification is sent. +func scanForLoginRequired( + ctx context.Context, + cfg *config.Config, + agentMgr loginScanAgentManager, + notifier loginScanNotifier, + dashSrv loginScanAuditor, + logger *slog.Logger, + sightings *loginSightingTracker, +) { + patterns := cfg.Governor.Sensing.LoginPatterns + if len(patterns) == 0 { + return + } + + // Compile regex patterns, skipping empty and invalid ones + compiled := make([]*regexp.Regexp, 0, len(patterns)) + for _, p := range patterns { + if strings.TrimSpace(p) == "" { + continue + } + re, err := regexp.Compile("(?i)" + p) + if err != nil { + logger.Warn("invalid login pattern regex", "pattern", p, "error", err) + continue + } + compiled = append(compiled, re) + } + if len(compiled) == 0 { + return + } + + // Scan the pane TAIL only. A login prompt the CLI is genuinely stuck at + // sits at the BOTTOM of the pane; the 50-line window this used to read + // reached deep into scrollback, where agent WORK OUTPUT that merely + // mentions a pattern phrase lives — quality's scan findings quoting + // "gh auth login" from auth documentation got the agent paused mid-kick + // (hivecommons/hive, 2026-08-22 08:27, on a fully-authenticated CLI). + // Same discipline as the poller's tail-only match (#4577). + const paneLines = 12 + statuses := agentMgr.AllStatuses() + scanned := make(map[string]bool, len(statuses)) + for name, proc := range statuses { + if proc.State != "running" { + continue + } + scanned[name] = true + + output, err := agentMgr.GetOutput(name, paneLines) + if err != nil || len(output) == 0 { + continue + } + + joined := strings.Join(output, "\n") + backend := cfg.Agents[name].Backend + + // #5291: ask the CREDENTIAL, not just the pane. A valid credential plus + // a login prompt is the token-restart heal's case; only an invalid one + // needs a human. + credentialValid := agentMgr.AgentHasValidCredential(name) + + // Match once. The streak has to reflect what the pane SHOWED, including + // on the cycles where a gate below declines to act on it, so the + // sighting is recorded before the verdict is taken. + re := loginScanMatch(backend, joined, compiled) + streak := sightings.observe(name, re != nil) + + switch loginScanVerdict(re != nil, credentialValid, streak) { + case loginScanIgnore: + continue + case loginScanDeferAuthenticated: + // Logged at Info, not Warn: this is the detector working correctly, + // and it is the line that explains an agent staying up with login + // text on its pane. + logger.Info("login pattern matched but the backend credential is valid — leaving it to the token-restart heal", + "agent", name, "backend", backend, "pattern", re.String()) + continue + case loginScanDeferStreak: + logger.Info("login pattern matched but not yet on enough consecutive cycles — deferring", + "agent", name, "backend", backend, "pattern", re.String(), + "sightings", streak, "required", loginPauseMinSightings) + continue + case loginScanPause: + logger.Warn("login required detected", + "agent", name, + "pattern", re.String(), + "sightings", streak, + ) + sightings.forget(name) + + // Attempt a per-agent token re-cache BEFORE pausing. On an + // App-authenticated hive the likeliest cause of a "gh auth + // login" prompt is an expired scoped-token cache (#4072); + // re-minting it now means the operator's Resume immediately + // works instead of 401ing straight back into this pause. + // Best-effort: hives without App auth (or agents without a + // dedicated UID) simply skip it. + if refreshErr := agentMgr.RefreshAgentTokenFor(ctx, name); refreshErr == nil { + logger.Info("re-cached per-agent scoped token before login-detector pause", "agent", name) + } + + // Pause the agent instead of restarting + if pauseErr := agentMgr.Pause(name, "login-detector", "login required detected"); pauseErr != nil { + logger.Warn("failed to pause agent after login detection", + "agent", name, "error", pauseErr) + } else { + dashSrv.AuditLog("system", "pause", "trigger=login-detector", name) + } + + // Determine the login instruction based on the agent's backend + loginCmd := loginCommandForBackend(backend) + + notifier.Send( + fmt.Sprintf("\U0001F511 Login required: %s", name), + fmt.Sprintf( + "Agent '%s' needs authentication. Open the agent's terminal "+ + "(tmux attach -t hive-%s) and run the login command for the CLI (%s). %s", + name, name, backend, loginCmd, + ), + notify.PriorityHigh, + ) + } + } + // Agents that vanished (removed from config, stopped) must not keep a + // streak alive in the map for the life of the process. + sightings.retain(scanned) +} diff --git a/src/cmd/hive/main.go b/src/cmd/hive/main.go index 87b0ba6530..c8413b1eed 100644 --- a/src/cmd/hive/main.go +++ b/src/cmd/hive/main.go @@ -65,7 +65,6 @@ import ( "github.com/hivecommons/hive/pkg/hooks" "github.com/hivecommons/hive/pkg/hub" "github.com/hivecommons/hive/pkg/inference" - "github.com/hivecommons/hive/pkg/intent" "github.com/hivecommons/hive/pkg/ioscan" "github.com/hivecommons/hive/pkg/knowledge" "github.com/hivecommons/hive/pkg/logscrub" @@ -190,39 +189,6 @@ var ( gitBranch = "unknown" ) -// GitHub App private-key locations on a spoke, and how the two differ. -// -// - spokeProvisionedAppKeyPath is a read-only Kubernetes Secret mount, written -// at PROVISIONING time from a key an operator supplied for THIS hive -// specifically. Its presence is the marker of a deliberate per-hive -// credential, which the hub's cluster-wide reconcile must never overwrite. -// - spokeAppKeyPath is on the PVC and is where a hub-delivered (cluster -// default) key lands. It is also what cfg.GitHub.KeyFile is repointed at -// once the hub delivers one, so it takes effect over the provisioned mount. -// -// Vars rather than consts so tests can point them at a temp dir and exercise -// the real resolution order; production never reassigns them. -var ( - spokeProvisionedAppKeyPath = "/secrets/gh-app-key.pem" - spokeAppKeyPath = "/data/gh-app-key.pem" - // spokeAppKeyDir is where per-app-id keys the hub delivers land, one file per - // App the fleet knows: gh-app-key-.pem. It is the PVC directory that - // already holds spokeAppKeyPath, so both survive restarts. A var so tests can - // redirect it; production never reassigns it. - spokeAppKeyDir = "/data" - // spokeProvisionedAppKeyDir is the read-only projected-Secret mount where - // PROVISIONING places per-app-id keys (gh-app-key-.pem), mirroring - // spokeAppKeyDir on the PVC. A hive provisioned with the fleet's full key set - // holds them here from its very first boot — before any heartbeat has run — so - // a forge switch never has to wait a beat for the target forge's key. The - // mount is readOnly, so nothing ever writes here; it is a lookup source only. - spokeProvisionedAppKeyDir = "/secrets" -) - -// spokeAppKeyFileMode is rw------- : signing material must never be readable by -// anything else sharing the PVC or the pod. -const spokeAppKeyFileMode = 0o600 - // traceShutdownTimeout bounds how long we wait for the OTel exporter to flush // pending spans during shutdown, so a slow/unreachable collector can't hang // process exit. @@ -623,319 +589,6 @@ func nextInstallationID(current int64, ghCfg *hub.HeartbeatGitHubAppConfig) (nex return current, false } -func perAppIDKeyPath(appID int64) string { - if appID <= 0 { - return "" - } - return filepath.Join(spokeAppKeyDir, fmt.Sprintf("gh-app-key-%d.pem", appID)) -} - -// deliveredKeyPath is where a hub-delivered private key for appID is stored. -// -// The filename NAMES the App, so a key can only ever be found under the App it -// was delivered for. The generic /data/gh-app-key.pem carries no such evidence: -// a key written there for one App silently becomes "the key" for whatever -// app_id the config later claims, which is how all 33 heartbeat-only-cluster spokes ended up -// signing as the public App with the GHE key and getting -// 404 Integration not found. -// -// Falls back to the generic path only when the delivery names no App, so a key -// is never dropped on the floor. -func deliveredKeyPath(appID int64) string { - if p := perAppIDKeyPath(appID); p != "" { - return p - } - return spokeAppKeyPath -} - -// perAppIDProvisionedKeyPath is perAppIDKeyPath's read-only twin: the same -// per-app-id filename under the provisioning Secret mount. It is consulted only -// when the PVC has no usable key for the app_id, so a heartbeat-delivered key -// (which can be rotated) always wins over the one baked in at provision time. -func perAppIDProvisionedKeyPath(appID int64) string { - if appID <= 0 { - return "" - } - return filepath.Join(spokeProvisionedAppKeyDir, fmt.Sprintf("gh-app-key-%d.pem", appID)) -} - -// perAppIDKeyFilePrefix / Suffix bracket the per-app-id key filename so a scan -// can recover the app_id from the name. Named so the format lives in exactly one -// place alongside perAppIDKeyPath. -const ( - perAppIDKeyFilePrefix = "gh-app-key-" - perAppIDKeyFileSuffix = ".pem" -) - -// heldPerAppIDKeyFingerprints scans the PVC for per-app-id key files -// (gh-app-key-.pem) and returns app_id (decimal string) → fingerprint for -// every one that holds a usable key. It is what the spoke reports so the hub -// delivers the fleet's additional keys idempotently: a key already present with -// the right fingerprint is not re-sent. -// -// It never returns key material — only fingerprints. A missing directory, -// unreadable file, or unparseable key is silently skipped: the worst case is the -// hub re-delivers a key the spoke already writes idempotently, never a crash. -func heldPerAppIDKeyFingerprints() map[string]string { - entries, err := os.ReadDir(spokeAppKeyDir) - if err != nil { - return nil - } - var held map[string]string - for _, e := range entries { - if e.IsDir() { - continue - } - name := e.Name() - if !strings.HasPrefix(name, perAppIDKeyFilePrefix) || !strings.HasSuffix(name, perAppIDKeyFileSuffix) { - continue - } - idStr := strings.TrimSuffix(strings.TrimPrefix(name, perAppIDKeyFilePrefix), perAppIDKeyFileSuffix) - id, convErr := strconv.ParseInt(idStr, 10, 64) - if convErr != nil || id <= 0 { - continue - } - fp, fpErr := config.AppKeyFingerprintFromFile(filepath.Join(spokeAppKeyDir, name)) - if fpErr != nil || fp == "" { - continue - } - if held == nil { - held = make(map[string]string) - } - held[idStr] = fp - } - return held -} - -// writePerAppIDKey persists a hub-delivered per-app-id key to its PVC file -// atomically (temp file in the same dir, then rename) with a restrictive 0600 -// mode from creation, so a spoke can never sign with a half-written key. Returns -// the resulting fingerprint (never the key) for auditable logging, or an error. -func writePerAppIDKey(appID int64, pemData string) (string, error) { - path := perAppIDKeyPath(appID) - if path == "" { - return "", fmt.Errorf("refusing to write key for non-positive app_id %d", appID) - } - trimmed := strings.TrimSpace(pemData) - if !strings.HasPrefix(trimmed, "-----BEGIN") { - return "", fmt.Errorf("app key for app_id %d is not PEM", appID) - } - fp, err := config.AppKeyFingerprint(trimmed) - if err != nil { - return "", fmt.Errorf("app key for app_id %d is unusable: %w", appID, err) - } - if err := os.MkdirAll(spokeAppKeyDir, 0o700); err != nil { - return "", fmt.Errorf("create app key dir: %w", err) - } - tmp, err := os.CreateTemp(spokeAppKeyDir, "."+filepath.Base(path)+".tmp*") - if err != nil { - return "", fmt.Errorf("create temp app key file: %w", err) - } - tmpName := tmp.Name() - defer func() { _ = os.Remove(tmpName) }() // no-op once the rename below succeeds - if err := tmp.Chmod(spokeAppKeyFileMode); err != nil { - _ = tmp.Close() // best-effort cleanup; the chmod error is what's returned - return "", fmt.Errorf("chmod temp app key file: %w", err) - } - if _, err := tmp.WriteString(trimmed + "\n"); err != nil { - _ = tmp.Close() // best-effort cleanup; the write error is what's returned - return "", fmt.Errorf("write temp app key file: %w", err) - } - if err := tmp.Sync(); err != nil { - _ = tmp.Close() // best-effort cleanup; the sync error is what's returned - return "", fmt.Errorf("sync temp app key file: %w", err) - } - if err := tmp.Close(); err != nil { - return "", fmt.Errorf("close temp app key file: %w", err) - } - if err := os.Rename(tmpName, path); err != nil { - return "", fmt.Errorf("rename app key into place: %w", err) - } - return fp, nil -} - -// reportedAppKeyFingerprint returns the non-secret fingerprint of the App key -// this spoke is ACTUALLY using, for the heartbeat payload. It fingerprints the -// resolved key file rather than a hard-coded path so the hub compares against -// the key that would really sign a JWT. -// -// Returns "" whenever there is no usable key — no file, empty file, or -// unparseable contents. All three mean the same thing to the hub ("this spoke -// cannot authenticate") and are repaired identically. The private key itself is -// never returned, and never enters the payload. -func reportedAppKeyFingerprint(keyFile string, appID int64) string { - // Lead with the same path resolveAppKeyFile would sign with, so the hub is - // told about the key actually in effect and never about a shadowed one. - candidates := []string{ - resolveAppKeyFile(keyFile, os.Getenv("GH_APP_KEY_FILE"), appID), - perAppIDKeyPath(appID), - keyFile, spokeAppKeyPath, spokeProvisionedAppKeyPath, - } - for _, p := range candidates { - if strings.TrimSpace(p) == "" { - continue - } - if fp, err := config.AppKeyFingerprintFromFile(p); err == nil && fp != "" { - return fp - } - } - return "" -} - -// hasPerHiveAppKey reports whether this spoke's key came from a per-hive -// provisioning secret rather than the cluster default. The provisioned mount is -// read-only, so its mere existence with real PEM content is the signal — a -// hub-delivered key can never create or alter it. -// -// When BOTH exist the hub-delivered PVC key is the one in effect (the callback -// repoints cfg.GitHub.KeyFile at it), so this only claims an override while the -// provisioned key is genuinely the one being used. -func hasPerHiveAppKey(keyFile string, appID int64) bool { - fp, err := config.AppKeyFingerprintFromFile(spokeProvisionedAppKeyPath) - if err != nil || fp == "" { - return false - } - // The provisioned key exists. It is the effective credential only if the - // resolved key file still points at it — resolveAppKeyFile is the single - // authority on that, so an unconfigured hive that has already taken delivery - // of a /data key (or a per-app-id key) correctly stops claiming a per-hive - // override. - return resolveAppKeyFile(keyFile, os.Getenv("GH_APP_KEY_FILE"), appID) == spokeProvisionedAppKeyPath -} - -// resolveAppKeyFile picks which App private key this process will actually sign -// with, given the configured key_file and the GH_APP_KEY_FILE env override. -// -// WHY THE /data PREFERENCE MATTERS -// -// A hub-delivered key lands at spokeAppKeyPath (/data, on the PVC) and the -// heartbeat callback repoints cfg.GitHub.KeyFile at it — but only in memory, for -// the life of that process. A hive whose config carries NO key_file (which is -// the state of the three live GHE hives this repairs) used to fall straight -// through to the read-only /secrets provisioning mount. That mount holds the -// stale, wrong key, and the spoke cannot write to it. So on every restart the -// hive would silently go back to signing with the key that cannot work, and the -// hub — seeing the wrong fingerprint reported again — would redeliver forever. -// The key would be delivered and never used: a fault that reads as fixed. -// -// So when nothing is explicitly configured, a key already present on the PVC is -// preferred over the provisioning mount. An EXPLICIT key_file or env override -// still wins outright: those are deliberate, and this must not silently redirect -// an operator who named a path. -// -// PER-APP-ID SELECTION (the both-keys fix) -// -// appID is the App this process is configured to authenticate AS -// (cfg.GitHub.AppID). When the hub has delivered a per-app-id key for exactly -// that App — /data/gh-app-key-.pem — it is preferred over the generic -// single-file paths, because it is provably the RIGHT key for the app_id we -// claim. This is what lets a github.com hive on a GitHub-Enterprise cluster sign -// with the github.com key even though its cluster default (and its single -// /data/gh-app-key.pem) is the GHE key. It sits just below an explicit -// key_file/env override — an operator who named a path still wins — and above -// the generic fallbacks. appID <= 0 disables it entirely, so nothing changes for -// a hive that reports no app_id. -func resolveAppKeyFile(configured, envOverride string, appID int64) string { - if v := strings.TrimSpace(envOverride); v != "" { - return v - } - if v := strings.TrimSpace(configured); v != "" { - // MIGRATION. A configured key_file that is the GENERIC path is not an - // operator's choice — it is a value older builds wrote automatically on - // every key delivery, and it does not name the App it holds. When we can - // see a per-app-id key for the app_id we actually claim, that key is - // correct by construction and the generic pin is stale, so ignore it. - // - // Without this, the ~33 spokes already carrying - // key_file: /data/gh-app-key.pem keep signing with whichever App's key - // happens to sit there — the live 404 Integration not found — because an - // explicit value short-circuits the per-app-id lookup below. - // - // Deliberately narrow: only the exact generic path is overridden, and - // only when a usable per-app-id key exists. Any other path is a genuine - // operator override (a hive on a third App with a bespoke key location) - // and still wins outright. - // - // BOTH generic paths qualify. /data/gh-app-key.pem is what older builds - // wrote on every key delivery; /secrets/gh-app-key.pem is what the - // PROVISIONING TEMPLATE hardcodes for every App-using hive - // (saas_provision.go). Neither names the App it holds, and neither was - // typed by an operator. Until /secrets was included here, a provisioned - // hive could never change forges: the hub could correct app_id all it - // liked and the spoke kept signing with the provisioned key, which on - // the spoke-cluster pool was a placeholder matching NEITHER real App. - if v == spokeAppKeyPath || v == spokeProvisionedAppKeyPath { - if p := perAppIDKeyPath(appID); p != "" { - if fp, err := config.AppKeyFingerprintFromFile(p); err == nil && fp != "" { - return p - } - } - } - return v - } - // Nothing explicitly configured. Prefer a per-app-id key matching the App we - // claim — the only key that is CORRECT-by-construction for this app_id — over - // the generic cluster/provisioned files. The fingerprint check (not mere - // existence) keeps an empty or truncated per-app file from shadowing a good - // generic key. - if p := perAppIDKeyPath(appID); p != "" { - if fp, err := config.AppKeyFingerprintFromFile(p); err == nil && fp != "" { - return p - } - } - // Same idea, but from the read-only provisioning mount: a hive provisioned - // with the fleet's full key set can sign as its configured App on its very - // first boot, before any heartbeat has delivered anything to the PVC. Ranked - // BELOW the PVC copy so a rotated key delivered by heartbeat always wins over - // the one frozen into the Secret at provision time. - if p := perAppIDProvisionedKeyPath(appID); p != "" { - if fp, err := config.AppKeyFingerprintFromFile(p); err == nil && fp != "" { - return p - } - } - // Prefer a usable hub-delivered key on the PVC; fall back to the provisioning - // mount only when /data has no parseable key. - if fp, err := config.AppKeyFingerprintFromFile(spokeAppKeyPath); err == nil && fp != "" { - return spokeAppKeyPath - } - return spokeProvisionedAppKeyPath -} - -// describeAppKeyFailure turns a bare wrapped os error from github.NewAppAuth -// into a message an operator can act on without reading the source: it names -// the path actually tried, the full resolution order that produced it, and the -// underlying cause. -// -// The generic "reading app key /secrets/gh-app-key.pem: no such file" that this -// replaces gave no hint that key_file, $GH_APP_KEY_FILE, the PVC path and the -// provisioning mount are all consulted in a fixed order — so the usual response -// was to put the key in the wrong one of the four. -func describeAppKeyFailure(configured, envOverride, resolved string, err error) string { - order := []string{ - fmt.Sprintf("$GH_APP_KEY_FILE=%s", describeKeySource(envOverride)), - fmt.Sprintf("github.key_file=%s", describeKeySource(configured)), - fmt.Sprintf("per-app-id PVC key %s/gh-app-key-.pem", spokeAppKeyDir), - fmt.Sprintf("per-app-id provisioning key %s/gh-app-key-.pem", spokeProvisionedAppKeyDir), - fmt.Sprintf("PVC fallback %s", spokeAppKeyPath), - fmt.Sprintf("provisioning mount %s", spokeProvisionedAppKeyPath), - } - return fmt.Sprintf( - "GitHub App private key could not be loaded from %q: %v. "+ - "Resolution order (first non-empty wins): %s. "+ - "Write a PEM-encoded RSA private key to that path, or point github.key_file at one.", - resolved, err, strings.Join(order, " → "), - ) -} - -// describeKeySource renders an unset key-file source as "(unset)" so the -// resolution order in describeAppKeyFailure reads unambiguously. -func describeKeySource(v string) string { - if strings.TrimSpace(v) == "" { - return "(unset)" - } - return v -} - var githubAppTokenCachePath = github.TokenCachePath func githubAppTokenHeartbeatFields(cfg *config.Config, detail string) (status, lastMintAt, lastErr string) { @@ -5763,13 +5416,6 @@ func (s labelPlanSink) QueuedPlan(epic *beads.Bead, paused bool) { s.logger.Warn("plan-from-label: architect unavailable, plan queued", "epic", epic.ID, "ref", epic.ExternalRef) } -// appKeyPaths snapshots the two App key path vars for a pkg/apphealth call. -// Read at call time on purpose: tests repoint these vars, and capturing them -// once would silently ignore that. -func appKeyPaths() apphealth.KeyPaths { - return apphealth.KeyPaths{Spoke: spokeAppKeyPath, Provisioned: spokeProvisionedAppKeyPath} -} - func healGitHubAppInstallation(ctx context.Context, appAuth *github.AppAuth, cfg *config.Config, logger *slog.Logger) { apphealth.Heal(ctx, appAuth, cfg, logger) } @@ -6905,334 +6551,6 @@ func runEvalCycle( } } -// loginCommandForBackend returns the login instruction for a given CLI backend. -func loginCommandForBackend(backend string) string { - switch backend { - case "claude": - return "Run: claude login" - case "copilot": - return "Run: copilot auth login" - case "gemini": - return "Run: gemini auth login" - case "goose": - return "Run: goose auth login" - default: - return "Run the login command for " + backend - } -} - -// loginScanAction is what the detector should do about one agent this cycle. -type loginScanAction int - -const ( - // loginScanIgnore: nothing that looks like a login problem, or a startup - // modal is on screen. Any sighting streak is cleared. - loginScanIgnore loginScanAction = iota - // loginScanDeferAuthenticated: the pane matched, but the backend credential - // is demonstrably valid, so this is residue or a stuck CLI — the manager's - // token-restart heal's case, not an operator's (kubestellar/hive#5291). - loginScanDeferAuthenticated - // loginScanDeferStreak: the pane matched and the credential is not provably - // good, but this is the first consecutive cycle to see it. - loginScanDeferStreak - // loginScanPause: pause the agent and page the operator. - loginScanPause -) - -// loginPauseMinSightings is how many CONSECUTIVE governor cycles must see a -// login pattern before the detector pauses (kubestellar/hive#5291). -// -// The manager's own pane poller learned this at its ~3s cadence, where a single -// sighting restarted healthy agents; it now requires loginStreakRestartMin = 3. -// The detector had no equivalent, and a pause is far more expensive than a -// restart — it is sticky, it needs a human to undo, and it cancels the agent -// context that hosts the heal. Two is deliberate rather than three: a governor -// cycle is minutes, not seconds, so each extra cycle is real delay for a -// genuine logout, and the credential gate above already covers the case this -// backstops. It matters most for backends with no credential file this process -// can check, where it is the only new protection. -const loginPauseMinSightings = 2 - -// loginSightingTracker counts CONSECUTIVE cycles in which each agent's pane -// matched a login pattern. A clean cycle resets the count to zero, so a match -// has to persist to accumulate — a single flicker never reaches the threshold. -type loginSightingTracker struct { - mu sync.Mutex - streak map[string]int -} - -func newLoginSightingTracker() *loginSightingTracker { - return &loginSightingTracker{streak: map[string]int{}} -} - -// loginSightings is the detector's process-scoped state. The governor cycle is -// a function rather than an object, so the consecutive-sighting counts have to -// outlive a single call; tests build their own tracker and pass it explicitly. -var loginSightings = newLoginSightingTracker() - -// observe records this cycle's reading for one agent and returns the resulting -// consecutive-sighting count (1 on the first sighting). -func (t *loginSightingTracker) observe(agent string, matched bool) int { - if t != nil { - t.mu.Lock() - defer t.mu.Unlock() - } - if t == nil { - // No tracker wired: behave as if every sighting is its own streak, which - // is exactly the pre-#5291 single-observation behaviour. - if matched { - return loginPauseMinSightings - } - return 0 - } - if !matched { - delete(t.streak, agent) - return 0 - } - t.streak[agent]++ - return t.streak[agent] -} - -// forget drops an agent's streak — on pause (it stops being scanned) and for -// agents that are no longer present, so the map cannot grow without bound -// across a long-lived process. -func (t *loginSightingTracker) forget(agent string) { - if t == nil { - return - } - t.mu.Lock() - defer t.mu.Unlock() - delete(t.streak, agent) -} - -// retain drops every agent not in the given set. -func (t *loginSightingTracker) retain(present map[string]bool) { - if t == nil { - return - } - t.mu.Lock() - defer t.mu.Unlock() - for name := range t.streak { - if !present[name] { - delete(t.streak, name) - } - } -} - -// loginScanDecision is the detector's whole judgement about one agent, as a -// pure function of what was observed. It exists apart from scanForLoginRequired -// so the decision can be tested against real pane text without a tmux session, -// a manager, or a governor cycle. -// -// sightings is the consecutive-cycle count INCLUDING this one. -// -// The credential gate is the fix for kubestellar/hive#5291: the detector used -// to pause on pane text alone, and the pane during and just after an -// interactive /login necessarily contains login-screen chrome — so it fired on -// the evidence the operator's own fix had just produced, seven minutes after -// the credential was already valid. Worse, Pause() cancels the agent context -// and tears down the poller that hosts the token-restart heal (#4606), which is -// the mechanism built for exactly "login prompt on screen, credential valid". -// Pausing first therefore disabled the machinery that would have fixed the pane -// it misread. -// -// Text matching cannot be narrowed out of this: two earlier fixes tried -// (tail-only matching, then a tighter copilot pattern) and this incident is the -// third false positive. The pane legitimately contains login text at the moment -// the credential is freshest, so the credential has to be consulted. -func loginScanDecision( - backend, paneText string, - compiled []*regexp.Regexp, - credentialValid bool, - sightings int, -) (loginScanAction, *regexp.Regexp) { - matched := loginScanMatch(backend, paneText, compiled) - return loginScanVerdict(matched != nil, credentialValid, sightings), matched -} - -// loginScanMatch reports which login pattern this pane trips, or nil for none. -// Separate from the verdict so the scan loop can match ONCE and use the answer -// both to advance the sighting streak and to decide. -func loginScanMatch(backend, paneText string, compiled []*regexp.Regexp) *regexp.Regexp { - // Stand down while a startup-blocking modal (folder trust, codex update, …) - // is on screen: that is not a login problem, and pausing the agent for it - // cancels the trust-prompt watcher that would answer it — the deadlock that - // kept copilot agents "sitting at login prompt" through every operator - // re-login (hivecommons/hive, 2026-08-22). The watcher answers the modal - // within seconds; if a REAL login prompt follows, the next detector tick - // sees it on a clean pane. - if agent.PaneShowsBlockingPrompt(backend, paneText) { - return nil - } - for _, re := range compiled { - if re.MatchString(paneText) { - return re - } - } - return nil -} - -// loginScanVerdict turns "what the pane showed" into "what to do". It returns -// loginScanIgnore whenever matched is false, which is what lets the scan loop -// rely on a non-Ignore verdict implying a non-nil pattern to log. -func loginScanVerdict(matched, credentialValid bool, sightings int) loginScanAction { - if !matched { - return loginScanIgnore - } - if credentialValid { - return loginScanDeferAuthenticated - } - if sightings < loginPauseMinSightings { - return loginScanDeferStreak - } - return loginScanPause -} - -type loginScanAgentManager interface { - AllStatuses() map[string]*agent.AgentProcess - GetOutput(name string, lines int) ([]string, error) - AgentHasValidCredential(agentName string) bool - RefreshAgentTokenFor(ctx context.Context, name string) error - Pause(name, trigger, reason string) error -} - -type loginScanNotifier interface { - Send(title, message string, priority notify.Priority) -} - -type loginScanAuditor interface { - AuditLog(user, action, detail, agent string) -} - -// scanForLoginRequired checks each running agent's tmux pane output for login-required -// patterns. When a match is found, the agent is paused and a notification is sent. -func scanForLoginRequired( - ctx context.Context, - cfg *config.Config, - agentMgr loginScanAgentManager, - notifier loginScanNotifier, - dashSrv loginScanAuditor, - logger *slog.Logger, - sightings *loginSightingTracker, -) { - patterns := cfg.Governor.Sensing.LoginPatterns - if len(patterns) == 0 { - return - } - - // Compile regex patterns, skipping empty and invalid ones - compiled := make([]*regexp.Regexp, 0, len(patterns)) - for _, p := range patterns { - if strings.TrimSpace(p) == "" { - continue - } - re, err := regexp.Compile("(?i)" + p) - if err != nil { - logger.Warn("invalid login pattern regex", "pattern", p, "error", err) - continue - } - compiled = append(compiled, re) - } - if len(compiled) == 0 { - return - } - - // Scan the pane TAIL only. A login prompt the CLI is genuinely stuck at - // sits at the BOTTOM of the pane; the 50-line window this used to read - // reached deep into scrollback, where agent WORK OUTPUT that merely - // mentions a pattern phrase lives — quality's scan findings quoting - // "gh auth login" from auth documentation got the agent paused mid-kick - // (hivecommons/hive, 2026-08-22 08:27, on a fully-authenticated CLI). - // Same discipline as the poller's tail-only match (#4577). - const paneLines = 12 - statuses := agentMgr.AllStatuses() - scanned := make(map[string]bool, len(statuses)) - for name, proc := range statuses { - if proc.State != "running" { - continue - } - scanned[name] = true - - output, err := agentMgr.GetOutput(name, paneLines) - if err != nil || len(output) == 0 { - continue - } - - joined := strings.Join(output, "\n") - backend := cfg.Agents[name].Backend - - // #5291: ask the CREDENTIAL, not just the pane. A valid credential plus - // a login prompt is the token-restart heal's case; only an invalid one - // needs a human. - credentialValid := agentMgr.AgentHasValidCredential(name) - - // Match once. The streak has to reflect what the pane SHOWED, including - // on the cycles where a gate below declines to act on it, so the - // sighting is recorded before the verdict is taken. - re := loginScanMatch(backend, joined, compiled) - streak := sightings.observe(name, re != nil) - - switch loginScanVerdict(re != nil, credentialValid, streak) { - case loginScanIgnore: - continue - case loginScanDeferAuthenticated: - // Logged at Info, not Warn: this is the detector working correctly, - // and it is the line that explains an agent staying up with login - // text on its pane. - logger.Info("login pattern matched but the backend credential is valid — leaving it to the token-restart heal", - "agent", name, "backend", backend, "pattern", re.String()) - continue - case loginScanDeferStreak: - logger.Info("login pattern matched but not yet on enough consecutive cycles — deferring", - "agent", name, "backend", backend, "pattern", re.String(), - "sightings", streak, "required", loginPauseMinSightings) - continue - case loginScanPause: - logger.Warn("login required detected", - "agent", name, - "pattern", re.String(), - "sightings", streak, - ) - sightings.forget(name) - - // Attempt a per-agent token re-cache BEFORE pausing. On an - // App-authenticated hive the likeliest cause of a "gh auth - // login" prompt is an expired scoped-token cache (#4072); - // re-minting it now means the operator's Resume immediately - // works instead of 401ing straight back into this pause. - // Best-effort: hives without App auth (or agents without a - // dedicated UID) simply skip it. - if refreshErr := agentMgr.RefreshAgentTokenFor(ctx, name); refreshErr == nil { - logger.Info("re-cached per-agent scoped token before login-detector pause", "agent", name) - } - - // Pause the agent instead of restarting - if pauseErr := agentMgr.Pause(name, "login-detector", "login required detected"); pauseErr != nil { - logger.Warn("failed to pause agent after login detection", - "agent", name, "error", pauseErr) - } else { - dashSrv.AuditLog("system", "pause", "trigger=login-detector", name) - } - - // Determine the login instruction based on the agent's backend - loginCmd := loginCommandForBackend(backend) - - notifier.Send( - fmt.Sprintf("\U0001F511 Login required: %s", name), - fmt.Sprintf( - "Agent '%s' needs authentication. Open the agent's terminal "+ - "(tmux attach -t hive-%s) and run the login command for the CLI (%s). %s", - name, name, backend, loginCmd, - ), - notify.PriorityHigh, - ) - } - } - // Agents that vanished (removed from config, stopped) must not keep a - // streak alive in the map for the life of the process. - sightings.retain(scanned) -} - func convertKnowledgeLayers(cfgLayers []config.KnowledgeLayer) []knowledge.LayerConfig { layers := make([]knowledge.LayerConfig, len(cfgLayers)) for i, l := range cfgLayers { @@ -7265,185 +6583,6 @@ func curatorConfigFromHive(c config.KnowledgeCurator) knowledge.CuratorConfig { // hiveIDFilePath is the persistent file where the Hive ID is stored across restarts. var hiveIDFilePath = "/data/hive-id" -// loadOrGenerateHiveID reads the Hive ID from disk, or generates and persists a new one. -const ( - // selfUpgradeMaxAttempts bounds how many times a spoke retries an upgrade - // that keeps leaving the image unchanged. Bounded rather than unlimited so a - // genuinely broken hive (e.g. missing RBAC) stops thrashing its pod, and - // bounded rather than "never again" so a transient failure still converges. - selfUpgradeMaxAttempts = 5 - // selfUpgradeBaseBackoff is the delay before retry #2; it doubles per - // attempt up to selfUpgradeMaxBackoff. - selfUpgradeBaseBackoff = 2 * time.Minute - // selfUpgradeMaxBackoff caps the exponential backoff between retries. - selfUpgradeMaxBackoff = 30 * time.Minute - // selfUpgradeFailureExitCode marks a process exit caused by a FAILED - // self-upgrade. Distinct from 0 so the failure is visible in the container's - // termination state instead of looking like a clean shutdown. - selfUpgradeFailureExitCode = 17 -) - -// upgradeMarker is the on-PVC record at /data/upgrade-requested. It survives -// pod restarts (that is the whole point: the process exits as part of an -// upgrade), so it is the only place attempt bookkeeping can live. -type upgradeMarker struct { - TargetSHA string `json:"target_sha"` - CurrentSHA string `json:"current_sha"` - RequestedAt time.Time `json:"requested_at"` - Attempts int `json:"attempts"` - LastError string `json:"last_error,omitempty"` -} - -// parseUpgradeMarker decodes a marker, tolerating the legacy format that had no -// attempts/last_error fields. A legacy marker counts as one prior attempt so an -// already-wedged hive gets retries under the new budget instead of being -// treated as fresh. -func parseUpgradeMarker(data []byte) upgradeMarker { - var m upgradeMarker - if err := json.Unmarshal(data, &m); err != nil { - return upgradeMarker{} - } - if m.Attempts < 1 { - m.Attempts = 1 - } - return m -} - -// sameUpgradeTarget reports whether two target SHAs refer to the same commit, -// tolerating short/full SHA length mismatch the way the hub's sameCommit does. -// A DIFFERENT target must reset the attempt budget, so this comparison is what -// keeps the latch from outliving the upgrade it was created for. -func sameUpgradeTarget(a, b string) bool { - if a == "" || b == "" { - return false - } - n := len(a) - if len(b) < n { - n = len(b) - } - return strings.EqualFold(a[:n], b[:n]) -} - -func writeUpgradeMarker(path string, m upgradeMarker, logger *slog.Logger) { - data, err := json.Marshal(m) - if err != nil { - logger.Warn("failed to encode upgrade marker", "error", err) - return - } - if err := os.WriteFile(path, data, 0o644); err != nil { - logger.Warn("failed to write upgrade marker", "path", path, "error", err) - } -} - -// recordUpgradeError annotates the existing marker with the cause of the failed -// attempt so the NEXT boot can log why the previous one did not land — without -// it the reason dies with the process and the failure is invisible. -// -// upgradeMarkerPath and lastUpgradeOutcomePath are the two on-PVC records the -// spoke keeps for auto-upgrade visibility (#7092). The marker at -// upgradeMarkerPath is present ONLY while an instructed upgrade has not landed -// (in flight or terminally failed) and is cleared the moment the new image -// boots — so it can NEVER represent a success. lastUpgradeOutcomePath is the -// durable companion that records the last upgrade that actually LANDED, so the -// dashboard can tell "attempted and succeeded" apart from "never attempted" -// instead of letting a blank panel masquerade as success. -const ( - upgradeMarkerPath = "/data/upgrade-requested" - lastUpgradeOutcomePath = "/data/last-upgrade-outcome" -) - -// upgradeOutcome is the durable "last upgrade LANDED" record. Written on the -// boot that completes an upgrade (reconcileUpgradeOutcomeAtBoot), it survives — -// unlike upgradeMarker, which is removed the moment the target image boots. -type upgradeOutcome struct { - TargetSHA string `json:"target_sha"` - CurrentSHA string `json:"current_sha"` - RequestedAt time.Time `json:"requested_at"` - CompletedAt time.Time `json:"completed_at"` -} - -func writeUpgradeOutcome(path string, o upgradeOutcome, logger *slog.Logger) { - data, err := json.Marshal(o) - if err != nil { - logger.Warn("failed to encode upgrade outcome", "error", err) - return - } - if err := os.WriteFile(path, data, 0o644); err != nil { - logger.Warn("failed to write upgrade outcome", "path", path, "error", err) - } -} - -// reconcileUpgradeOutcomeAtBoot records a SUCCESSFUL self-upgrade. An in-flight -// marker whose target equals the now-running commit means the instructed -// upgrade LANDED: the pod booted on the target image. That success would -// otherwise vanish — the next upgrade instruction silently discards the stale -// marker, so a hive that updated cleanly looks identical to one that never -// tried. This persists the success durably and clears the in-flight marker so -// it stops reading as "not landed". A marker whose target does NOT match the -// running commit is still in flight or failed and is left untouched for that -// surface. Called once at startup, before the heartbeat loop and dashboard come -// up, so the dashboard always sees the reconciled state. -func reconcileUpgradeOutcomeAtBoot(markerPath, outcomePath, runningSHA string, logger *slog.Logger) { - data, err := os.ReadFile(markerPath) - if err != nil { - return - } - m := parseUpgradeMarker(data) - if m.TargetSHA == "" || runningSHA == "" || !sameUpgradeTarget(m.TargetSHA, runningSHA) { - return - } - writeUpgradeOutcome(outcomePath, upgradeOutcome{ - TargetSHA: m.TargetSHA, - CurrentSHA: m.CurrentSHA, - RequestedAt: m.RequestedAt, - CompletedAt: time.Now().UTC(), - }, logger) - if err := os.Remove(markerPath); err != nil && !os.IsNotExist(err) { - logger.Warn("failed to clear landed upgrade marker", "path", markerPath, "error", err) - } - logger.Info("self-upgrade landed: recorded successful upgrade outcome", - "target", m.TargetSHA, "current", runningSHA) -} - -// upgradeFailureSummary renders what the hub shows an operator. An empty -// LastError must never render as a dangling "attempts: " - a colon promising a -// reason and delivering none is worse than saying the reason was not captured, -// because it reads as truncation and sends the reader looking for the rest. -func upgradeFailureSummary(attempts int, lastError string) string { - if strings.TrimSpace(lastError) == "" { - return fmt.Sprintf("self-upgrade failed after %d attempts (no error recorded; the image never changed - check that the deployment tracks a tag carrying the target SHA)", attempts) - } - return fmt.Sprintf("self-upgrade failed after %d attempts: %s", attempts, lastError) -} - -func recordUpgradeError(path string, upgradeErr error, logger *slog.Logger) { - if upgradeErr == nil { - return - } - // A marker that cannot be read is not a reason to drop the cause. The - // earlier version returned on ANY read error, which left LastError empty - // and produced the bare "self-upgrade failed after 5 attempts: " the hub - // relays to the dashboard - an alert naming a failure and nothing about - // it. Losing the attempt count is survivable; losing the reason is what - // makes the failure undiagnosable, so rebuild the marker around the error - // instead. An ABSENT marker is different: no attempt is in flight, and - // creating one here would later be mistaken for a real attempt, so the - // no-op stands for that case only. - var m upgradeMarker - data, err := os.ReadFile(path) - switch { - case os.IsNotExist(err): - return - case err != nil: - logger.Warn("upgrade marker unreadable; recording the error against a fresh marker", - "path", path, "error", err) - default: - m = parseUpgradeMarker(data) - } - m.LastError = upgradeErr.Error() - writeUpgradeMarker(path, m, logger) -} - func loadOrGenerateHiveID(logger *slog.Logger) string { if envID := os.Getenv("HIVE_ID"); envID != "" { if err := os.WriteFile(hiveIDFilePath, []byte(envID+"\n"), 0o644); err == nil { @@ -8108,28 +7247,6 @@ const taskListSweepInterval = 15 * time.Minute // chance of editing a suggestion under a reader's cursor. const duplicateSweepInterval = time.Hour -// trustedMergerFunc resolves a GitHub login against the hive's authorized-users -// allowlist and reports whether it holds at least config.RoleMerger — the same -// bar requireMergerOrOwnerRole enforces on the dashboard queue endpoint (audit -// F3). -// -// Fails CLOSED: a nil config or a login absent from the allowlist is NOT -// trusted, so an unclassifiable actor can never merge. cfg is read on every -// call so a config reload that grants or revokes the merger tier takes effect -// without a restart. -func trustedMergerFunc(cfg *config.Config) github.MergerAuthorizer { - return func(login string) bool { - if cfg == nil || strings.TrimSpace(login) == "" { - return false - } - role, ok := cfg.Dashboard.AuthorizedRole(login) - if !ok { - return false - } - return config.RoleAtLeast(role, config.RoleMerger) - } -} - // runAutoMergeSweepIfDue drains the label-queued auto-merge queue (the human // "Approved ... for Hive auto-merge" path) at most once per // autoMergeSweepInterval. All merge-eligibility decisions — queue-approval @@ -8354,141 +7471,13 @@ func runDuplicateSweepIfDue(ctx context.Context, cfg *config.Config, ghClient *g "commented": strconv.Itoa(result.Commented), }, }) -} // mergeEligiblePath is a var (not a const) only so tests can point -// mergeTargetEligible at a temp file; production never reassigns it. -var mergeEligiblePath = "/var/run/hive-metrics/merge-eligible.json" +} var ( ciFailingPath = "/var/run/hive-metrics/ci-failing.json" intentVerdictsPath = "/var/run/hive-metrics/intent-verdicts.json" ) -// acmmHoldGatedMinLevel / acmmHoldGatedMaxLevel bracket the ACMM levels whose -// merge policy is "hold-gated" — every agent-opened PR gets a "hold" label and -// no agent merges (see src/pkg/config/packs/level-{3,4,5}.yaml). L1/L2 are -// "manual" (agents open no PRs) and L6 is "auto-merge on green CI, no hold -// label", so both fall outside this range. Used by the F6 hold-label decider. -const ( - acmmHoldGatedMinLevel = 3 - acmmHoldGatedMaxLevel = 5 -) - -// shouldHoldAgentPR keeps public outreach claims human-reviewed even at L6, -// where ordinary agent PRs may auto-merge. The general ACMM hold gate remains -// unchanged for all roles at L3-L5. -func shouldHoldAgentPR(agentName string, level int) bool { - if strings.EqualFold(strings.TrimSpace(agentName), "outreach") { - return true - } - return level >= acmmHoldGatedMinLevel && level <= acmmHoldGatedMaxLevel -} - -// mergeableJSONUnknown is the explicit wire value for "mergeability was never -// determined". It is spelled out rather than left as "" so a consumer reading -// merge-eligible.json cannot mistake an unpopulated field for a definitive -// "no" — the failure mode that made every PR read as unmergeable. -const mergeableJSONUnknown = "unknown" - -// mergeTargetEligible reports whether (repo, number) currently appears in the -// governor's merge-eligible.json AT the expected head SHA. It reads the file -// FRESH on every call (never caches) because eligibility is recomputed each -// governor cycle — a stale cache could authorize a PR that has since fallen out -// of the list. On any read/parse error it returns false (FAIL CLOSED): if we -// cannot prove the target is eligible, we must not authorize the merge. -// -// M4 (CWE-367, TOCTOU): the governor records the head SHA it observed when it -// deemed the PR eligible (eligiblePR.HeadSHA). A branch can move between that -// review and the merge relay firing, so matching (repo, number) alone would let -// a moved head merge at a commit the governor never vetted. We therefore also -// require the entry's stored head_sha to equal expectSHA. A mismatch — or a -// stored SHA that is empty (governor could not observe it) — fails closed; the -// relay's SHA pin then fails the merge cleanly if a stale request slips through. -// -// merge-eligible.json stores repos as "owner/repo"; a MergeRequest.Repo may be -// bare ("repo") or fully qualified ("owner/repo"). We match on the bare repo -// name (the segment after the last "/") plus the number, so both request forms -// resolve to the same eligible entry without depending on the org prefix. -func mergeTargetEligible(repo string, number int, expectSHA string) bool { - data, err := os.ReadFile(mergeEligiblePath) - if err != nil { - return false // fail closed: no list ⇒ nothing is eligible - } - var payload struct { - Items []struct { - Number int `json:"number"` - Repo string `json:"repo"` - HeadSHA string `json:"head_sha"` - } `json:"merge_eligible"` - } - if err := json.Unmarshal(data, &payload); err != nil { - return false // fail closed: unparseable list ⇒ deny - } - want := bareRepoName(repo) - wantSHA := strings.TrimSpace(expectSHA) - for _, it := range payload.Items { - if it.Number == number && bareRepoName(it.Repo) == want { - // M4: bind authorization to the governor-observed head. An empty - // stored SHA cannot be proven to match, so it fails closed rather - // than authorizing an unpinned head. - return strings.TrimSpace(it.HeadSHA) != "" && strings.TrimSpace(it.HeadSHA) == wantSHA - } - } - return false -} - -// bareRepoName returns the repo segment after the last "/", so "owner/repo" and -// "repo" compare equal. Used to match a MergeRequest.Repo against the -// "owner/repo" entries in merge-eligible.json regardless of prefix. -func bareRepoName(repo string) string { - if i := strings.LastIndex(repo, "/"); i >= 0 { - return repo[i+1:] - } - return repo -} - -// bindMergeAuthz wraps the manager's agent/UID/CanMerge authorizer with the -// F4 target-binding checks (CWE-863). The inner authz owns the "may this agent -// merge at all" decision; this wrapper owns "is THIS specific target one the -// governor deemed eligible, at a pinned SHA". Both must pass before MergePR is -// reached. Ordering: run the agent/UID/CanMerge check first (cheapest, and it -// gives the clearest denial reason), then the SHA + eligible-list binding. -func bindMergeAuthz(inner func(agent string, fileUID int) error) github.MergeRequestAuthorizer { - return func(agent string, fileUID int, repo string, number int, expectSHA string) error { - if err := inner(agent, fileUID); err != nil { - return err - } - // (a) Require a pinned head SHA. An empty expectSHA means "merge whatever - // HEAD is now", which is the TOCTOU hole: a PR that was eligible when the - // governor last looked could have had a malicious commit pushed since. - // MergePR passes expectSHA as the required head SHA, so a moved head fails - // cleanly — but only if we insist it is set. - if strings.TrimSpace(expectSHA) == "" { - return fmt.Errorf("merge target %s#%d has no expected head SHA — refusing to merge an unpinned head (TOCTOU guard)", repo, number) - } - // (b) Require the target to be in the governor's current merge-eligible - // list AT the expected head SHA. This binds authorization to a PR the - // hive actually deemed landable this cycle, at the exact commit it - // reviewed, so an injected agent cannot request landing an arbitrary - // reachable PR (e.g. its own) whose required checks happen to pass, nor - // land an eligible PR at a head that moved after review (M4, CWE-367). - // Read fresh + fail closed (see mergeTargetEligible). - if !mergeTargetEligible(repo, number, expectSHA) { - return fmt.Errorf("merge target %s#%d is not in the current merge-eligible list at head %s — only governor-approved PRs may be landed via the merge relay, and only at the reviewed head SHA", repo, number, expectSHA) - } - return nil - } -} - -// mergeableJSON renders a tri-state mergeability verdict for the -// merge-eligible.json marker, mapping the unknown zero value to an explicit -// "unknown" rather than an empty string. -func mergeableJSON(m github.Mergeable) string { - if m == github.MergeableUnknown { - return mergeableJSONUnknown - } - return string(m) -} - // claimLedger holds the duplicate-PR guard's persisted issue→PR claim mapping // across eval cycles. It is loaded lazily on first use (and retried on a load // failure) rather than at startup, so a missing or corrupt /data ledger can @@ -8593,330 +7582,6 @@ func claimingPRRedStale(cfg *config.Config, actionable *github.ActionableResult) } } -func writeIntentVerdicts( - ctx context.Context, - cfg *config.Config, - ghClient *github.Client, - actionable *github.ActionableResult, - beadStores map[string]*beads.Store, - logger *slog.Logger, -) map[string]intent.Verdict { - verdicts := make(map[string]intent.Verdict) - if cfg == nil || actionable == nil { - return verdicts - } - _ = os.MkdirAll("/var/run/hive-metrics", 0o755) - aiAuthor := strings.TrimSpace(cfg.EffectiveAIAuthor()) - intentCfg := intent.Config{ - TestPathPatterns: cfg.Intent.TestPathPatterns, - DocsPathPatterns: cfg.Intent.DocsPathPatterns, - GuardrailPathPatterns: cfg.Intent.GuardrailPathPatterns, - FeatureSignals: cfg.Intent.FeatureSignals, - } - var alignmentReviewer *intent.AlignmentReviewer - if strings.TrimSpace(cfg.Intent.AlignmentModel) != "" { - endpoint, apiKey, _ := cfg.Governor.ResolveReviewer() - var err error - alignmentReviewer, err = intent.NewAlignmentReviewer(intent.AlignmentReviewerConfig{ - Endpoint: endpoint, - APIKey: apiKey, - Model: cfg.Intent.AlignmentModel, - }) - if err != nil { - logger.Warn("intent alignment reviewer disabled", "error", err) - } - } - type verdictRecord struct { - Repo string `json:"repo"` - Number int `json:"number"` - Title string `json:"title"` - Author string `json:"author"` - Enforced bool `json:"enforced"` - Verdict intent.Verdict `json:"verdict"` - Classify string `json:"classification_reason"` - FetchError string `json:"fetch_error,omitempty"` - } - records := make([]verdictRecord, 0, len(actionable.PRs.Items)) - for _, pr := range actionable.PRs.Items { - fullRepo := fullRepoName(pr.Repo, cfg.Project.Org) - key := fmt.Sprintf("%s/%d", fullRepo, pr.Number) - agentPR := aiAuthor != "" && strings.EqualFold(pr.Author, aiAuthor) - record := verdictRecord{ - Repo: fullRepo, - Number: pr.Number, - Title: pr.Title, - Author: pr.Author, - Enforced: cfg.Intent.Enforce, - } - if !agentPR { - class := intent.Classify(intent.PR{Title: pr.Title, Labels: pr.Labels, Author: pr.Author, AgentAuthor: false}, intentCfg) - verdict := intent.Evaluate(class, intent.Evidence{}) - verdicts[key] = verdict - record.Verdict = verdict - record.Classify = class.Reason - records = append(records, record) - continue - } - body, files, approved, err := fetchIntentPREvidence(ctx, ghClient, fullRepo, pr.Number) - if err != nil { - verdict := intent.Verdict{ - Tier: intent.Tier1, - Authorized: false, - Reason: "intent evidence unavailable: " + err.Error(), - AgentPR: true, - } - verdicts[key] = verdict - record.Verdict = verdict - record.FetchError = err.Error() - records = append(records, record) - logger.Warn("intent verification evidence fetch failed", "repo", fullRepo, "number", pr.Number, "error", err) - continue - } - class := intent.Classify(intent.PR{ - Title: pr.Title, - Body: body, - Labels: pr.Labels, - Files: files, - Author: pr.Author, - AgentAuthor: true, - }, intentCfg) - evidence := intent.BuildEvidenceForRepo(body, fullRepo, beadStores, approved) - verdict := intent.Evaluate(class, evidence) - issueTexts, issueErr := fetchIntentIssueTexts(ctx, ghClient, fullRepo, body) - if issueErr != nil { - logger.Warn("intent alignment issue evidence fetch failed", "repo", fullRepo, "number", pr.Number, "error", issueErr) - } - refs := intent.LinkedIssueRefs(body, fullRepo) - alignCtx := intent.BuildAlignmentContext(intent.PR{ - Title: pr.Title, - Body: body, - Labels: pr.Labels, - Files: files, - Author: pr.Author, - AgentAuthor: true, - }, issueTexts, beadStores, refs) - alignment := intent.EvaluateAlignment(alignCtx, class.Tier, intentCfg) - if alignmentReviewer != nil { - modelVerdict, err := alignmentReviewer.Review(ctx, alignCtx) - if err != nil { - logger.Warn("intent alignment model review failed open", "repo", fullRepo, "number", pr.Number, "error", err) - alignment = intent.MergeAlignment(alignment, nil, err) - } else { - alignment = intent.MergeAlignment(alignment, &modelVerdict, nil) - } - } - verdict.Alignment = &alignment - verdicts[key] = verdict - record.Verdict = verdict - record.Classify = class.Reason - records = append(records, record) - if !verdict.Authorized { - logger.Info("intent authorization denied", "repo", fullRepo, "number", pr.Number, "tier", verdict.Tier, "reason", verdict.Reason, "enforce", cfg.Intent.Enforce) - } - if alignment.Misaligned() { - logger.Info("intent alignment denied", "repo", fullRepo, "number", pr.Number, "reason", alignment.Rationale, "enforce", cfg.Intent.Enforce) - recordIntentAlignmentAdvisory(beadStores, fullRepo, pr.Number, alignment, logger) - } - } - payload := map[string]any{ - "generated_at": time.Now().UTC().Format(time.RFC3339), - "enforced": cfg.Intent.Enforce, - "verdicts": records, - } - if data, err := json.Marshal(payload); err == nil { - atomicWrite(intentVerdictsPath, data) - } else { - logger.Warn("failed to marshal intent verdicts", "error", err) - } - return verdicts -} - -func fetchIntentPREvidence(ctx context.Context, ghClient *github.Client, repo string, number int) (string, []intent.ChangedFile, bool, error) { - if ghClient == nil || ghClient.GoGitHub() == nil { - return "", nil, false, github.ErrNoGitHubClient - } - owner, repoName, ok := strings.Cut(repo, "/") - if !ok || owner == "" || repoName == "" { - return "", nil, false, fmt.Errorf("invalid repo %q", repo) - } - client := ghClient.GoGitHub() - pr, _, err := client.PullRequests.Get(ctx, owner, repoName, number) - if err != nil { - return "", nil, false, fmt.Errorf("getting PR: %w", err) - } - var files []intent.ChangedFile - fileOpts := &gh.ListOptions{PerPage: 100} - for { - page, resp, err := client.PullRequests.ListFiles(ctx, owner, repoName, number, fileOpts) - if err != nil { - return "", nil, false, fmt.Errorf("listing PR files: %w", err) - } - for _, f := range page { - files = append(files, intent.ChangedFile{ - Filename: f.GetFilename(), - Status: f.GetStatus(), - Additions: f.GetAdditions(), - Deletions: f.GetDeletions(), - }) - } - if resp == nil || resp.NextPage == 0 { - break - } - fileOpts.Page = resp.NextPage - } - if reported := pr.GetChangedFiles(); reported > len(files) { - return "", nil, false, fmt.Errorf("incomplete PR file list: GitHub reported %d changed files but API returned %d; intent alignment requires the complete changed-file list", reported, len(files)) - } - approved, err := hasMaintainerApproval(ctx, client, owner, repoName, number) - if err != nil { - return "", nil, false, err - } - return pr.GetBody(), files, approved, nil -} - -func fetchIntentIssueTexts(ctx context.Context, ghClient *github.Client, defaultRepo, body string) ([]intent.TextEvidence, error) { - if ghClient == nil || ghClient.GoGitHub() == nil { - return nil, github.ErrNoGitHubClient - } - client := ghClient.GoGitHub() - refs := intent.LinkedIssueRefs(body, defaultRepo) - out := make([]intent.TextEvidence, 0, len(refs)) - for _, ref := range refs { - repo := ref.Repo - if repo == "" { - repo = defaultRepo - } - owner, repoName, ok := strings.Cut(repo, "/") - if !ok || owner == "" || repoName == "" { - continue - } - issue, _, err := client.Issues.Get(ctx, owner, repoName, ref.Number) - if err != nil { - return out, fmt.Errorf("getting linked issue %s#%d: %w", repo, ref.Number, err) - } - out = append(out, intent.TextEvidence{ - Source: fmt.Sprintf("issue %s#%d", repo, ref.Number), - Title: issue.GetTitle(), - Body: issue.GetBody(), - }) - } - return out, nil -} - -func recordIntentAlignmentAdvisory(stores map[string]*beads.Store, repo string, number int, alignment intent.AlignmentVerdict, logger *slog.Logger) { - store := stores["intent"] - if store == nil { - store = stores["quality"] - } - if store == nil { - for _, candidate := range stores { - if candidate != nil { - store = candidate - break - } - } - } - if store == nil { - return - } - title := fmt.Sprintf("Intent alignment drift in %s#%d", repo, number) - // "/#", NOT "gh-/#". The old form fused the - // source prefix into the org when the digest built its URL, so every one of - // these rendered a link to a github.com/gh- that does not exist - // (#6080). The renderer strips the prefix defensively for beads already - // written this way; this stops writing new ones. - ref := fmt.Sprintf("%s#%d", repo, number) - // Beads created before that change carry the prefixed form. Matching both - // keeps this idempotent across the change: without it the first run after - // upgrading would fail to recognise the existing bead and open a duplicate. - legacyRef := "gh-" + ref - for _, b := range store.List(beads.ListFilter{}) { - if b.Type == beads.TypeAdvisory && b.Title == title && - (b.ExternalRef == ref || b.ExternalRef == legacyRef) && - b.Status != beads.StatusClosed && b.Status != beads.StatusDone { - return - } - } - b, err := store.Create(title, beads.TypeAdvisory, beads.PriorityHigh, "intent", ref) - if err != nil { - logger.Warn("failed to record intent alignment advisory", "repo", repo, "number", number, "error", err) - return - } - _ = store.Update(b.ID, func(bead *beads.Bead) { - bead.Notes = alignmentSummary(alignment) - }) -} - -func alignmentSummary(alignment intent.AlignmentVerdict) string { - var parts []string - if alignment.Rationale != "" { - parts = append(parts, alignment.Rationale) - } - for _, f := range alignment.DeterministicFindings { - if f.Status == intent.AlignmentStatusMisaligned { - parts = append(parts, f.Code+": "+f.Reason+" ("+strings.Join(f.Files, ", ")+")") - } - } - if alignment.Model != nil && alignment.Model.Status == intent.AlignmentStatusMisaligned { - parts = append(parts, "model: "+alignment.Model.Rationale) - } - if len(parts) == 0 { - return "intent alignment check reported misalignment" - } - return strings.Join(parts, "\n") -} - -func hasMaintainerApproval(ctx context.Context, client *gh.Client, owner, repo string, number int) (bool, error) { - opts := &gh.ListOptions{PerPage: 100} - latest := make(map[string]string) - maintainer := make(map[string]bool) - for { - reviews, resp, err := client.PullRequests.ListReviews(ctx, owner, repo, number, opts) - if err != nil { - return false, fmt.Errorf("listing PR reviews: %w", err) - } - for _, review := range reviews { - login := review.GetUser().GetLogin() - if login == "" { - continue - } - if maintainerAssociation(review.GetAuthorAssociation()) { - switch review.GetState() { - case "APPROVED", "CHANGES_REQUESTED", "DISMISSED": - latest[login] = review.GetState() - } - maintainer[login] = true - } - } - if resp == nil || resp.NextPage == 0 { - break - } - opts.Page = resp.NextPage - } - approved := false - for login, state := range latest { - if !maintainer[login] { - continue - } - switch state { - case "CHANGES_REQUESTED": - return false, nil - case "APPROVED": - approved = true - } - } - return approved, nil -} - -func maintainerAssociation(association string) bool { - switch strings.ToUpper(strings.TrimSpace(association)) { - case "OWNER", "MEMBER", "COLLABORATOR": - return true - default: - return false - } -} - func fullRepoName(repo, org string) string { if strings.Contains(repo, "/") || org == "" { return repo @@ -9034,441 +7699,6 @@ func auditPRAgents(org string, since time.Time, auditPath string) map[string]str return out } -// anyRequiredCheckFailing reports whether any of a PR's failing check names -// is in the operator-declared required set. -func anyRequiredCheckFailing(failing []string, required map[string]bool) bool { - for _, name := range failing { - if required[name] { - return true - } - } - return false -} - -// mergeBucket is where the merge-eligible classifier files a PR: the -// merge-eligible.json list, the ci-failing.json list, or neither. -type mergeBucket int - -const ( - mergeBucketSkip mergeBucket = iota - mergeBucketFailing - mergeBucketEligible -) - -// mergeGates bundles the per-tick inputs the classifier applies beyond the -// PR itself: the intent and review artifacts and the operator's -// required-check set. -type mergeGates struct { - enforceIntent bool - intentVerdicts map[string]intent.Verdict - requireReviewApproval bool - reviewArtifact review.Artifact - reviewLoaded bool - requiredChecks map[string]bool -} - -// classifyMergeEligibility is THE merge-eligibility rule: the one place that -// decides whether a PR goes to merge-eligible.json (the sweep would merge it -// now), ci-failing.json (its author has CI to fix), or neither. It returns -// the bucket and, for the dashboard, the same decision as a MergeVerdict with -// the reason spelled out (hivecommons/hive#7478): the pill is painted from -// this verdict, so green on the card means exactly what the sweep means by -// eligible, and never the looser "GitHub says mergeable". -// -// intentReason is non-empty only when the intent gate excluded the PR; the -// caller logs it (the log line carries the verdict tier, which this function -// does not need). -func classifyMergeEligibility(pr github.PullRequest, held bool, fullRepo string, g mergeGates) (bucket mergeBucket, verdict github.MergeVerdict, intentReason string) { - // blockedOrOutstanding is the verdict for a PR the sweep will not take - // for a reason of its own: the state still depends on what GitHub says, - // because a conflicting PR is blocked whatever else is outstanding, and - // one whose mergeability was never fetched is unknown, not amber. The - // sweep's reason is kept in every state (hivecommons/hive#7515): on a - // protected branch a red required check or a missing review is exactly - // what makes GitHub say "blocked", so dropping it for the bare enum sent - // the operator to GitHub to learn what this function already knew. - blockedOrOutstanding := func(reason string) github.MergeVerdict { - switch pr.Mergeable { - case github.MergeableNo: - return github.MergeVerdict{State: github.MergeVerdictBlocked, Reason: notMergeableReason(pr, reason)} - case github.MergeableUnknown: - return github.MergeVerdict{State: github.MergeVerdictUnknown, Reason: mergeabilityUnknownReason + "; " + reason} - } - return github.MergeVerdict{State: github.MergeVerdictOutstanding, Reason: reason} - } - - if pr.Draft { - return mergeBucketSkip, github.MergeVerdict{State: github.MergeVerdictBlocked, Reason: "draft — mark ready for review to enter the sweep"}, "" - } - if g.enforceIntent { - if v, ok := g.intentVerdicts[fmt.Sprintf("%s/%d", fullRepo, pr.Number)]; ok && v.AgentPR && !v.MergeAllowed() { - reason := v.Reason - if v.Authorized && v.Alignment != nil && v.Alignment.Misaligned() { - reason = intent.ReasonAlignmentMisaligned + ": " + v.Alignment.Rationale - } - return mergeBucketSkip, blockedOrOutstanding("intent verification: " + reason), reason - } - } - - if pr.CIStatus == "failure" { - // A PR red ONLY on non-required checks (perma-red Playwright - // shards, coverage) that GitHub itself reports mergeable is NOT a - // failing PR — it is merge-eligible, mirroring the - // pending-but-mergeable rule below. Without this, every dependabot - // PR on a repo with permanently-red optional checks classified as - // "failure", landed in ci-failing.json where no sweep or agent - // would ever merge it, and accumulated indefinitely (observed on - // kubestellar/console 2026-08-28: 16 dependabot PRs, oldest 11 - // days). Gated on an operator-declared required-check set: with no - // set configured we cannot distinguish required from optional and - // keep the old fail-closed behavior. The merge step re-enforces - // branch protection, so this cannot merge anything GitHub blocks. - onlyOptionalRed := len(g.requiredChecks) > 0 && - !anyRequiredCheckFailing(pr.FailingChecks, g.requiredChecks) && - pr.Mergeable == github.MergeableYes - if !onlyOptionalRed { - reason := "CI failing" - if len(pr.FailingChecks) > 0 { - reason += ": " + strings.Join(pr.FailingChecks, ", ") - } - if len(g.requiredChecks) == 0 && pr.Mergeable == github.MergeableYes { - // GitHub calls it mergeable (unstable): nothing REQUIRED is - // red. The sweep still refuses it because, with no - // required-check set declared, it cannot tell optional from - // required. Say so — this is the shape #7478 was filed on. - reason += " (GitHub reports it mergeable; declare auto_merge.required_checks for the sweep to treat non-required checks as optional)" - } - return mergeBucketFailing, blockedOrOutstanding(reason), "" - } - } - - // The hold check sits AFTER the red classification on purpose - // (hivecommons/hive#7438): a held PR must never become merge-eligible, - // but a held RED PR is still its author's to repair. When this skip ran - // first, a level-held agent PR with a failing check vanished from - // ci-failing.json, its author never got a fix-before-new block for it, - // and it sat red and held until a human did the agent's repair. - if held { - return mergeBucketSkip, blockedOrOutstanding("held: a hold label keeps it out of the sweep"), "" - } - - // A PR whose CI is still "pending" is nonetheless merge-eligible when - // GitHub itself reports it as mergeable (mergeStateStatus=unstable): - // that state means every REQUIRED check has passed and only - // non-required checks remain outstanding. Those non-required checks — - // a cancelled Mobile Browser Tests, a still-running coverage-report, - // perpetually-pending tide — can never complete on their own, so - // waiting for CIStatus=="success" (all checks done) leaves cleanly - // mergeable PRs frozen out of the sweep indefinitely (observed - // 2026-08-04: three green console PRs stuck for hours). The merge step - // re-enforces branch protection, so trusting the mergeable verdict here - // cannot merge anything GitHub would actually block. - if pr.CIStatus == "pending" && pr.Mergeable != github.MergeableYes { - // Genuinely not ready: a required check is still running (or - // mergeability is unknown/no). Leave it out of both buckets, as - // before — it neither merges nor gets a fix dispatched. - return mergeBucketSkip, blockedOrOutstanding("CI pending"), "" - } - - // The review gate runs BEFORE the GitHub-says-no return below so that a - // PR GitHub calls "blocked" for want of a review carries that reason - // (hivecommons/hive#7515). Both paths file a MergeableNo PR in the skip - // bucket, so the order changes only the verdict's wording. - if g.requireReviewApproval { - if !g.reviewLoaded { - return mergeBucketSkip, blockedOrOutstanding("review approval required, but review-verdicts.json is unavailable"), "" - } - if !g.reviewArtifact.HasAggregateApproval(fullRepo, pr.Number, pr.HeadSHA) { - return mergeBucketSkip, blockedOrOutstanding("awaiting review approval"), "" - } - } - - if pr.Mergeable == github.MergeableNo { - // A conflicting PR cannot merge no matter how green its checks - // are. Listing it as merge-eligible left the eligible count stuck - // at N forever while nothing could actually merge (console - // #23002/#23003, 2026-08-31: the only two build-gate-green PRs - // were DIRTY go.mod dependabot bumps). Conflicts are the - // rebase/needs-human path's job, not the sweep's — keep them out - // of the eligible bucket. No sweep gate explains this one: for - // "blocked" that means a branch-protection rule we do not read yet - // (a review GitHub requires, a required check that never reported); - // notMergeableReason says so rather than the bare enum. - return mergeBucketSkip, github.MergeVerdict{State: github.MergeVerdictBlocked, Reason: notMergeableReason(pr, "")}, "" - } - - // Eligible. The reason names what GitHub still shows outstanding that - // the sweep chooses to ignore, so a green pill beside a red optional - // check does not read as "all green". - reason := "the sweep would merge this now" - switch { - case pr.CIStatus == "failure": - reason += " — only non-required checks are red (" + strings.Join(pr.FailingChecks, ", ") + ")" - case pr.CIStatus == "pending": - reason += " — non-required checks still pending (GitHub: " + pr.MergeableState + ")" - case pr.MergeableState == "unstable": - reason += " — non-required checks outstanding (GitHub: unstable)" - case pr.Mergeable == github.MergeableUnknown: - reason += " — mergeability not yet fetched; the sweep re-checks it at merge time" - } - return mergeBucketEligible, github.MergeVerdict{State: github.MergeVerdictEligible, Reason: reason}, "" -} - -// mergeabilityUnknownReason is the verdict prefix for a PR whose -// mergeability GitHub has not computed yet (or the fetch failed); the -// classifier appends the sweep's own reason after it. -const mergeabilityUnknownReason = "mergeability not yet computed by GitHub — re-checked next tick" - -// notMergeableReason explains, in words an operator can act on, why GitHub -// reports a PR as not mergeable — what to do, not the API enum -// (hivecommons/hive#7515). sweepReason is the gate the sweep itself failed -// the PR on, or "" when every sweep gate passed: -// -// - "blocked" folds every unsatisfied branch-protection rule into one -// word. When the sweep has a reason it is almost always the rule -// ("blocked — CI failing: build"); when GitHub's own facts name a -// different rule (a review decision, a required check that never -// reported) that rule is named too; with neither, say that a rule we -// cannot read is unsatisfied rather than nothing at all. -// - "dirty" and "behind" name the base branch and the fix (rebase / -// update); a sweep reason is appended, since it still stands once the -// branch is fixed. -// - Any other state falls back to naming it. -func notMergeableReason(pr github.PullRequest, sweepReason string) string { - base, from := pr.BaseRef, "the base branch" - if base == "" { - base, from = "the base branch", "it" - } - var msg string - switch pr.MergeableState { - case "blocked": - // The branch-protection rule GitHub is hiding behind the word - // "blocked", when the sweep collected enough to name it - // (hivecommons/hive#7515 step 2). The wording itself lives in - // github.PullRequest.BranchProtectionBlockReason — one place, under - // test — not in this switch and not in the dashboard's JS. - rule, ruleKnown := pr.BranchProtectionBlockReason() - switch { - case sweepReason == "" && ruleKnown: - return "blocked — " + rule - case sweepReason == "": - return "blocked — all sweep gates pass; a branch-protection rule is unsatisfied" - case ruleKnown && !strings.Contains(sweepReason, rule): - // Both are true and neither subsumes the other: the sweep's own - // gate is what it will act on, and GitHub's rule is what the - // operator must also clear. - return "blocked — " + sweepReason + "; GitHub also requires: " + rule - } - return "blocked — " + sweepReason - case "dirty": - msg = "has merge conflicts with " + base + " — needs a rebase" - case "behind": - msg = "behind " + base + " — needs an update from " + from - case "": - msg = "not mergeable on GitHub" - default: - msg = "not mergeable on GitHub (" + pr.MergeableState + ")" - } - if sweepReason != "" { - msg += "; also " + sweepReason - } - return msg -} - -func writeMergeEligible(actionable *github.ActionableResult, hold github.HoldResult, org string, escalatedPRs map[string]bool, enforceIntent bool, intentVerdicts map[string]intent.Verdict, requireReviewApproval bool, requiredChecks map[string]bool, logger *slog.Logger) map[string]github.MergeVerdict { - verdicts := make(map[string]github.MergeVerdict) - holdSet := make(map[string]bool) - for _, h := range hold.Items { - key := fmt.Sprintf("%s/%d", h.Repo, h.Number) - holdSet[key] = true - } - - type eligiblePR struct { - Number int `json:"number"` - Repo string `json:"repo"` - Title string `json:"title"` - Author string `json:"author"` - Labels []string `json:"labels,omitempty"` - // Mergeable is a tri-state string ("yes"/"no"/"unknown"), not a bool. - // A bool here defaulted to false for every PR, because the value was - // read from a list endpoint that never returns it. - Mergeable string `json:"mergeable"` - DCO string `json:"dco"` - // HeadSHA is the governor-observed head commit at the moment eligibility - // was decided. mergeTargetEligible compares the relay's expected SHA - // against this value (M4, CWE-367): a branch that moved after review - // no longer matches and fails closed. - HeadSHA string `json:"head_sha,omitempty"` - } - - type failingPR struct { - Number int `json:"number"` - Repo string `json:"repo"` - Title string `json:"title"` - Author string `json:"author"` - HeadSHA string `json:"head_sha,omitempty"` - // FailingChecks + Excerpt carry the raw CI evidence into the kick - // work list so fix agents see the actual error, not just "red". - FailingChecks []string `json:"failing_checks,omitempty"` - Excerpt string `json:"excerpt,omitempty"` - // Escalated marks PRs past the fix-loop breaker threshold: kick - // builders list them separately and agents must NOT dispatch more - // fix work for them. - Escalated bool `json:"escalated,omitempty"` - // Agent is the hive agent whose relay request opened this PR (from the - // audit trail's agent_pr_created entries). The scheduler's - // fix-before-new section routes each red PR back to its author; empty - // means unattributed (kick builders default it to scanner). - Agent string `json:"agent,omitempty"` - // HeadRef / HeadRepo / FromFork say where the red branch actually - // lives (hivecommons/hive#7386). The hive's App token pushes only to - // the base repository, so a fork PR is comment-only for every agent: - // ReachableAction spells that out ("push" | "comment-only") so no - // kick consumer has to discover it with a failed push — the failure - // mode that burned a scanner session and left a stray branch on the - // base repo under the fork's head-ref name. - HeadRef string `json:"head_ref,omitempty"` - HeadRepo string `json:"head_repo,omitempty"` - FromFork bool `json:"from_fork,omitempty"` - ReachableAction string `json:"reachable_action"` - // Held marks a PR carrying a hold label (the ACMM level gate's, or a - // human's). The hold is a MERGE checkpoint, not a repair checkpoint - // (hivecommons/hive#7438): a held red PR is still its author's to fix, - // so it is listed here with the flag rather than dropped — the owning - // agent's fix-before-new block says "fix CI, do not remove the hold". - Held bool `json:"held,omitempty"` - } - - prAgents := auditPRAgents(org, time.Now().Add(-auditPRAttributionWindow), "") - - var eligible []eligiblePR - var failing []failingPR - var reviewArtifact review.Artifact - reviewLoaded := false - if requireReviewApproval { - var err error - reviewArtifact, err = review.LoadArtifact("") - if err != nil { - logger.Warn("review approval required but review-verdicts.json is unavailable; merge eligibility will fail closed", "error", err) - } else { - reviewLoaded = true - } - } - // Two populations, one classifier (hivecommons/hive#7438). PRs.Items are - // the merge candidates. PRs.Held are PRs the hold gate removed from Items - // — they can never become merge-eligible, but a RED one still has to reach - // its authoring agent, otherwise it deadlocks: it stays red, so it stays - // held, so nothing ever repairs it. - type prCandidate struct { - pr github.PullRequest - held bool - } - candidates := make([]prCandidate, 0, len(actionable.PRs.Items)+len(actionable.PRs.Held)) - for _, pr := range actionable.PRs.Items { - candidates = append(candidates, prCandidate{pr: pr}) - } - for _, pr := range actionable.PRs.Held { - candidates = append(candidates, prCandidate{pr: pr, held: true}) - } - - gates := mergeGates{ - enforceIntent: enforceIntent, - intentVerdicts: intentVerdicts, - requireReviewApproval: requireReviewApproval, - reviewArtifact: reviewArtifact, - reviewLoaded: reviewLoaded, - requiredChecks: requiredChecks, - } - seen := make(map[string]bool, len(candidates)) - for _, cand := range candidates { - pr := cand.pr - key := fmt.Sprintf("%s/%d", pr.Repo, pr.Number) - if seen[key] { - continue - } - seen[key] = true - // The hold can arrive either as membership in PRs.Held or as a row in - // the hold snapshot; both mean the same thing here. - held := cand.held || holdSet[key] - fullRepo := fullRepoName(pr.Repo, org) - - bucket, verdict, intentReason := classifyMergeEligibility(pr, held, fullRepo, gates) - verdicts[github.MergeVerdictKey(pr)] = verdict - if intentReason != "" { - iv := intentVerdicts[fmt.Sprintf("%s/%d", fullRepo, pr.Number)] - logger.Info("excluding PR from merge-eligible due to intent verification", "repo", fullRepo, "number", pr.Number, "tier", iv.Tier, "reason", intentReason) - } - switch bucket { - case mergeBucketSkip: - continue - case mergeBucketFailing: - failing = append(failing, failingPR{ - Number: pr.Number, - Repo: fullRepo, - Title: pr.Title, - Author: pr.Author, - HeadSHA: pr.HeadSHA, - FailingChecks: pr.FailingChecks, - Excerpt: pr.CIFailureExcerpt, - Escalated: escalatedPRs[escalation.Key(fullRepo, pr.Number)], - Agent: prAgents[fmt.Sprintf("%s#%d", fullRepo, pr.Number)], - HeadRef: pr.HeadRef, - HeadRepo: pr.HeadRepo, - FromFork: pr.FromFork, - ReachableAction: github.ReachableAction(pr), - Held: held, - }) - continue - } - - dco := "unknown" - for _, l := range pr.Labels { - switch l { - case "dco-signoff: yes": - dco = "yes" - case "dco-signoff: no": - dco = "no" - } - } - eligible = append(eligible, eligiblePR{ - Number: pr.Number, - Repo: fullRepo, - Title: pr.Title, - Author: pr.Author, - Labels: pr.Labels, - Mergeable: mergeableJSON(pr.Mergeable), - DCO: dco, - HeadSHA: pr.HeadSHA, - }) - } - - _ = os.MkdirAll("/var/run/hive-metrics", 0o755) - - payload := map[string]any{ - "generated_at": time.Now().UTC().Format(time.RFC3339), - "merge_eligible": eligible, - } - data, err := json.Marshal(payload) - if err != nil { - logger.Warn("failed to marshal merge-eligible", "error", err) - return verdicts - } - atomicWrite(mergeEligiblePath, data) - logger.Info("merge-eligible.json updated", "eligible", len(eligible), "ci_failing", len(failing), "total_prs", len(actionable.PRs.Items)) - - failPayload := map[string]any{ - "generated_at": time.Now().UTC().Format(time.RFC3339), - "ci_failing": failing, - } - failData, err := json.Marshal(failPayload) - if err != nil { - logger.Warn("failed to marshal ci-failing", "error", err) - return verdicts - } - atomicWrite(ciFailingPath, failData) - return verdicts -} - func planReviewDispatch(cfg *config.Config, actionable *github.ActionableResult, agentMgr *agent.Manager, logger *slog.Logger) review.DispatchPlan { if cfg == nil || actionable == nil || !cfg.Review.RequireApproval || !cfg.Review.FanOut { return review.DispatchPlan{} @@ -9551,19 +7781,6 @@ func persistReviewDispatchState(plan review.DispatchPlan, delivered []review.Dis } } -// normalizedAutoMergeLabel resolves the configured queue label, falling back -// to the shared default when the value is blank. Client.SetAutoMergeLabel -// ignores blank input (keeping whatever was set before) and -// Client.AutoMergeLabel falls back on read, but the cmd layer normalizes -// eagerly too so a partially-populated config can never propagate an unnamed -// label to a fresh client. -func normalizedAutoMergeLabel(label string) string { - if label = strings.TrimSpace(label); label != "" { - return label - } - return github.AutoMergeQueuedLabel -} - func atomicWrite(path string, data []byte) { tmp := path + ".tmp" if os.WriteFile(tmp, data, 0o644) == nil { diff --git a/src/cmd/hive/merge_eligibility.go b/src/cmd/hive/merge_eligibility.go new file mode 100644 index 0000000000..16f0cf27f8 --- /dev/null +++ b/src/cmd/hive/merge_eligibility.go @@ -0,0 +1,639 @@ +package main + +// Merge eligibility: deciding whether a pull request may be auto-merged -- +// hold gates, required-check state, mergeability classification into buckets, +// the authz binding for the trusted merger, and the merge-eligible report file. + +import ( + "encoding/json" + "fmt" + "log/slog" + "os" + "strings" + "time" + + // automaxprocs sets GOMAXPROCS to match the container's CPU quota (Linux + // CFS) at init. Without it the Go runtime sizes its P count to the whole + // NODE's core count, so on a many-core IKS worker a pod limited to a few + // CPUs spawns far more runnable Ps than its CFS quota can service; when the + // quota is exhausted mid-period EVERY goroutine — including the netpoller + // that answers the :3002 liveness probe and the heartbeat loop — is + // throttled until the next CFS period, which stacks on top of the NFS + // stalls to push probe latency past the kubelet timeout. Matching GOMAXPROCS + // to the quota removes that self-inflicted throttling. + // + // This is called explicitly rather than via the package's blank import + // because that import's init writes a line to the default logger (stderr) + // unconditionally. `hive` re-execs itself as a Git transport shim, and the + // setup path captures a child's stdout and stderr into a single buffer to + // parse (e.g. `symbolic-ref --short origin/HEAD`), so an init-time banner + // is indistinguishable from Git's answer and corrupts the parsed branch + // name. Setting it with a no-op logger keeps the GOMAXPROCS behaviour and + // drops the banner. + + "github.com/hivecommons/hive/pkg/config" + "github.com/hivecommons/hive/pkg/escalation" + "github.com/hivecommons/hive/pkg/github" + "github.com/hivecommons/hive/pkg/intent" + "github.com/hivecommons/hive/pkg/review" +) + +// trustedMergerFunc resolves a GitHub login against the hive's authorized-users +// allowlist and reports whether it holds at least config.RoleMerger — the same +// bar requireMergerOrOwnerRole enforces on the dashboard queue endpoint (audit +// F3). +// +// Fails CLOSED: a nil config or a login absent from the allowlist is NOT +// trusted, so an unclassifiable actor can never merge. cfg is read on every +// call so a config reload that grants or revokes the merger tier takes effect +// without a restart. +func trustedMergerFunc(cfg *config.Config) github.MergerAuthorizer { + return func(login string) bool { + if cfg == nil || strings.TrimSpace(login) == "" { + return false + } + role, ok := cfg.Dashboard.AuthorizedRole(login) + if !ok { + return false + } + return config.RoleAtLeast(role, config.RoleMerger) + } +} + +// mergeEligiblePath is a var (not a const) only so tests can point +// mergeTargetEligible at a temp file; production never reassigns it. +var mergeEligiblePath = "/var/run/hive-metrics/merge-eligible.json" + +// acmmHoldGatedMinLevel / acmmHoldGatedMaxLevel bracket the ACMM levels whose +// merge policy is "hold-gated" — every agent-opened PR gets a "hold" label and +// no agent merges (see src/pkg/config/packs/level-{3,4,5}.yaml). L1/L2 are +// "manual" (agents open no PRs) and L6 is "auto-merge on green CI, no hold +// label", so both fall outside this range. Used by the F6 hold-label decider. +const ( + acmmHoldGatedMinLevel = 3 + acmmHoldGatedMaxLevel = 5 +) + +// shouldHoldAgentPR keeps public outreach claims human-reviewed even at L6, +// where ordinary agent PRs may auto-merge. The general ACMM hold gate remains +// unchanged for all roles at L3-L5. +func shouldHoldAgentPR(agentName string, level int) bool { + if strings.EqualFold(strings.TrimSpace(agentName), "outreach") { + return true + } + return level >= acmmHoldGatedMinLevel && level <= acmmHoldGatedMaxLevel +} + +// mergeableJSONUnknown is the explicit wire value for "mergeability was never +// determined". It is spelled out rather than left as "" so a consumer reading +// merge-eligible.json cannot mistake an unpopulated field for a definitive +// "no" — the failure mode that made every PR read as unmergeable. +const mergeableJSONUnknown = "unknown" + +// mergeTargetEligible reports whether (repo, number) currently appears in the +// governor's merge-eligible.json AT the expected head SHA. It reads the file +// FRESH on every call (never caches) because eligibility is recomputed each +// governor cycle — a stale cache could authorize a PR that has since fallen out +// of the list. On any read/parse error it returns false (FAIL CLOSED): if we +// cannot prove the target is eligible, we must not authorize the merge. +// +// M4 (CWE-367, TOCTOU): the governor records the head SHA it observed when it +// deemed the PR eligible (eligiblePR.HeadSHA). A branch can move between that +// review and the merge relay firing, so matching (repo, number) alone would let +// a moved head merge at a commit the governor never vetted. We therefore also +// require the entry's stored head_sha to equal expectSHA. A mismatch — or a +// stored SHA that is empty (governor could not observe it) — fails closed; the +// relay's SHA pin then fails the merge cleanly if a stale request slips through. +// +// merge-eligible.json stores repos as "owner/repo"; a MergeRequest.Repo may be +// bare ("repo") or fully qualified ("owner/repo"). We match on the bare repo +// name (the segment after the last "/") plus the number, so both request forms +// resolve to the same eligible entry without depending on the org prefix. +func mergeTargetEligible(repo string, number int, expectSHA string) bool { + data, err := os.ReadFile(mergeEligiblePath) + if err != nil { + return false // fail closed: no list ⇒ nothing is eligible + } + var payload struct { + Items []struct { + Number int `json:"number"` + Repo string `json:"repo"` + HeadSHA string `json:"head_sha"` + } `json:"merge_eligible"` + } + if err := json.Unmarshal(data, &payload); err != nil { + return false // fail closed: unparseable list ⇒ deny + } + want := bareRepoName(repo) + wantSHA := strings.TrimSpace(expectSHA) + for _, it := range payload.Items { + if it.Number == number && bareRepoName(it.Repo) == want { + // M4: bind authorization to the governor-observed head. An empty + // stored SHA cannot be proven to match, so it fails closed rather + // than authorizing an unpinned head. + return strings.TrimSpace(it.HeadSHA) != "" && strings.TrimSpace(it.HeadSHA) == wantSHA + } + } + return false +} + +// bareRepoName returns the repo segment after the last "/", so "owner/repo" and +// "repo" compare equal. Used to match a MergeRequest.Repo against the +// "owner/repo" entries in merge-eligible.json regardless of prefix. +func bareRepoName(repo string) string { + if i := strings.LastIndex(repo, "/"); i >= 0 { + return repo[i+1:] + } + return repo +} + +// bindMergeAuthz wraps the manager's agent/UID/CanMerge authorizer with the +// F4 target-binding checks (CWE-863). The inner authz owns the "may this agent +// merge at all" decision; this wrapper owns "is THIS specific target one the +// governor deemed eligible, at a pinned SHA". Both must pass before MergePR is +// reached. Ordering: run the agent/UID/CanMerge check first (cheapest, and it +// gives the clearest denial reason), then the SHA + eligible-list binding. +func bindMergeAuthz(inner func(agent string, fileUID int) error) github.MergeRequestAuthorizer { + return func(agent string, fileUID int, repo string, number int, expectSHA string) error { + if err := inner(agent, fileUID); err != nil { + return err + } + // (a) Require a pinned head SHA. An empty expectSHA means "merge whatever + // HEAD is now", which is the TOCTOU hole: a PR that was eligible when the + // governor last looked could have had a malicious commit pushed since. + // MergePR passes expectSHA as the required head SHA, so a moved head fails + // cleanly — but only if we insist it is set. + if strings.TrimSpace(expectSHA) == "" { + return fmt.Errorf("merge target %s#%d has no expected head SHA — refusing to merge an unpinned head (TOCTOU guard)", repo, number) + } + // (b) Require the target to be in the governor's current merge-eligible + // list AT the expected head SHA. This binds authorization to a PR the + // hive actually deemed landable this cycle, at the exact commit it + // reviewed, so an injected agent cannot request landing an arbitrary + // reachable PR (e.g. its own) whose required checks happen to pass, nor + // land an eligible PR at a head that moved after review (M4, CWE-367). + // Read fresh + fail closed (see mergeTargetEligible). + if !mergeTargetEligible(repo, number, expectSHA) { + return fmt.Errorf("merge target %s#%d is not in the current merge-eligible list at head %s — only governor-approved PRs may be landed via the merge relay, and only at the reviewed head SHA", repo, number, expectSHA) + } + return nil + } +} + +// mergeableJSON renders a tri-state mergeability verdict for the +// merge-eligible.json marker, mapping the unknown zero value to an explicit +// "unknown" rather than an empty string. +func mergeableJSON(m github.Mergeable) string { + if m == github.MergeableUnknown { + return mergeableJSONUnknown + } + return string(m) +} + +// anyRequiredCheckFailing reports whether any of a PR's failing check names +// is in the operator-declared required set. +func anyRequiredCheckFailing(failing []string, required map[string]bool) bool { + for _, name := range failing { + if required[name] { + return true + } + } + return false +} + +// mergeBucket is where the merge-eligible classifier files a PR: the +// merge-eligible.json list, the ci-failing.json list, or neither. +type mergeBucket int + +const ( + mergeBucketSkip mergeBucket = iota + mergeBucketFailing + mergeBucketEligible +) + +// mergeGates bundles the per-tick inputs the classifier applies beyond the +// PR itself: the intent and review artifacts and the operator's +// required-check set. +type mergeGates struct { + enforceIntent bool + intentVerdicts map[string]intent.Verdict + requireReviewApproval bool + reviewArtifact review.Artifact + reviewLoaded bool + requiredChecks map[string]bool +} + +// classifyMergeEligibility is THE merge-eligibility rule: the one place that +// decides whether a PR goes to merge-eligible.json (the sweep would merge it +// now), ci-failing.json (its author has CI to fix), or neither. It returns +// the bucket and, for the dashboard, the same decision as a MergeVerdict with +// the reason spelled out (hivecommons/hive#7478): the pill is painted from +// this verdict, so green on the card means exactly what the sweep means by +// eligible, and never the looser "GitHub says mergeable". +// +// intentReason is non-empty only when the intent gate excluded the PR; the +// caller logs it (the log line carries the verdict tier, which this function +// does not need). +func classifyMergeEligibility(pr github.PullRequest, held bool, fullRepo string, g mergeGates) (bucket mergeBucket, verdict github.MergeVerdict, intentReason string) { + // blockedOrOutstanding is the verdict for a PR the sweep will not take + // for a reason of its own: the state still depends on what GitHub says, + // because a conflicting PR is blocked whatever else is outstanding, and + // one whose mergeability was never fetched is unknown, not amber. The + // sweep's reason is kept in every state (hivecommons/hive#7515): on a + // protected branch a red required check or a missing review is exactly + // what makes GitHub say "blocked", so dropping it for the bare enum sent + // the operator to GitHub to learn what this function already knew. + blockedOrOutstanding := func(reason string) github.MergeVerdict { + switch pr.Mergeable { + case github.MergeableNo: + return github.MergeVerdict{State: github.MergeVerdictBlocked, Reason: notMergeableReason(pr, reason)} + case github.MergeableUnknown: + return github.MergeVerdict{State: github.MergeVerdictUnknown, Reason: mergeabilityUnknownReason + "; " + reason} + } + return github.MergeVerdict{State: github.MergeVerdictOutstanding, Reason: reason} + } + + if pr.Draft { + return mergeBucketSkip, github.MergeVerdict{State: github.MergeVerdictBlocked, Reason: "draft — mark ready for review to enter the sweep"}, "" + } + if g.enforceIntent { + if v, ok := g.intentVerdicts[fmt.Sprintf("%s/%d", fullRepo, pr.Number)]; ok && v.AgentPR && !v.MergeAllowed() { + reason := v.Reason + if v.Authorized && v.Alignment != nil && v.Alignment.Misaligned() { + reason = intent.ReasonAlignmentMisaligned + ": " + v.Alignment.Rationale + } + return mergeBucketSkip, blockedOrOutstanding("intent verification: " + reason), reason + } + } + + if pr.CIStatus == "failure" { + // A PR red ONLY on non-required checks (perma-red Playwright + // shards, coverage) that GitHub itself reports mergeable is NOT a + // failing PR — it is merge-eligible, mirroring the + // pending-but-mergeable rule below. Without this, every dependabot + // PR on a repo with permanently-red optional checks classified as + // "failure", landed in ci-failing.json where no sweep or agent + // would ever merge it, and accumulated indefinitely (observed on + // kubestellar/console 2026-08-28: 16 dependabot PRs, oldest 11 + // days). Gated on an operator-declared required-check set: with no + // set configured we cannot distinguish required from optional and + // keep the old fail-closed behavior. The merge step re-enforces + // branch protection, so this cannot merge anything GitHub blocks. + onlyOptionalRed := len(g.requiredChecks) > 0 && + !anyRequiredCheckFailing(pr.FailingChecks, g.requiredChecks) && + pr.Mergeable == github.MergeableYes + if !onlyOptionalRed { + reason := "CI failing" + if len(pr.FailingChecks) > 0 { + reason += ": " + strings.Join(pr.FailingChecks, ", ") + } + if len(g.requiredChecks) == 0 && pr.Mergeable == github.MergeableYes { + // GitHub calls it mergeable (unstable): nothing REQUIRED is + // red. The sweep still refuses it because, with no + // required-check set declared, it cannot tell optional from + // required. Say so — this is the shape #7478 was filed on. + reason += " (GitHub reports it mergeable; declare auto_merge.required_checks for the sweep to treat non-required checks as optional)" + } + return mergeBucketFailing, blockedOrOutstanding(reason), "" + } + } + + // The hold check sits AFTER the red classification on purpose + // (hivecommons/hive#7438): a held PR must never become merge-eligible, + // but a held RED PR is still its author's to repair. When this skip ran + // first, a level-held agent PR with a failing check vanished from + // ci-failing.json, its author never got a fix-before-new block for it, + // and it sat red and held until a human did the agent's repair. + if held { + return mergeBucketSkip, blockedOrOutstanding("held: a hold label keeps it out of the sweep"), "" + } + + // A PR whose CI is still "pending" is nonetheless merge-eligible when + // GitHub itself reports it as mergeable (mergeStateStatus=unstable): + // that state means every REQUIRED check has passed and only + // non-required checks remain outstanding. Those non-required checks — + // a cancelled Mobile Browser Tests, a still-running coverage-report, + // perpetually-pending tide — can never complete on their own, so + // waiting for CIStatus=="success" (all checks done) leaves cleanly + // mergeable PRs frozen out of the sweep indefinitely (observed + // 2026-08-04: three green console PRs stuck for hours). The merge step + // re-enforces branch protection, so trusting the mergeable verdict here + // cannot merge anything GitHub would actually block. + if pr.CIStatus == "pending" && pr.Mergeable != github.MergeableYes { + // Genuinely not ready: a required check is still running (or + // mergeability is unknown/no). Leave it out of both buckets, as + // before — it neither merges nor gets a fix dispatched. + return mergeBucketSkip, blockedOrOutstanding("CI pending"), "" + } + + // The review gate runs BEFORE the GitHub-says-no return below so that a + // PR GitHub calls "blocked" for want of a review carries that reason + // (hivecommons/hive#7515). Both paths file a MergeableNo PR in the skip + // bucket, so the order changes only the verdict's wording. + if g.requireReviewApproval { + if !g.reviewLoaded { + return mergeBucketSkip, blockedOrOutstanding("review approval required, but review-verdicts.json is unavailable"), "" + } + if !g.reviewArtifact.HasAggregateApproval(fullRepo, pr.Number, pr.HeadSHA) { + return mergeBucketSkip, blockedOrOutstanding("awaiting review approval"), "" + } + } + + if pr.Mergeable == github.MergeableNo { + // A conflicting PR cannot merge no matter how green its checks + // are. Listing it as merge-eligible left the eligible count stuck + // at N forever while nothing could actually merge (console + // #23002/#23003, 2026-08-31: the only two build-gate-green PRs + // were DIRTY go.mod dependabot bumps). Conflicts are the + // rebase/needs-human path's job, not the sweep's — keep them out + // of the eligible bucket. No sweep gate explains this one: for + // "blocked" that means a branch-protection rule we do not read yet + // (a review GitHub requires, a required check that never reported); + // notMergeableReason says so rather than the bare enum. + return mergeBucketSkip, github.MergeVerdict{State: github.MergeVerdictBlocked, Reason: notMergeableReason(pr, "")}, "" + } + + // Eligible. The reason names what GitHub still shows outstanding that + // the sweep chooses to ignore, so a green pill beside a red optional + // check does not read as "all green". + reason := "the sweep would merge this now" + switch { + case pr.CIStatus == "failure": + reason += " — only non-required checks are red (" + strings.Join(pr.FailingChecks, ", ") + ")" + case pr.CIStatus == "pending": + reason += " — non-required checks still pending (GitHub: " + pr.MergeableState + ")" + case pr.MergeableState == "unstable": + reason += " — non-required checks outstanding (GitHub: unstable)" + case pr.Mergeable == github.MergeableUnknown: + reason += " — mergeability not yet fetched; the sweep re-checks it at merge time" + } + return mergeBucketEligible, github.MergeVerdict{State: github.MergeVerdictEligible, Reason: reason}, "" +} + +// mergeabilityUnknownReason is the verdict prefix for a PR whose +// mergeability GitHub has not computed yet (or the fetch failed); the +// classifier appends the sweep's own reason after it. +const mergeabilityUnknownReason = "mergeability not yet computed by GitHub — re-checked next tick" + +// notMergeableReason explains, in words an operator can act on, why GitHub +// reports a PR as not mergeable — what to do, not the API enum +// (hivecommons/hive#7515). sweepReason is the gate the sweep itself failed +// the PR on, or "" when every sweep gate passed: +// +// - "blocked" folds every unsatisfied branch-protection rule into one +// word. When the sweep has a reason it is almost always the rule +// ("blocked — CI failing: build"); when GitHub's own facts name a +// different rule (a review decision, a required check that never +// reported) that rule is named too; with neither, say that a rule we +// cannot read is unsatisfied rather than nothing at all. +// - "dirty" and "behind" name the base branch and the fix (rebase / +// update); a sweep reason is appended, since it still stands once the +// branch is fixed. +// - Any other state falls back to naming it. +func notMergeableReason(pr github.PullRequest, sweepReason string) string { + base, from := pr.BaseRef, "the base branch" + if base == "" { + base, from = "the base branch", "it" + } + var msg string + switch pr.MergeableState { + case "blocked": + // The branch-protection rule GitHub is hiding behind the word + // "blocked", when the sweep collected enough to name it + // (hivecommons/hive#7515 step 2). The wording itself lives in + // github.PullRequest.BranchProtectionBlockReason — one place, under + // test — not in this switch and not in the dashboard's JS. + rule, ruleKnown := pr.BranchProtectionBlockReason() + switch { + case sweepReason == "" && ruleKnown: + return "blocked — " + rule + case sweepReason == "": + return "blocked — all sweep gates pass; a branch-protection rule is unsatisfied" + case ruleKnown && !strings.Contains(sweepReason, rule): + // Both are true and neither subsumes the other: the sweep's own + // gate is what it will act on, and GitHub's rule is what the + // operator must also clear. + return "blocked — " + sweepReason + "; GitHub also requires: " + rule + } + return "blocked — " + sweepReason + case "dirty": + msg = "has merge conflicts with " + base + " — needs a rebase" + case "behind": + msg = "behind " + base + " — needs an update from " + from + case "": + msg = "not mergeable on GitHub" + default: + msg = "not mergeable on GitHub (" + pr.MergeableState + ")" + } + if sweepReason != "" { + msg += "; also " + sweepReason + } + return msg +} + +func writeMergeEligible(actionable *github.ActionableResult, hold github.HoldResult, org string, escalatedPRs map[string]bool, enforceIntent bool, intentVerdicts map[string]intent.Verdict, requireReviewApproval bool, requiredChecks map[string]bool, logger *slog.Logger) map[string]github.MergeVerdict { + verdicts := make(map[string]github.MergeVerdict) + holdSet := make(map[string]bool) + for _, h := range hold.Items { + key := fmt.Sprintf("%s/%d", h.Repo, h.Number) + holdSet[key] = true + } + + type eligiblePR struct { + Number int `json:"number"` + Repo string `json:"repo"` + Title string `json:"title"` + Author string `json:"author"` + Labels []string `json:"labels,omitempty"` + // Mergeable is a tri-state string ("yes"/"no"/"unknown"), not a bool. + // A bool here defaulted to false for every PR, because the value was + // read from a list endpoint that never returns it. + Mergeable string `json:"mergeable"` + DCO string `json:"dco"` + // HeadSHA is the governor-observed head commit at the moment eligibility + // was decided. mergeTargetEligible compares the relay's expected SHA + // against this value (M4, CWE-367): a branch that moved after review + // no longer matches and fails closed. + HeadSHA string `json:"head_sha,omitempty"` + } + + type failingPR struct { + Number int `json:"number"` + Repo string `json:"repo"` + Title string `json:"title"` + Author string `json:"author"` + HeadSHA string `json:"head_sha,omitempty"` + // FailingChecks + Excerpt carry the raw CI evidence into the kick + // work list so fix agents see the actual error, not just "red". + FailingChecks []string `json:"failing_checks,omitempty"` + Excerpt string `json:"excerpt,omitempty"` + // Escalated marks PRs past the fix-loop breaker threshold: kick + // builders list them separately and agents must NOT dispatch more + // fix work for them. + Escalated bool `json:"escalated,omitempty"` + // Agent is the hive agent whose relay request opened this PR (from the + // audit trail's agent_pr_created entries). The scheduler's + // fix-before-new section routes each red PR back to its author; empty + // means unattributed (kick builders default it to scanner). + Agent string `json:"agent,omitempty"` + // HeadRef / HeadRepo / FromFork say where the red branch actually + // lives (hivecommons/hive#7386). The hive's App token pushes only to + // the base repository, so a fork PR is comment-only for every agent: + // ReachableAction spells that out ("push" | "comment-only") so no + // kick consumer has to discover it with a failed push — the failure + // mode that burned a scanner session and left a stray branch on the + // base repo under the fork's head-ref name. + HeadRef string `json:"head_ref,omitempty"` + HeadRepo string `json:"head_repo,omitempty"` + FromFork bool `json:"from_fork,omitempty"` + ReachableAction string `json:"reachable_action"` + // Held marks a PR carrying a hold label (the ACMM level gate's, or a + // human's). The hold is a MERGE checkpoint, not a repair checkpoint + // (hivecommons/hive#7438): a held red PR is still its author's to fix, + // so it is listed here with the flag rather than dropped — the owning + // agent's fix-before-new block says "fix CI, do not remove the hold". + Held bool `json:"held,omitempty"` + } + + prAgents := auditPRAgents(org, time.Now().Add(-auditPRAttributionWindow), "") + + var eligible []eligiblePR + var failing []failingPR + var reviewArtifact review.Artifact + reviewLoaded := false + if requireReviewApproval { + var err error + reviewArtifact, err = review.LoadArtifact("") + if err != nil { + logger.Warn("review approval required but review-verdicts.json is unavailable; merge eligibility will fail closed", "error", err) + } else { + reviewLoaded = true + } + } + // Two populations, one classifier (hivecommons/hive#7438). PRs.Items are + // the merge candidates. PRs.Held are PRs the hold gate removed from Items + // — they can never become merge-eligible, but a RED one still has to reach + // its authoring agent, otherwise it deadlocks: it stays red, so it stays + // held, so nothing ever repairs it. + type prCandidate struct { + pr github.PullRequest + held bool + } + candidates := make([]prCandidate, 0, len(actionable.PRs.Items)+len(actionable.PRs.Held)) + for _, pr := range actionable.PRs.Items { + candidates = append(candidates, prCandidate{pr: pr}) + } + for _, pr := range actionable.PRs.Held { + candidates = append(candidates, prCandidate{pr: pr, held: true}) + } + + gates := mergeGates{ + enforceIntent: enforceIntent, + intentVerdicts: intentVerdicts, + requireReviewApproval: requireReviewApproval, + reviewArtifact: reviewArtifact, + reviewLoaded: reviewLoaded, + requiredChecks: requiredChecks, + } + seen := make(map[string]bool, len(candidates)) + for _, cand := range candidates { + pr := cand.pr + key := fmt.Sprintf("%s/%d", pr.Repo, pr.Number) + if seen[key] { + continue + } + seen[key] = true + // The hold can arrive either as membership in PRs.Held or as a row in + // the hold snapshot; both mean the same thing here. + held := cand.held || holdSet[key] + fullRepo := fullRepoName(pr.Repo, org) + + bucket, verdict, intentReason := classifyMergeEligibility(pr, held, fullRepo, gates) + verdicts[github.MergeVerdictKey(pr)] = verdict + if intentReason != "" { + iv := intentVerdicts[fmt.Sprintf("%s/%d", fullRepo, pr.Number)] + logger.Info("excluding PR from merge-eligible due to intent verification", "repo", fullRepo, "number", pr.Number, "tier", iv.Tier, "reason", intentReason) + } + switch bucket { + case mergeBucketSkip: + continue + case mergeBucketFailing: + failing = append(failing, failingPR{ + Number: pr.Number, + Repo: fullRepo, + Title: pr.Title, + Author: pr.Author, + HeadSHA: pr.HeadSHA, + FailingChecks: pr.FailingChecks, + Excerpt: pr.CIFailureExcerpt, + Escalated: escalatedPRs[escalation.Key(fullRepo, pr.Number)], + Agent: prAgents[fmt.Sprintf("%s#%d", fullRepo, pr.Number)], + HeadRef: pr.HeadRef, + HeadRepo: pr.HeadRepo, + FromFork: pr.FromFork, + ReachableAction: github.ReachableAction(pr), + Held: held, + }) + continue + } + + dco := "unknown" + for _, l := range pr.Labels { + switch l { + case "dco-signoff: yes": + dco = "yes" + case "dco-signoff: no": + dco = "no" + } + } + eligible = append(eligible, eligiblePR{ + Number: pr.Number, + Repo: fullRepo, + Title: pr.Title, + Author: pr.Author, + Labels: pr.Labels, + Mergeable: mergeableJSON(pr.Mergeable), + DCO: dco, + HeadSHA: pr.HeadSHA, + }) + } + + _ = os.MkdirAll("/var/run/hive-metrics", 0o755) + + payload := map[string]any{ + "generated_at": time.Now().UTC().Format(time.RFC3339), + "merge_eligible": eligible, + } + data, err := json.Marshal(payload) + if err != nil { + logger.Warn("failed to marshal merge-eligible", "error", err) + return verdicts + } + atomicWrite(mergeEligiblePath, data) + logger.Info("merge-eligible.json updated", "eligible", len(eligible), "ci_failing", len(failing), "total_prs", len(actionable.PRs.Items)) + + failPayload := map[string]any{ + "generated_at": time.Now().UTC().Format(time.RFC3339), + "ci_failing": failing, + } + failData, err := json.Marshal(failPayload) + if err != nil { + logger.Warn("failed to marshal ci-failing", "error", err) + return verdicts + } + atomicWrite(ciFailingPath, failData) + return verdicts +} + +// normalizedAutoMergeLabel resolves the configured queue label, falling back +// to the shared default when the value is blank. Client.SetAutoMergeLabel +// ignores blank input (keeping whatever was set before) and +// Client.AutoMergeLabel falls back on read, but the cmd layer normalizes +// eagerly too so a partially-populated config can never propagate an unnamed +// label to a fresh client. +func normalizedAutoMergeLabel(label string) string { + if label = strings.TrimSpace(label); label != "" { + return label + } + return github.AutoMergeQueuedLabel +} diff --git a/src/cmd/hive/selfupgrade.go b/src/cmd/hive/selfupgrade.go new file mode 100644 index 0000000000..8d7eee4182 --- /dev/null +++ b/src/cmd/hive/selfupgrade.go @@ -0,0 +1,211 @@ +package main + +// Self-upgrade bookkeeping: the on-disk upgrade marker and last-outcome files +// that let a restarted hive tell a successful upgrade from a crash loop, plus +// the retry/backoff budget and the boot-time reconciliation of the two. + +import ( + "encoding/json" + "fmt" + "log/slog" + "os" + "strings" + "time" + // automaxprocs sets GOMAXPROCS to match the container's CPU quota (Linux + // CFS) at init. Without it the Go runtime sizes its P count to the whole + // NODE's core count, so on a many-core IKS worker a pod limited to a few + // CPUs spawns far more runnable Ps than its CFS quota can service; when the + // quota is exhausted mid-period EVERY goroutine — including the netpoller + // that answers the :3002 liveness probe and the heartbeat loop — is + // throttled until the next CFS period, which stacks on top of the NFS + // stalls to push probe latency past the kubelet timeout. Matching GOMAXPROCS + // to the quota removes that self-inflicted throttling. + // + // This is called explicitly rather than via the package's blank import + // because that import's init writes a line to the default logger (stderr) + // unconditionally. `hive` re-execs itself as a Git transport shim, and the + // setup path captures a child's stdout and stderr into a single buffer to + // parse (e.g. `symbolic-ref --short origin/HEAD`), so an init-time banner + // is indistinguishable from Git's answer and corrupts the parsed branch + // name. Setting it with a no-op logger keeps the GOMAXPROCS behaviour and + // drops the banner. +) + +// loadOrGenerateHiveID reads the Hive ID from disk, or generates and persists a new one. +const ( + // selfUpgradeMaxAttempts bounds how many times a spoke retries an upgrade + // that keeps leaving the image unchanged. Bounded rather than unlimited so a + // genuinely broken hive (e.g. missing RBAC) stops thrashing its pod, and + // bounded rather than "never again" so a transient failure still converges. + selfUpgradeMaxAttempts = 5 + // selfUpgradeBaseBackoff is the delay before retry #2; it doubles per + // attempt up to selfUpgradeMaxBackoff. + selfUpgradeBaseBackoff = 2 * time.Minute + // selfUpgradeMaxBackoff caps the exponential backoff between retries. + selfUpgradeMaxBackoff = 30 * time.Minute + // selfUpgradeFailureExitCode marks a process exit caused by a FAILED + // self-upgrade. Distinct from 0 so the failure is visible in the container's + // termination state instead of looking like a clean shutdown. + selfUpgradeFailureExitCode = 17 +) + +// upgradeMarker is the on-PVC record at /data/upgrade-requested. It survives +// pod restarts (that is the whole point: the process exits as part of an +// upgrade), so it is the only place attempt bookkeeping can live. +type upgradeMarker struct { + TargetSHA string `json:"target_sha"` + CurrentSHA string `json:"current_sha"` + RequestedAt time.Time `json:"requested_at"` + Attempts int `json:"attempts"` + LastError string `json:"last_error,omitempty"` +} + +// parseUpgradeMarker decodes a marker, tolerating the legacy format that had no +// attempts/last_error fields. A legacy marker counts as one prior attempt so an +// already-wedged hive gets retries under the new budget instead of being +// treated as fresh. +func parseUpgradeMarker(data []byte) upgradeMarker { + var m upgradeMarker + if err := json.Unmarshal(data, &m); err != nil { + return upgradeMarker{} + } + if m.Attempts < 1 { + m.Attempts = 1 + } + return m +} + +// sameUpgradeTarget reports whether two target SHAs refer to the same commit, +// tolerating short/full SHA length mismatch the way the hub's sameCommit does. +// A DIFFERENT target must reset the attempt budget, so this comparison is what +// keeps the latch from outliving the upgrade it was created for. +func sameUpgradeTarget(a, b string) bool { + if a == "" || b == "" { + return false + } + n := len(a) + if len(b) < n { + n = len(b) + } + return strings.EqualFold(a[:n], b[:n]) +} + +func writeUpgradeMarker(path string, m upgradeMarker, logger *slog.Logger) { + data, err := json.Marshal(m) + if err != nil { + logger.Warn("failed to encode upgrade marker", "error", err) + return + } + if err := os.WriteFile(path, data, 0o644); err != nil { + logger.Warn("failed to write upgrade marker", "path", path, "error", err) + } +} + +// recordUpgradeError annotates the existing marker with the cause of the failed +// attempt so the NEXT boot can log why the previous one did not land — without +// it the reason dies with the process and the failure is invisible. +// +// upgradeMarkerPath and lastUpgradeOutcomePath are the two on-PVC records the +// spoke keeps for auto-upgrade visibility (#7092). The marker at +// upgradeMarkerPath is present ONLY while an instructed upgrade has not landed +// (in flight or terminally failed) and is cleared the moment the new image +// boots — so it can NEVER represent a success. lastUpgradeOutcomePath is the +// durable companion that records the last upgrade that actually LANDED, so the +// dashboard can tell "attempted and succeeded" apart from "never attempted" +// instead of letting a blank panel masquerade as success. +const ( + upgradeMarkerPath = "/data/upgrade-requested" + lastUpgradeOutcomePath = "/data/last-upgrade-outcome" +) + +// upgradeOutcome is the durable "last upgrade LANDED" record. Written on the +// boot that completes an upgrade (reconcileUpgradeOutcomeAtBoot), it survives — +// unlike upgradeMarker, which is removed the moment the target image boots. +type upgradeOutcome struct { + TargetSHA string `json:"target_sha"` + CurrentSHA string `json:"current_sha"` + RequestedAt time.Time `json:"requested_at"` + CompletedAt time.Time `json:"completed_at"` +} + +func writeUpgradeOutcome(path string, o upgradeOutcome, logger *slog.Logger) { + data, err := json.Marshal(o) + if err != nil { + logger.Warn("failed to encode upgrade outcome", "error", err) + return + } + if err := os.WriteFile(path, data, 0o644); err != nil { + logger.Warn("failed to write upgrade outcome", "path", path, "error", err) + } +} + +// reconcileUpgradeOutcomeAtBoot records a SUCCESSFUL self-upgrade. An in-flight +// marker whose target equals the now-running commit means the instructed +// upgrade LANDED: the pod booted on the target image. That success would +// otherwise vanish — the next upgrade instruction silently discards the stale +// marker, so a hive that updated cleanly looks identical to one that never +// tried. This persists the success durably and clears the in-flight marker so +// it stops reading as "not landed". A marker whose target does NOT match the +// running commit is still in flight or failed and is left untouched for that +// surface. Called once at startup, before the heartbeat loop and dashboard come +// up, so the dashboard always sees the reconciled state. +func reconcileUpgradeOutcomeAtBoot(markerPath, outcomePath, runningSHA string, logger *slog.Logger) { + data, err := os.ReadFile(markerPath) + if err != nil { + return + } + m := parseUpgradeMarker(data) + if m.TargetSHA == "" || runningSHA == "" || !sameUpgradeTarget(m.TargetSHA, runningSHA) { + return + } + writeUpgradeOutcome(outcomePath, upgradeOutcome{ + TargetSHA: m.TargetSHA, + CurrentSHA: m.CurrentSHA, + RequestedAt: m.RequestedAt, + CompletedAt: time.Now().UTC(), + }, logger) + if err := os.Remove(markerPath); err != nil && !os.IsNotExist(err) { + logger.Warn("failed to clear landed upgrade marker", "path", markerPath, "error", err) + } + logger.Info("self-upgrade landed: recorded successful upgrade outcome", + "target", m.TargetSHA, "current", runningSHA) +} + +// upgradeFailureSummary renders what the hub shows an operator. An empty +// LastError must never render as a dangling "attempts: " - a colon promising a +// reason and delivering none is worse than saying the reason was not captured, +// because it reads as truncation and sends the reader looking for the rest. +func upgradeFailureSummary(attempts int, lastError string) string { + if strings.TrimSpace(lastError) == "" { + return fmt.Sprintf("self-upgrade failed after %d attempts (no error recorded; the image never changed - check that the deployment tracks a tag carrying the target SHA)", attempts) + } + return fmt.Sprintf("self-upgrade failed after %d attempts: %s", attempts, lastError) +} + +func recordUpgradeError(path string, upgradeErr error, logger *slog.Logger) { + if upgradeErr == nil { + return + } + // A marker that cannot be read is not a reason to drop the cause. The + // earlier version returned on ANY read error, which left LastError empty + // and produced the bare "self-upgrade failed after 5 attempts: " the hub + // relays to the dashboard - an alert naming a failure and nothing about + // it. Losing the attempt count is survivable; losing the reason is what + // makes the failure undiagnosable, so rebuild the marker around the error + // instead. An ABSENT marker is different: no attempt is in flight, and + // creating one here would later be mistaken for a real attempt, so the + // no-op stands for that case only. + var m upgradeMarker + data, err := os.ReadFile(path) + switch { + case os.IsNotExist(err): + return + case err != nil: + logger.Warn("upgrade marker unreadable; recording the error against a fresh marker", + "path", path, "error", err) + default: + m = parseUpgradeMarker(data) + } + m.LastError = upgradeErr.Error() + writeUpgradeMarker(path, m, logger) +} diff --git a/src/docs/design/agent-turn-model.md b/src/docs/design/agent-turn-model.md index b50946e32d..1ceac8c7f2 100644 --- a/src/docs/design/agent-turn-model.md +++ b/src/docs/design/agent-turn-model.md @@ -128,11 +128,11 @@ Kick *timing* lives in `pkg/governor`; kick *text* is built in `pkg/scheduler`. - `Governor.Evaluate` (`src/pkg/governor/governor.go:323`) calls it at `src/pkg/governor/governor.go:420` and returns the due list. - The driving loop is a single ticker in `main`: - `time.NewTicker(… EvalIntervalS …)` at `src/cmd/hive/main.go:5332`, loop at - `src/cmd/hive/main.go:5388`, evaluation at `src/cmd/hive/main.go:6156`, - message assembly via `sched.BuildKickMessages` at `src/cmd/hive/main.go:6277` + `time.NewTicker(… EvalIntervalS …)` at `src/cmd/hive/main.go:4985`, loop at + `src/cmd/hive/main.go:5041`, evaluation at `src/cmd/hive/main.go:5802`, + message assembly via `sched.BuildKickMessages` at `src/cmd/hive/main.go:5923` (`src/pkg/scheduler/scheduler.go:658`), and delivery via - `agentMgr.SendKick` at `src/cmd/hive/main.go:6367`. + `agentMgr.SendKick` at `src/cmd/hive/main.go:6013`. This matters for the RFC: the scheduler is already **stateless with respect to turns**. It does not hold a continuation, does not await turn *N* before @@ -150,11 +150,11 @@ CLI subprocess. | State | Where | Citation | |---|---|---| | Pause flag (one bool per agent) | `/data/hive.yaml` via `AgentConfig.Paused` | `src/pkg/config/config.go:921`; writer `SetAgentPausedAndSave` `src/pkg/config/config.go:5736` | -| Pause provenance (`PausedAt`, `PausedReason`, `PausedTrigger`, `PausedBy`), CLI/model pins, model/backend overrides, restart count, `LastKick`, truncated kick history | `/data/hive-state.json` via `snapshot.AgentState` | `src/pkg/snapshot/state.go:78-101`; path `src/cmd/hive/main.go:2162` | +| Pause provenance (`PausedAt`, `PausedReason`, `PausedTrigger`, `PausedBy`), CLI/model pins, model/backend overrides, restart count, `LastKick`, truncated kick history | `/data/hive-state.json` via `snapshot.AgentState` | `src/pkg/snapshot/state.go:78-101`; path `src/cmd/hive/main.go:1815` | | Watchdog failure count, crash-loop latch, backoff deadline, healthy-since, conditions | same file, `snapshot.PersistedState.Watchdog` | `src/pkg/snapshot/state.go:41`; `watchdog.PersistedAgent` `src/pkg/watchdog/reconciler.go:205` | | Fleet-breaker engagement + held set | same file, `BreakerState` | `src/pkg/snapshot/state.go:49` | | Governor budget/spend/eval history, cadence overrides, ACMM level | same file | `src/pkg/snapshot/state.go:16-42` | -| **Full text of every delivered prompt** | `/data/prompt-history.jsonl` (lumberjack-rotated JSONL) | `src/pkg/dashboard/prompt_history.go:44`; writer `Server.RecordPrompt` `src/pkg/dashboard/prompt_history.go:371`, wired at `src/cmd/hive/main.go:3625` | +| **Full text of every delivered prompt** | `/data/prompt-history.jsonl` (lumberjack-rotated JSONL) | `src/pkg/dashboard/prompt_history.go:44`; writer `Server.RecordPrompt` `src/pkg/dashboard/prompt_history.go:371`, wired at `src/cmd/hive/main.go:3278` | | **Rendered terminal scrollback, per kick** | `/data/logs/kicks//-.log` | `src/pkg/agent/kick_logs.go:43` (`defaultKickLogDir`); writer `archiveKickLogLocked` `src/pkg/agent/kick_logs.go:180` | | Token-usage summary | `/data/metrics/token-summary.json` | `src/pkg/tokens/collector.go:194`, `:121` | | Structured audit trail | `/data/audit.jsonl`, reloaded into a ring at boot | `src/pkg/dashboard/audit.go:22`, `loadFromDisk` `:88` | @@ -235,7 +235,7 @@ Three things about this are worth stating precisely: (`src/pkg/agent/manager.go:1909-1911`) — a surviving session is reused, not recreated. Boot reaches both: `main` unconditionally calls `agentMgr.Start(ctx, name)` for every enabled agent - (`src/cmd/hive/main.go:4020`) and the reuse-vs-relaunch decision is taken + (`src/cmd/hive/main.go:3673`) and the reuse-vs-relaunch decision is taken inside. There is no `Adopt`, `Reattach`, or `RecoverAgents` function; searching for one finds only `RestoreBreaker` (`src/pkg/agent/manager.go:6367`), which restores control metadata and is diff --git a/src/docs/design/copilot-cost-capture.md b/src/docs/design/copilot-cost-capture.md index fc275dd0db..4cc36d4465 100644 --- a/src/docs/design/copilot-cost-capture.md +++ b/src/docs/design/copilot-cost-capture.md @@ -66,7 +66,7 @@ moment. That is the per-request grain phase 4 wants, and it already exists. `liveCaptureSinceMs` comes from exactly one production writer: `tokenCollector.SetCopilotLiveCapture(time.Now().UnixMilli())` at -`src/cmd/hive/main.go:3734`, called immediately after `SetTokenSink`. The +`src/cmd/hive/main.go:3387`, called immediately after `SetTokenSink`. The collector stores it (`src/pkg/tokens/collector.go:273`) and passes it to `ScanCopilotSessions` (`collector.go:212`), which zeroes shutdown tokens for sessions whose `LastActive` is at or after that moment diff --git a/src/docs/design/github-mention-triggers.md b/src/docs/design/github-mention-triggers.md index a411c46662..3bcad64672 100644 --- a/src/docs/design/github-mention-triggers.md +++ b/src/docs/design/github-mention-triggers.md @@ -197,7 +197,7 @@ mechanism that already exists; none is new policy. the hive already trusts, using the dashboard's own role list (`DashboardConfig.AuthorizedRole`, `src/pkg/config/config.go:4182`) at `read-write` or above by default — the same lookup `trustedMergerFunc` - uses for the merge queue (`src/cmd/hive/main.go:8076`). An explicit + uses for the merge queue (`src/cmd/hive/merge_eligibility.go:50`). An explicit `github.mentions.summoners` list widens it. No configuration means the feature is **off**, never "any commenter"; that is the rule the Discord design already set for its channel list. @@ -337,5 +337,5 @@ From the issue, restated as boundaries this design must not cross: - `src/pkg/hub/webhook.go:53` — the fail-closed GitHub webhook verifier. - `src/pkg/ioscan/enforce.go:24`, `src/pkg/scheduler/ioscan_enforce.go:122`, [ADR-0008](../adr/0008-ioscan-untrusted-input.md) — untrusted kick input. -- `src/cmd/hive/main.go:8076` — `trustedMergerFunc`, the role-list lookup to +- `src/cmd/hive/merge_eligibility.go:50` — `trustedMergerFunc`, the role-list lookup to reuse for summoners. diff --git a/src/docs/design/master-delivery-wrapped.md b/src/docs/design/master-delivery-wrapped.md index bdff9923f5..08122502a8 100644 --- a/src/docs/design/master-delivery-wrapped.md +++ b/src/docs/design/master-delivery-wrapped.md @@ -145,17 +145,17 @@ ambiguous (`master-key-rotation.md:461-463`). This mirrors an established pattern rather than inventing one. The spoke already persists private key material on the same PVC at the same mode: -`spokeAppKeyPath = "/data/gh-app-key.pem"` (`src/cmd/hive/main.go:207`) and +`spokeAppKeyPath = "/data/gh-app-key.pem"` (`src/cmd/hive/appkeyfile.go:51`) and `spokeAppKeyDir = "/data"` (`:212`), with `spokeAppKeyFileMode = 0o600` (`:224`) and the comment "signing material must never be readable by anything else sharing the PVC or the pod" (`:222-223`). `/data` is the PVC mount in the spoke template -(`src/pkg/hub/saas_provision.go:2585`), and `/data/hive-id` (`src/cmd/hive/main.go:7232`) +(`src/pkg/hub/saas_provision.go:2585`), and `/data/hive-id` (`src/cmd/hive/main.go:6550`) already establishes that identity-critical state persists there across restarts. Following that precedent, the path should be a `var` not a `const`, so tests can redirect it and exercise the real resolution order — the reason given at -`src/cmd/hive/main.go:203-204`. +`src/cmd/hive/appkeyfile.go:47-48`. ### First boot, pod roll, PVC loss @@ -364,7 +364,7 @@ context, not by the hub. The template already injects per-hive secret material `TerminalKey`, `InviteKey`), and the `/secrets` read-only projected mount (`src/pkg/hub/saas_provision.go:3473`) already carries private key material at provision time — `spokeProvisionedAppKeyPath = "/secrets/gh-app-key.pem"` -(`src/cmd/hive/main.go:206`), which the spoke holds "from its very first boot — +(`src/cmd/hive/appkeyfile.go:50`), which the spoke holds "from its very first boot — before any heartbeat has run" (`:216`). So there is an existing, precedented channel for giving a spoke a secret at diff --git a/src/docs/design/master-key-rotation.md b/src/docs/design/master-key-rotation.md index 0b0cdd6c74..cb2e5f49a3 100644 --- a/src/docs/design/master-key-rotation.md +++ b/src/docs/design/master-key-rotation.md @@ -361,7 +361,7 @@ empty master. It is not a bug. Those three sites are inside `spokeDomainKey`, `SpokeHeartbeatKey`, and `SpokeSSOPublicKey` — all SPOKE-side resolvers. The `hive` binary serves both roles but selects between them at startup: -`runHub()` (`src/cmd/hive/main.go:9608`) is the only caller of +`runHub()` (`src/cmd/hive/main.go:8104`) is the only caller of `hub.NewHubServer`, and it is a distinct mode from the spoke path. On a hub pod those three functions are never called, so the empty `HIVE_HUB_SECRET` they would read is never consulted. diff --git a/src/docs/knowledge-curator.md b/src/docs/knowledge-curator.md index ea151857b4..427f487f19 100644 --- a/src/docs/knowledge-curator.md +++ b/src/docs/knowledge-curator.md @@ -61,7 +61,7 @@ subdirectory of one) as a knowledge source, so agents get facts from an external repo — a runbook repo, an upstream docs repo, a shared pattern library — primed into their kicks the same way wiki-layer facts are. This is implemented and live, unlike curator scheduling above: `pkg/knowledge/gitsource.go` -does the cloning, indexing, and periodic sync; `cmd/hive/main.go:2650-2694` +does the cloning, indexing, and periodic sync; `cmd/hive/main.go:2303-2347` wires configured entries at startup. ```yaml @@ -171,7 +171,7 @@ and are the *same* underlying list as `knowledge.git_sources` in - `DELETE` disconnects the live source and removes matching entries from `Config.Knowledge.GitSources`, then persists (`api.go:8155-8192`). - Editing `git_sources:` directly in `hive.yaml` takes effect on the next - process restart (main.go's startup loop at `cmd/hive/main.go:2650-2694`); + process restart (main.go's startup loop at `cmd/hive/main.go:2303-2347`); it does not hot-reload while the process is running. Use the API for a live change without a restart. diff --git a/src/pkg/github/f3_trusted_merger_source_test.go b/src/pkg/github/f3_trusted_merger_source_test.go index 547ab78492..86c359a0d6 100644 --- a/src/pkg/github/f3_trusted_merger_source_test.go +++ b/src/pkg/github/f3_trusted_merger_source_test.go @@ -2,6 +2,7 @@ package github import ( "os" + "path/filepath" "regexp" "strings" "testing" @@ -45,6 +46,38 @@ func f3ReadSource(t *testing.T, file string) string { return string(raw) } +// f3ReadPackage returns every non-test .go file in dir concatenated, so a +// source-level assertion follows a declaration that moves between files in the +// same package. cmd/hive is being split file-by-file out of a god-file, and a +// guard pinned to one filename fails on pure code motion — which reads as a +// lost fix when nothing was lost. The package is still the unit that matters: +// the declaration must exist SOMEWHERE in cmd/hive, and the wiring assertion +// below stays pinned to main.go because that is where startup lives. +func f3ReadPackage(t *testing.T, dir string) string { + t.Helper() + entries, err := os.ReadDir(dir) + if err != nil { + t.Fatalf("read dir %s: %v", dir, err) + } + var b strings.Builder + for _, e := range entries { + name := e.Name() + if e.IsDir() || !strings.HasSuffix(name, ".go") || strings.HasSuffix(name, "_test.go") { + continue + } + raw, err := os.ReadFile(filepath.Join(dir, name)) + if err != nil { + t.Fatalf("read %s: %v", name, err) + } + b.Write(raw) + b.WriteString("\n") + } + if b.Len() == 0 { + t.Fatalf("no non-test .go files found under %s", dir) + } + return b.String() +} + // f3FuncBody returns the source of a named func, from its declaration line to // the first line that is exactly "}". func f3FuncBody(t *testing.T, src, decl, name string) string { @@ -166,7 +199,7 @@ func TestF3AuthorizerIsWiredInMain(t *testing.T) { "ghClient.SetMergerAuthorizer(trustedMergerFunc(cfg)) beside StartMergeRequestWatcher.") } - body := f3FuncBody(t, src, "func trustedMergerFunc(", "trustedMergerFunc") + body := f3FuncBody(t, f3ReadPackage(t, "../../cmd/hive"), "func trustedMergerFunc(", "trustedMergerFunc") if !strings.Contains(body, "config.RoleAtLeast(role, config.RoleMerger)") { t.Error("trustedMergerFunc no longer requires at least config.RoleMerger — the sweep would " + "admit a tier below the one the dashboard queue endpoint enforces (audit F3)") From d5f311d3a5db78fbded51b61ea8b1498529af428 Mon Sep 17 00:00:00 2001 From: Andy Anderson Date: Fri, 18 Sep 2026 00:09:44 -0400 Subject: [PATCH 16/17] =?UTF-8?q?=F0=9F=90=9B=20fix(dashboard):=20refresh?= =?UTF-8?q?=20served=20status=20agent=20block=20on=20the=20fast=20tick=20(?= =?UTF-8?q?#7526)=20(#7539)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit /api/status (and the SSE replay frame a reconnecting tab receives) serves the cached s.status snapshot, which is only replaced by UpdateStatusIfFresh at the end of an eval cycle — after the whole-fleet GitHub enumeration. On a rate-limited 16-repo spoke that cycle runs far past eval_interval_s, so agents the manager started (state=running assigned on the same code path that logs "audit: agent started") still read state=stopped for 10-15+ minutes and the dashboard paints a healthy fleet red. BroadcastAgentStatus already carries correct in-memory liveness on a 10s tick with no GitHub calls, but only fanned it out over SSE. It now also patches the agent block (Agents/HiddenAgents/ConfiguredAgents) of the cached snapshot, copy-on-write so handleStatus — which marshals its loaded pointer outside statusMu — can never see a torn payload. The refresh runs ahead of the redundant-frame skip, so a skipped broadcast still freshens the cache. Signed-off-by: Andrew Anderson Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../fixed-7526-agent-liveness-snapshot.md | 1 + .../agent_snapshot_freshness_7526_test.go | 101 ++++++++++++++++++ src/pkg/dashboard/server.go | 35 ++++++ 3 files changed, 137 insertions(+) create mode 100644 changelog.d/fixed-7526-agent-liveness-snapshot.md create mode 100644 src/pkg/dashboard/agent_snapshot_freshness_7526_test.go diff --git a/changelog.d/fixed-7526-agent-liveness-snapshot.md b/changelog.d/fixed-7526-agent-liveness-snapshot.md new file mode 100644 index 0000000000..9ea7e7e905 --- /dev/null +++ b/changelog.d/fixed-7526-agent-liveness-snapshot.md @@ -0,0 +1 @@ +- Agent dots no longer show red (down) for 10-15+ minutes while the fleet is healthy ([#7526](https://github.com/hivecommons/hive/issues/7526)). `/api/status` — and the replay frame a reconnecting dashboard tab gets on the SSE stream — served a snapshot that was only ever refreshed at the end of an eval cycle, and that cycle enumerates every configured repo against the GitHub API first. On a spoke with sixteen repos under rate-limit backoff the cycle ran far past its nominal `eval_interval_s`, so agents the manager had started (and whose tmux sessions were alive and posting PR reviews) still read `state=stopped` and the whole fleet rendered as crashed. The existing 10-second agent-only tick — in-memory manager state, no GitHub calls — now also patches the agent block of the served snapshot, so liveness on every dashboard surface is at most one tick old regardless of how long the GitHub enumeration takes. diff --git a/src/pkg/dashboard/agent_snapshot_freshness_7526_test.go b/src/pkg/dashboard/agent_snapshot_freshness_7526_test.go new file mode 100644 index 0000000000..18794f83a9 --- /dev/null +++ b/src/pkg/dashboard/agent_snapshot_freshness_7526_test.go @@ -0,0 +1,101 @@ +package dashboard + +import ( + "encoding/json" + "testing" +) + +// #7526: /api/status serves a cached snapshot that used to be refreshed only by +// the eval cycle, which enumerates every configured repo against the GitHub API +// first. On a rate-limited spoke that cycle runs far past its nominal interval, +// so agents the manager started minutes ago still read state=stopped and the +// dashboard paints the whole fleet red. The fast agent-only tick must refresh +// the agent block of the served snapshot. +func TestBroadcastAgentStatusRefreshesServedSnapshot(t *testing.T) { + s := newTestServer() + + stale := minimalPayload() + stale.Agents = []FrontendAgent{ + {Name: "reviewer", State: "stopped", Busy: "idle"}, + {Name: "scanner", State: "stopped", Busy: "idle"}, + } + s.UpdateStatus(stale) + + // The 10s agent tick: manager reports both agents running. + s.BroadcastAgentStatus(&AgentStatusPayload{ + Agents: []FrontendAgent{ + {Name: "reviewer", State: "running", Busy: "working", Session: "hive-reviewer"}, + {Name: "scanner", State: "running", Busy: "idle", Session: "hive-scanner"}, + }, + ConfiguredAgents: []FrontendConfiguredAgent{{Name: "reviewer"}, {Name: "scanner"}}, + }) + + rec := doGet(s, "/api/status") + if rec.Code != 200 { + t.Fatalf("GET /api/status = %d, want 200", rec.Code) + } + var got StatusPayload + if err := json.Unmarshal(rec.Body.Bytes(), &got); err != nil { + t.Fatalf("unmarshal status: %v", err) + } + if len(got.Agents) != 2 { + t.Fatalf("agents = %d, want 2", len(got.Agents)) + } + for _, a := range got.Agents { + if a.State != "running" { + t.Errorf("agent %s state = %q, want %q — /api/status is serving "+ + "eval-cycle-stale liveness (#7526)", a.Name, a.State, "running") + } + } + if len(got.ConfiguredAgents) != 2 { + t.Errorf("configuredAgents = %d, want 2", len(got.ConfiguredAgents)) + } +} + +// The refresh must be copy-on-write: a payload already handed to a reader (as +// handleStatus does, marshalling outside statusMu) must not be mutated. +func TestRefreshAgentSnapshotIsCopyOnWrite(t *testing.T) { + s := newTestServer() + base := minimalPayload() + base.Agents = []FrontendAgent{{Name: "reviewer", State: "stopped"}} + s.UpdateStatus(base) + + s.statusMu.RLock() + held := s.status + s.statusMu.RUnlock() + + s.RefreshAgentSnapshot(&AgentStatusPayload{ + Agents: []FrontendAgent{{Name: "reviewer", State: "running"}}, + }) + + if held.Agents[0].State != "stopped" { + t.Errorf("previously loaded snapshot was mutated in place: state = %q", held.Agents[0].State) + } + s.statusMu.RLock() + current := s.status + s.statusMu.RUnlock() + if current == held { + t.Fatal("snapshot pointer unchanged, want copy-on-write replacement") + } + if current.Agents[0].State != "running" { + t.Errorf("current snapshot state = %q, want running", current.Agents[0].State) + } + if current.HiveID != base.HiveID || current.Governor.Mode != base.Governor.Mode { + t.Error("non-agent fields lost by the agent-block patch") + } +} + +// A nil cached snapshot (pre-first-eval boot) must not panic or fabricate one: +// handleStatus's "initializing" response stays intact. +func TestRefreshAgentSnapshotNoopsBeforeFirstStatus(t *testing.T) { + s := newTestServer() + s.RefreshAgentSnapshot(&AgentStatusPayload{Agents: []FrontendAgent{{Name: "reviewer", State: "running"}}}) + s.RefreshAgentSnapshot(nil) + + s.statusMu.RLock() + got := s.status + s.statusMu.RUnlock() + if got != nil { + t.Fatalf("status = %#v, want nil before the first full snapshot", got) + } +} diff --git a/src/pkg/dashboard/server.go b/src/pkg/dashboard/server.go index d2c797c2b1..55fbebbeeb 100644 --- a/src/pkg/dashboard/server.go +++ b/src/pkg/dashboard/server.go @@ -2224,10 +2224,45 @@ func (s *Server) handleGitHubAppRecheck(w http.ResponseWriter, r *http.Request) } } +// RefreshAgentSnapshot patches the agent block of the cached status snapshot +// that /api/status (and the SSE replay frame a reconnecting tab receives) +// serves, using the fast-tick agent-only payload. +// +// Without this, agent liveness on those two surfaces is only as fresh as the +// last completed eval cycle, which enumerates every configured repo against +// the GitHub API and can therefore run far past its nominal interval under +// rate-limit backoff. A whole fleet that started seconds ago then reads +// state=stopped — red "crashed" dots — for 10-15+ minutes (#7526). Agent state +// is in-memory manager state that needs no GitHub call, so it is refreshed on +// the 10s tick independently of the enumeration. +// +// Copy-on-write: handleStatus marshals the *StatusPayload it loaded outside +// statusMu, so the cached payload must never be mutated in place. +func (s *Server) RefreshAgentSnapshot(payload *AgentStatusPayload) { + if payload == nil { + return + } + s.statusMu.Lock() + defer s.statusMu.Unlock() + if s.status == nil { + return + } + patched := *s.status + patched.Agents = payload.Agents + patched.HiddenAgents = payload.HiddenAgents + patched.ConfiguredAgents = payload.ConfiguredAgents + s.status = &patched +} + // BroadcastAgentStatus sends a lightweight agent-only SSE event on a fast // cadence. Skipped if a full status was broadcast within the last 5 seconds // to avoid redundant renders on the frontend. func (s *Server) BroadcastAgentStatus(payload *AgentStatusPayload) { + // Refresh the cached snapshot FIRST, ahead of the skip below: even when the + // SSE frame is redundant, /api/status and the SSE replay frame must not be + // left serving eval-cycle-old liveness (#7526). + s.RefreshAgentSnapshot(payload) + s.statusMu.RLock() recentFull := time.Since(s.lastFullBroadcast) < agentSkipAfterFullBroadcastS s.statusMu.RUnlock() From 26e9773a279801dd49b77ba59d0f35ce190585de Mon Sep 17 00:00:00 2001 From: hive-release-bot Date: Fri, 18 Sep 2026 04:22:32 +0000 Subject: [PATCH 17/17] =?UTF-8?q?=F0=9F=94=96=20release:=20v4.55.1?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Automated release commit. Compiles changelog.d/ fragments and moves the CHANGELOG.md Unreleased section into a dated v4.55.1 entry. See src/docs/releases.md. Signed-off-by: hive-release-bot --- CHANGELOG.md | 11 +++++++++++ changelog.d/changed-7238-cmd-hive-domain-split.md | 1 - changelog.d/fixed-7526-agent-liveness-snapshot.md | 1 - changelog.d/fixed-quality-sdk-helper-test-hermetic.md | 1 - 4 files changed, 11 insertions(+), 3 deletions(-) delete mode 100644 changelog.d/changed-7238-cmd-hive-domain-split.md delete mode 100644 changelog.d/fixed-7526-agent-liveness-snapshot.md delete mode 100644 changelog.d/fixed-quality-sdk-helper-test-hermetic.md diff --git a/CHANGELOG.md b/CHANGELOG.md index 36e6e8412b..b152576176 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -11,6 +11,17 @@ Hive did not historically maintain a complete changelog. This file starts a prag ## Unreleased +## 2026-09-18 (v4.55.1) + +### Changed + +- `cmd/hive`'s `main.go` shed five self-contained domains into their own files — GitHub App key-file resolution (`appkeyfile.go`), self-upgrade marker/outcome bookkeeping (`selfupgrade.go`), login-required scanning and its sighting debounce (`login_scan.go`), auto-merge eligibility classification (`merge_eligibility.go`), and PR/issue intent verdicts (`intent_verdicts.go`) ([#7238](https://github.com/hivecommons/hive/issues/7238)). `main.go` drops from 9,795 to 8,072 lines. This is pure code motion — every one of the 242 top-level declarations was moved byte-identically, verified by a structural declaration diff, so no operator-visible behaviour changes; it makes the merge, upgrade and App-key paths reviewable on their own instead of only as a slice of a 9.8K-line file. + +### Fixed + +- Agent dots no longer show red (down) for 10-15+ minutes while the fleet is healthy ([#7526](https://github.com/hivecommons/hive/issues/7526)). `/api/status` — and the replay frame a reconnecting dashboard tab gets on the SSE stream — served a snapshot that was only ever refreshed at the end of an eval cycle, and that cycle enumerates every configured repo against the GitHub API first. On a spoke with sixteen repos under rate-limit backoff the cycle ran far past its nominal `eval_interval_s`, so agents the manager had started (and whose tmux sessions were alive and posting PR reviews) still read `state=stopped` and the whole fleet rendered as crashed. The existing 10-second agent-only tick — in-memory manager state, no GitHub calls — now also patches the agent block of the served snapshot, so liveness on every dashboard surface is at most one tick old regardless of how long the GitHub enumeration takes. +- Tests: the copilot SDK-helper probe tests in `pkg/dashboard` are now hermetic. `TestProbeCopilotModelsSDK_AbsentHelperYieldsSentinel` and `TestProbeCopilotModelsSDK_HelperNotInstalled` used to run whatever was installed at the production helper path, so on any host that ships `copilot-models.mjs` without stored copilot auth (every live agent host) the "absent helper" sentinel never fired and the whole `pkg/dashboard` suite went red with `Not authenticated`. The helper path is now a test-seam var (`setCopilotSDKHelperPathForTest`, mirroring the `knowledge.SetBaseDirForTest` convention) pointed at a temp path, and a new `TestProbeCopilotModelsSDK_FailingHelperIsNotAbsent` covers the previously untestable-on-CI third state — helper present but exiting nonzero (the #7365 case) — via a stub script instead of the real helper. + ## 2026-09-18 (v4.55.0) ### Added diff --git a/changelog.d/changed-7238-cmd-hive-domain-split.md b/changelog.d/changed-7238-cmd-hive-domain-split.md deleted file mode 100644 index b21783324a..0000000000 --- a/changelog.d/changed-7238-cmd-hive-domain-split.md +++ /dev/null @@ -1 +0,0 @@ -- `cmd/hive`'s `main.go` shed five self-contained domains into their own files — GitHub App key-file resolution (`appkeyfile.go`), self-upgrade marker/outcome bookkeeping (`selfupgrade.go`), login-required scanning and its sighting debounce (`login_scan.go`), auto-merge eligibility classification (`merge_eligibility.go`), and PR/issue intent verdicts (`intent_verdicts.go`) ([#7238](https://github.com/hivecommons/hive/issues/7238)). `main.go` drops from 9,795 to 8,072 lines. This is pure code motion — every one of the 242 top-level declarations was moved byte-identically, verified by a structural declaration diff, so no operator-visible behaviour changes; it makes the merge, upgrade and App-key paths reviewable on their own instead of only as a slice of a 9.8K-line file. diff --git a/changelog.d/fixed-7526-agent-liveness-snapshot.md b/changelog.d/fixed-7526-agent-liveness-snapshot.md deleted file mode 100644 index 9ea7e7e905..0000000000 --- a/changelog.d/fixed-7526-agent-liveness-snapshot.md +++ /dev/null @@ -1 +0,0 @@ -- Agent dots no longer show red (down) for 10-15+ minutes while the fleet is healthy ([#7526](https://github.com/hivecommons/hive/issues/7526)). `/api/status` — and the replay frame a reconnecting dashboard tab gets on the SSE stream — served a snapshot that was only ever refreshed at the end of an eval cycle, and that cycle enumerates every configured repo against the GitHub API first. On a spoke with sixteen repos under rate-limit backoff the cycle ran far past its nominal `eval_interval_s`, so agents the manager had started (and whose tmux sessions were alive and posting PR reviews) still read `state=stopped` and the whole fleet rendered as crashed. The existing 10-second agent-only tick — in-memory manager state, no GitHub calls — now also patches the agent block of the served snapshot, so liveness on every dashboard surface is at most one tick old regardless of how long the GitHub enumeration takes. diff --git a/changelog.d/fixed-quality-sdk-helper-test-hermetic.md b/changelog.d/fixed-quality-sdk-helper-test-hermetic.md deleted file mode 100644 index c4a5d1ff2c..0000000000 --- a/changelog.d/fixed-quality-sdk-helper-test-hermetic.md +++ /dev/null @@ -1 +0,0 @@ -- Tests: the copilot SDK-helper probe tests in `pkg/dashboard` are now hermetic. `TestProbeCopilotModelsSDK_AbsentHelperYieldsSentinel` and `TestProbeCopilotModelsSDK_HelperNotInstalled` used to run whatever was installed at the production helper path, so on any host that ships `copilot-models.mjs` without stored copilot auth (every live agent host) the "absent helper" sentinel never fired and the whole `pkg/dashboard` suite went red with `Not authenticated`. The helper path is now a test-seam var (`setCopilotSDKHelperPathForTest`, mirroring the `knowledge.SetBaseDirForTest` convention) pointed at a temp path, and a new `TestProbeCopilotModelsSDK_FailingHelperIsNotAbsent` covers the previously untestable-on-CI third state — helper present but exiting nonzero (the #7365 case) — via a stub script instead of the real helper.