Repository navigation
Conversation
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
|
@hanthor this PR is back in your court ( Review asks:
Reply here or push a new head to this branch (same sign-off); CI and the review swarm re-run automatically and the PR returns to the normal path. Queued in your relay fix queue (1 waiting): fixes are handed out before new work as soon as you have capacity. |
There was a problem hiding this comment.
Security review (head 7985f62)
Per-hive bearer auth is sound: hive_id is bound to the HMAC key via the existing verifyHeartbeatBearer path (src/pkg/hub/hub_keys.go:165-210, constant-time compare), and PUT/DELETE require both lease.ID and lease.HiveID to match (shared_agent_pool.go:213-218), so one tenant cannot renew/release another's lease. The spoke uses hub.SpokeHeartbeatKey(), not the master (src/cmd/hive/main.go:2181).
Finding (medium, in-scope): one authenticated tenant can indefinitely deny pool capacity to every other tenant.
waitingHives (src/pkg/hub/shared_agent_pool.go:40) is an in-memory set with no timestamp, no expiry, and no persistence. A hive is inserted on any 409 (:178-184) and removed only when that same hive later acquires (:170, :194) or sends cancel_wait (:203). The gate at :178 rejects every other hive while any entry exists, even when len(leases) < max.
Consequences:
- Stale waiter = permanent fleet-wide 409. A spoke that hits 409 once and then crashes, loses network, or is simply deconfigured never sends
cancel_wait(the client only does so onctx.Done(),src/crux/agent/shared_pool.go:118-124). Its entry stays until Hub restart, and no other tenant can ever acquire a new lease despite free slots. - Two-waiter livelock. If hives A and B both receive 409 while the pool is full, then after a release A is rejected because B is waiting, and B is rejected because A is waiting (
hasOtherWaitingHive,:109-116). Neither ever acquires.TestSharedAgentPoolHTTPSerializesTwoHivesonly ever has a single waiter, so this is untested.
Because this is a multi-tenant control plane and the trigger requires only a valid per-hive key (or an accidental crash), it is a cross-tenant availability regression that the PR's "single-waiter FIFO queue" claim does not account for.
Suggested fix — time-stamp waiters, expire them a little past the client's 3s retry, and only yield to waiters that are older than the requester (true FIFO):
--- a/src/pkg/hub/shared_agent_pool.go
+++ b/src/pkg/hub/shared_agent_pool.go
@@
- waitingHives map[string]struct{}
+ waitingHives map[string]time.Time // hive_id -> first 409 (FIFO position)
loadError error
}
+
+// A waiter that has not retried within this window is treated as gone so a
+// crashed spoke cannot starve the fleet.
+const sharedPoolWaiterStaleAfter = 15 * time.Second
@@
- waitingHives: make(map[string]struct{}),
+ waitingHives: make(map[string]time.Time),
@@
-func (p *sharedAgentPool) hasOtherWaitingHive(hiveID string) bool {
- for waitingHive := range p.waitingHives {
- if waitingHive != hiveID {
- return true
- }
- }
- return false
-}
+// hasEarlierWaitingHive reports whether another hive has been waiting longer
+// than hiveID (which may not be waiting yet). Stale waiters are pruned first.
+func (p *sharedAgentPool) hasEarlierWaitingHive(hiveID string, now time.Time) bool {
+ for h, since := range p.waitingHives {
+ if now.Sub(p.lastSeen[h]) > sharedPoolWaiterStaleAfter {
+ delete(p.waitingHives, h)
+ delete(p.lastSeen, h)
+ }
+ _ = since
+ }
+ mySince, waiting := p.waitingHives[hiveID]
+ if !waiting {
+ mySince = now
+ }
+ for h, since := range p.waitingHives {
+ if h != hiveID && since.Before(mySince) {
+ return true
+ }
+ }
+ return false
+}
@@
- if len(p.leases) >= p.max || p.hasOtherWaitingHive(req.HiveID) {
- if p.waitingHives == nil {
- p.waitingHives = make(map[string]struct{})
- }
- p.waitingHives[req.HiveID] = struct{}{}
+ if len(p.leases) >= p.max || p.hasEarlierWaitingHive(req.HiveID, now) {
+ if _, ok := p.waitingHives[req.HiveID]; !ok {
+ p.waitingHives[req.HiveID] = now
+ }
+ p.lastSeen[req.HiveID] = now
w.Header().Set("Retry-After", "3")(add lastSeen map[string]time.Time alongside waitingHives and initialise it in newSharedAgentPool.) Plus a test with two hives both receiving 409 while full, then a release, asserting the earlier waiter acquires and the later one acquires after it — and one asserting a waiter that stops retrying stops blocking after the stale window.
— hive: agent=ci-maintainer backend=copilot model=claude-fable-5.1 copilot=1.0.88
There was a problem hiding this comment.
Style / conventions review (read at head 7985f62).
1. api-reference.md citation drift — this PR turns the docs check red (src/pkg/hub/server.go:1038, :1696-1698)
Inserting the sharedAgentPool field and the three HandleFunc lines above the existing route registrations shifted every pkg/hub/server.go:<line> citation that follows. CI job relative links and anchors resolve already reports it (DRIFT doc:728 POST /api/heartbeat -> pkg/hub/server.go:1691 (line does not register it) and ~20 more). Per AGENTS.md:18, don't hand-edit — run and commit:
bash src/scripts/check-api-reference-citations.sh --fix
While there, api-reference.md is the route reference; the new POST/PUT/DELETE /api/shared-agent-pool/leases endpoints should get an entry alongside /api/heartbeat (then re-run --fix).
2. New time.Sleep poll loop in a crux unit test (src/crux/agent/shared_pool_test.go:114-123)
src/internal/testutil/sleep_ratchet_test.go:36-38 walks src/crux and the header (:20-23) says a new wait must be testutil.Eventually, never a fixed sleep; the baseline (sleepBaseline = 198) only ratchets down, so this is at risk of failing TestSleepRatchet. Use the same helper kick_cancel_test.go:301 uses (import github.com/hivecommons/hive/crux/internal/testutil, see kick_cancel_test.go:15):
- // Wait briefly for the asynchronous eviction goroutine to complete.
- deadline := time.Now().Add(2 * time.Second)
- for time.Now().Before(deadline) {
- m.mu.RLock()
- st := a.State
- lease := a.poolLeaseID
- m.mu.RUnlock()
- if st == StateStopped && lease == "" {
- break
- }
- time.Sleep(10 * time.Millisecond)
- }
+ // The eviction runs on a goroutine spawned by notifyKickOutcome.
+ testutil.EventuallyEvery(t, 2*time.Second, 10*time.Millisecond, func() bool {
+ m.mu.RLock()
+ defer m.mu.RUnlock()
+ return a.State == StateStopped && a.poolLeaseID == ""
+ })3. Contribution conventions (AGENTS.md:17)
PR title should start with the emoji prefix (✨ feat(hub,agent): …), and the dco check reports commits missing Signed-off-by (git commit -s / git rebase --signoff). Only the author can fix these.
4. Minor duplication (advisory)
src/crux/agent/manager.go:1172-1177and:1187-1192are the same 5-line "start lease watcher" block; astartSharedPoolLeaseWatcherLocked(agent, name, leaseID)helper would keep the two paths from drifting.src/pkg/hub/shared_agent_pool.go:128and thedefault:arm at:234are unreachable:server.go:1696-1698registers the handler only forPOST/PUT/DELETEmethod patterns, soServeMuxalready answers 405 withAllow.
— hive: agent=reviewer-2 backend=copilot model=claude-fable-5.1 copilot=1.0.88
There was a problem hiding this comment.
Conclusion: the lease API's "other hive is waiting" gate can wedge every participating spoke once two hives have both been told to wait. This needs a fix before it lands; the rest of the change read correctly to me.
Livelock in the waiter gate (high)
src/pkg/hub/shared_agent_pool.go:178-185 returns 409 and adds the caller to waitingHives when either capacity is full or any other hive is waiting. A hive only leaves waitingHives when it successfully acquires (:170, :194) or sends cancel_wait (:200-204). Concrete sequence with MAX_SESSIONS=1, hives A and B, one agent each:
- A holds the slot. B POSTs → full → 409,
waitingHives={B}. - A's turn ends;
evictSharedPoolSessionAfterTurnreleases A's lease. Slot free. - A's next kick →
Start→ POST.len(leases)=0, buthasOtherWaitingHive(A)sees B → 409,waitingHives={A,B}. - B retries (3 s timer,
src/crux/agent/shared_pool.go:107) →hasOtherWaitingHive(B)sees A → 409. - Both loop forever: the slot is free and neither can take it.
cancel_waitis only sent onctx.Done()(shared_pool.go:109-115), and the ctx passed toStartfrom the governor's kick pass is the long-lived process ctx, so nothing ever breaks the cycle.
Consequence beyond the pool: Acquire blocks inside Start at src/crux/agent/manager.go:1149 while agent.launching is held, and the governor's dispatch pass waits on every lane's result (src/cmd/hive/eval_cycle_seams.go:570, :583), so the whole kick cycle on both spokes stalls, not just the one agent. A hub restart doesn't help either — waitingHives is in-memory only, so a restart actually clears it, which is the only recovery I can see.
TestSharedAgentPoolHTTPSerializesTwoHives misses this because reef's 100 ms ctx expires and sends cancel_wait before school re-acquires. A test where school releases and re-acquires while reef is still polling would reproduce it.
The PR body calls this a "single-waiter FIFO queue"; the code is a mutual-exclusion gate with no ordering and no expiry. A fix needs either a real ordered queue (grant the head on the next poll, let others through when the head is served or expires) or at minimum a TTL on waitingHives entries and no gate when len(leases) < max.
Smaller notes
src/crux/agent/manager.go:1176-1178: the newif agent.OperatorStopped { return error }runs for every manager, pool or not. Previously aStop()landing during Phase 2 was overwritten by the launch; nowStartfails with "stopped during launch". Probably the better behaviour, but it is a base-path change hidden in a pool PR — worth a sentence in the body or a separate commit.- Auth looks right: bearer is verified against the
hive_idin the body (shared_agent_pool.go:150), and the cross-tenant test covers the spoof case. - CI as of head
7985f62:dcofails (no sign-off),changelog-fragment-guardandrelative links and anchors resolveare red — those need attention independently of the above.
I have not approved or requested changes as a review event; this is a comment for the maintainers.
— hive: agent=reviewer backend=copilot model=claude-fable-5.1 copilot=1.0.88
There was a problem hiding this comment.
Correctness review (read at head 7985f62).
1. Hub waiter logic deadlocks with two or more waiting hives (src/pkg/hub/shared_agent_pool.go:124-131, :172-180)
hasOtherWaitingHive denies a POST whenever any other hive is in waitingHives, and a denied hive is itself added to the set. Entries only leave on a successful grant or an explicit cancel_wait. So:
- max=1, A holds. B POSTs → 409,
waiting={B}. A releases. A's next kick POSTs →hasOtherWaitingHive(A)sees B → 409,waiting={A,B}. B POSTs → sees A → 409. Nobody is ever granted; both spokes poll every 3s forever, and their governor dispatch lanes (src/cmd/hive/eval_cycle_seams.go:566-575waits on every lane) stall indefinitely becauseAcquireonly exits on ctx cancel (src/crux/agent/shared_pool.go:124-138). - Any spoke that crashes while waiting (no
cancel_wait) blocks every other hive until the hub restarts —waitingHiveshas no expiry. Acquirealso skipscancel_waitwhen ctx is cancelled during the in-flight POST (requestreturns a non-busy error at:118-120), leaking the waiter entry.
TestSharedAgentPoolHTTPSerializesTwoHives only passes because the 100ms busyCtx triggers cancel_wait. The PR body's "single-waiter FIFO queue" is not what is implemented — this is an unordered set with mutual veto.
Suggested fix — track a single, expiring priority waiter:
type sharedAgentPool struct {
mu sync.Mutex
path string
max int
ttl time.Duration
leases map[string]sharedAgentLease
- waitingHives map[string]struct{}
+ waiter string // hive with next-slot priority
+ waiterSeen time.Time // last poll from waiter; stale after sharedPoolWaiterGrace
loadError error
}
+
+// sharedPoolWaiterGrace is > 3× the spoke poll interval; a waiter that stops
+// polling loses priority instead of blocking everyone until hub restart.
+const sharedPoolWaiterGrace = 10 * time.Second
+
+func (p *sharedAgentPool) otherWaiterActive(hiveID string, now time.Time) bool {
+ return p.waiter != "" && p.waiter != hiveID && now.Sub(p.waiterSeen) < sharedPoolWaiterGrace
+}
+
+func (p *sharedAgentPool) clearWaiter(hiveID string) {
+ if p.waiter == hiveID {
+ p.waiter = ""
+ }
+}
@@ POST
- if len(p.leases) >= p.max || p.hasOtherWaitingHive(req.HiveID) {
- if p.waitingHives == nil {
- p.waitingHives = make(map[string]struct{})
- }
- p.waitingHives[req.HiveID] = struct{}{}
+ if len(p.leases) >= p.max || p.otherWaiterActive(req.HiveID, now) {
+ if p.waiter == "" || p.waiter == req.HiveID || now.Sub(p.waiterSeen) >= sharedPoolWaiterGrace {
+ p.waiter, p.waiterSeen = req.HiveID, now
+ }
w.Header().Set("Retry-After", "3")and replace the three delete(p.waitingHives, req.HiveID) sites with p.clearWaiter(req.HiveID). Please add a hub test with three hives (A holds, B and C poll, A releases → B then C get granted) — the current tests cannot catch this.
2. Restart paths launch a CLI without a lease (src/crux/agent/manager_restart.go:938, :787, :506)
Only Manager.Start acquires a lease (manager.go:1147-1153). restartWithReason, RestartMissingSessionWithReason and RestartWithBootstrap call launchInTmux directly. After the pool evicts an agent (State=StateStopped, poolLeaseID="", shared_pool.go:195-203), a dashboard Restart or RestartThenSendKick (manager_restart.go:514) relaunches it with no lease and no watcher, so it runs unbounded and is never evicted — the cap the PR exists to enforce is bypassed. Minimal fix: in those three paths, when m.sharedPool != nil && agent.poolLeaseID == "", refuse and point at Start (which acquires), e.g. in restartWithReason after the not-found check:
if m.sharedPool != nil && agent.poolLeaseID == "" {
m.mu.Unlock()
return fmt.Errorf("agent %s has no shared pool lease; use Start", name)
}(adjusting for the lock state at each site), or route them through the same acquire/watch sequence Start uses.
3. Advisory (in-scope, low)
Start→Acquireruns inside the governor's per-agent dispatch lane with the eval-loop ctx and no deadline (main.go:7640,shared_pool.go:112-139), so while another tenant holds the slot this hive's whole eval cycle waits (eval_cycle_seams.go:578). Even after fix #1, consider a bounded acquire timeout so a long turn elsewhere does not freeze sweeps here.- Post-turn eviction fires for every settled kind including
KickOutcomeQuestion(kick_outcome.go:241-258→shared_pool.go:160-162), so a session that stopped to ask the operator a question is killed before anyone can answer it.
— hive: agent=adjudicator backend=copilot model=claude-fable-5.1 copilot=1.0.88
|
hive/check-report (head 7985f62): every red check here is PR-local —
The hive cannot push to 🐝 Hive Agent: — hive: agent=scanner backend=copilot model=claude-fable-5.1 copilot=1.0.88 |
|
@hanthor this PR is back in your court ( Review asks:
Reply here or push a new head to this branch (same sign-off); CI and the review swarm re-run automatically and the PR returns to the normal path. Queued in your relay fix queue (1 waiting): fixes are handed out before new work as soon as you have capacity. |
There was a problem hiding this comment.
Conclusion: the new commit 14451be ("auto-start stopped agents on kick") adds a fatal-crash path in SendKick and does not touch the hub waiter gate that two earlier reviews flagged as a livelock. Needs a fix before it lands. Read at head 14451be against the tree.
1. SendKick double-unlocks m.mu on any Start failure — fatal, not recoverable (critical)
src/crux/agent/manager_kick.go:113 takes defer m.mu.Unlock(). The new branch at :126 unlocks manually before calling m.Start, and if Start returns an error the function returns at :128 without re-locking. The deferred Unlock then runs on an unlocked mutex, which is fatal error: sync: unlock of unlocked mutex — a runtime fatal that recover() cannot catch, so the whole hive process dies.
Start has plenty of ordinary error returns that reach this path: manager.go:961 (not found), :966 (ACMM gate), :972 (already running), :976 (launch already in progress), :1151 (Acquire error), plus the tmux launch failures below. The :976 case is the realistic trigger: two kicks for the same stopped agent (governor pass + dashboard kick, or SendKick and SendKickAsync racing) — the second sees agent.launching and crashes the spoke.
The async twin is correct: src/crux/agent/kick_async.go:252-270 holds no defer and balances every unlock explicitly (the header comment at :249-251 even says so). The fix in SendKick is to re-lock before returning:
m.mu.Unlock()
- if err := m.Start(context.Background(), name); err != nil {
- return fmt.Errorf("auto-start stopped agent for kick: %w", err)
- }
+ err := m.Start(context.Background(), name)
m.mu.Lock()
+ if err != nil {
+ return fmt.Errorf("auto-start stopped agent for kick: %w", err)
+ }2. context.Background() makes the kick path unbounded (high)
Both manager_kick.go:127 and kick_async.go:262 call Start(context.Background(), …). Inside Start, sharedPool.Acquire(ctx, name) at manager.go:1149 only exits on a grant or ctx.Done(), and cancel_wait is only sent on ctx.Done() (src/crux/agent/shared_pool.go:109-115). With a Background ctx a kick that cannot get a slot blocks forever, holds agent.launching, and never tells the hub to drop the waiter entry. Combined with the unchanged mutual-veto gate in src/pkg/hub/shared_agent_pool.go:178-185 (see the two reviews at 7985f62 — the hub file is byte-identical at this head), the livelock reported there is now reachable with no escape at all, since this path has no caller-supplied deadline. Please give the auto-start a bounded ctx (or thread the kick's ctx through) and address the hub gate.
3. Unaddressed from 7985f62
The diff 7985f62...14451be touches only kick_async.go, manager_kick.go and shared_pool.go:189-190 (a time.Equal fix — that one is correct and good). Still open:
- hub waiter livelock (
src/pkg/hub/shared_agent_pool.go:124-131,:172-185) — unchanged time.Sleeppoll loop insrc/crux/agent/shared_pool_test.go:114-123vsTestSleepRatchet— unchangedapi-reference.mdcitation drift (relative links and anchors resolvestill red) — runbash src/scripts/check-api-reference-citations.sh --fixchangelog-fragment-guard,test (rest 2/4),coverage, anddcoare red at this head
Smaller notes
- The auto-start commit adds no test;
shared_pool_test.gohas no case for kicking aStateStoppedagent with the pool enabled. A test that makesStartfail (e.g.launchingalready set) would have caught #1 immediately. - The auto-start behaviour is not described in the PR body; a sentence there would help the maintainer, since it changes what "stopped" means for pool-enabled spokes (
validatePaneKickLockednow admits stopped agents,kick_async.go:226-228). - The same 10-line auto-start block appears in both
SendKickanddeliverKickAsync; a sharedautoStartForKickLockedhelper would have made the lock discipline identical in both.
— hive: agent=reviewer-2 backend=copilot model=claude-fable-5.1 copilot=1.0.88
There was a problem hiding this comment.
Security review (head 14451be) — one tenant-isolation regression and one unbounded-wait DoS surface; both agent-fixable.
1. Any single tenant can starve every other tenant indefinitely (medium, in-scope)
src/pkg/hub/shared_agent_pool.go:109-116 (hasOtherWaitingHive) and :177-184: when a POST is refused, the caller's hive_id is added to waitingHives, and from then on every other hive gets 409 even when len(p.leases) < p.max. The entry is only removed when that same hive later acquires or sends cancel_wait (:197-200). Nothing expires it. A spoke that crashes, loses network, or is simply killed mid-wait (the client's cancel at src/crux/agent/shared_pool.go:109-113 is best-effort, 2s, errors ignored) leaves a permanent tombstone that blocks all other tenants until the hub restarts. A tenant with its own valid heartbeat key can also do this on purpose with one request. In a multi-tenant pool this makes one tenant's liveness gate everyone's capacity — the opposite of the isolation the PR body claims.
Fix: make waiting entries expire (the client retries every 3s, so ~15s is generous):
- waitingHives map[string]struct{}
+ waitingHives map[string]time.Time // hive -> last time it was told to wait
...
func (p *sharedAgentPool) hasOtherWaitingHive(hiveID string) bool {
- for waitingHive := range p.waitingHives {
- if waitingHive != hiveID {
+ cutoff := time.Now().Add(-15 * time.Second)
+ for waitingHive, seen := range p.waitingHives {
+ if seen.Before(cutoff) {
+ delete(p.waitingHives, waitingHive)
+ continue
+ }
+ if waitingHive != hiveID {
return true
}
}
return false
}
...
- p.waitingHives[req.HiveID] = struct{}{}
+ p.waitingHives[req.HiveID] = now(and make(map[string]time.Time) at the two construction sites). Please add a test: hive A waits, never cancels, hive B must still acquire after the window.
2. Webhook-driven kick blocks forever on a busy pool (medium, in-scope)
src/crux/agent/manager_kick.go:125-127 auto-starts a stopped agent with m.Start(context.Background(), name). Start calls m.sharedPool.Acquire(ctx, name) (src/crux/agent/manager.go:1149), whose retry loop at src/crux/agent/shared_pool.go:93-115 only exits on success, a non-409 error, or ctx.Done() — and this ctx never completes. SendKick is reached from inbound GitHub webhooks via kickerAdapter.Kick (src/cmd/hive/hookwire.go:302-306), so while another tenant holds the slot (or issue #1 has poisoned the queue) each webhook delivery parks a goroutine forever and the spoke accumulates them without bound.
Fix: bound the auto-start wait:
startCtx, cancelStart := context.WithTimeout(context.Background(), 2*time.Minute)
err := m.Start(startCtx, name)
cancelStart()
if err != nil {
return fmt.Errorf("auto-start stopped agent for kick: %w", err)
}
— hive: agent=ci-maintainer backend=copilot model=claude-fable-5.1 copilot=1.0.88
There was a problem hiding this comment.
Correctness review (head 14451be)
1. SendKick auto-start path double-unlocks m.mu → process crash (high).
src/crux/agent/manager_kick.go:113 has defer m.mu.Unlock(). The new branch at :126 calls m.mu.Unlock() and, if m.Start fails (:127-128, e.g. the Hub returns 401/503 or the pool is unavailable), returns without re-locking — the deferred Unlock then fires on an unlocked mutex: fatal error: sync: unlock of unlocked mutex. Any kick to a stopped agent while the pool is enabled and the Hub is unreachable takes the whole spoke down. deliverKickAsync (kick_async.go:259-269) does this correctly; SendKick does not.
m.mu.Unlock()
startErr := m.Start(context.Background(), name)
m.mu.Lock()
if startErr != nil {
return fmt.Errorf("auto-start stopped agent for kick: %w", startErr)
}
agent, ok = m.agents[name]
Please also add a test for this path (a recordingSharedSessionPool whose acquireFn returns an error, then SendKick) — none of the new tests exercise auto-start-on-kick.
2. Hub waitingHives never expires → one dead waiter starves every other Hive (medium).
src/pkg/hub/shared_agent_pool.go:170 sets waitingHives[hiveID] on every 409 and only removes it on a successful POST (:162,:185) or an explicit cancel_wait DELETE (:193). A spoke that crashes, restarts, or loses connectivity mid-wait never sends cancel_wait, and hasOtherWaitingHive (:113-120) then returns 409 to every other tenant forever — even with zero live leases. The client retries every 3s (crux/agent/shared_pool.go:109), so a waiter entry older than ~10s is stale. Store a timestamp and prune stale entries alongside expired leases (:149-153):
- waitingHives map[string]struct{}
+ waitingHives map[string]time.Time
...
for key, lease := range p.leases {
if !lease.ExpiresAt.After(now) {
delete(p.leases, key)
}
}
+ for hive, seen := range p.waitingHives {
+ if now.Sub(seen) > 10*time.Second {
+ delete(p.waitingHives, hive)
+ }
+ }(and p.waitingHives[req.HiveID] = now at :170).
3. CI is red on three gates this diff trips (medium).
TestCruxBoundaryRatchet(job 122059620597):cmd/hive/main.go:2183andpkg/hub/shared_agent_pool_test.go:14referenceagent.NewSharedSessionPoolClient; baseline is 0 for both packages. Expose the constructor viacrux/cruxruntime(e.g.cruxruntime.WithSharedSessionPoolClient(hubURL, hiveID, token)) and have the hub test use a plainnet/httpclient instead of importingcrux/agent.TestSleepRatchet:src/crux/agent/shared_pool_test.go:131addstime.Sleep(10 * time.Millisecond); poll on a channel/time.Afterselect instead.- api-reference citation drift from the
hub/server.goroute edits: runbash src/scripts/check-api-reference-citations.sh --fixand commit.
4. Advisory (low): kick auto-start acquires with context.Background() (manager_kick.go:127, kick_async.go:262), so Acquire's retry loop (crux/agent/shared_pool.go:97-119) has no deadline — a synchronous dashboard kick blocks for as long as another Hive's turn runs. Consider a bounded timeout and a "queued behind shared pool" refusal reason.
— hive: agent=adjudicator backend=copilot model=claude-fable-5.1 copilot=1.0.88
There was a problem hiding this comment.
Read the full diff against v5 plus the surrounding Manager code at head 14451be. The design (per-Hive HMAC bearer, lease persisted atomically, agent-local tmux/UID untouched) is sound, but two defects in the current head need fixing before this can land. Neither is caught by the added tests.
1. SendKick auto-start path unlocks an already-unlocked mutex → process-wide fatal (critical)
src/crux/agent/manager_kick.go:112 takes m.mu.Lock() with defer m.mu.Unlock(). The new branch at manager_kick.go:124-128 does m.mu.Unlock(), calls m.Start(...), and on error returns while the mutex is unlocked — the deferred Unlock then runs on an unlocked sync.Mutex, which is fatal error: sync: unlock of unlocked mutex (not a recoverable panic). Same shape at manager_kick.go:131: if !ok || agent.State != StateRunning { return ... } after re-locking is fine, but the error return at line 126 is not.
Trigger: any Acquire failure — Hub unreachable, 401 from a key rotation, 503 from loadError/persistence — on a kick to a stopped agent takes down the whole spoke. The deliverKickAsync variant at kick_async.go:259-274 is written correctly (explicit unlocks, no defer), so the fix is to mirror that or re-lock before returning.
2. Hub waiter queue deadlocks with two waiters, and a dead waiter starves everyone (high)
src/pkg/hub/shared_agent_pool.go:172-179: a POST is refused with 409 when len(p.leases) >= p.max or p.hasOtherWaitingHive(req.HiveID), and the refused hive is added to waitingHives. Entries are removed only when that same hive later acquires (:166, :193) or sends cancel_wait (:199-203). Consequences I traced:
- Hives A and B both waiting on a full pool. Slot frees. A retries →
hasOtherWaitingHive("A")sees B → 409. B retries → sees A → 409. Neither ever acquires; bothAcquireloops (crux/agent/shared_pool.go:98-122) spin forever on a 3s timer. Recovery only via Hub restart (waitingHivesis not persisted) or one side's ctx cancelling. - A spoke that is SIGKILLed mid-wait never sends
cancel_wait, so its entry blocks every other hive permanently, even with free capacity. There is no TTL on waiters, unlike leases. - Because
SendKick/deliverKickAsynccallStart(context.Background(), …), the kick caller blocks indefinitely in that situation.
TestSharedAgentPoolHTTPSerializesTwoHives has a single waiter whose ctx expires (sending cancel_wait) before the slot frees, so it never exercises either case. A waiter timestamp with expiry, or a plain FIFO where only the head-of-queue blocks others, would fix it.
CI red (already visible, listing for completeness)
test (rest 2/4): crux boundary ratchet —cmd/hive(main.go:29) andpkg/hub(shared_agent_pool_test.go:14) now importcrux/agent; and sleep ratchet +1 fromcrux/agent/shared_pool_test.go:123.changelog-fragment-guard:pkg/hub/server.goroute edits driftedsrc/docs/api-reference.mdcitations —bash src/scripts/check-api-reference-citations.sh --fix.- DCO: commits lack
Signed-off-by.
Docs for the three env vars and the changelog fragment look complete and accurate to the code. I have not approved or requested changes as a review event; a maintainer should decide once the two defects above are addressed.
— hive: agent=reviewer backend=copilot model=claude-fable-5.1 copilot=1.0.88
There was a problem hiding this comment.
Reviewed at head 14451be (style/maintainability perspective).
1. SendKick auto-start branch double-unlocks m.mu on Start failure — panic. src/crux/agent/manager_kick.go:112-113 holds defer m.mu.Unlock(); the new branch at :125-129 calls m.mu.Unlock() then returns on error without re-locking, so the deferred unlock fires on an unlocked mutex (sync: unlock of unlocked mutex). Every other unlock/relock in this function re-locks before returning (:171, :175). deliverKickAsync (kick_async.go:260-265) is correct because that function has no defer.
m.mu.Unlock()
err := m.Start(context.Background(), name)
m.mu.Lock()
if err != nil {
return fmt.Errorf("auto-start stopped agent for kick: %w", err)
}
2. CI: api-reference.md citations drifted (changelog-fragment-guard will fail). Adding the three HandleFunc lines at src/pkg/hub/server.go:1693-1699 shifted every hub route below it; src/docs/api-reference.md:728 still cites server.go:1691 for /api/heartbeat (now :1699). bash src/scripts/check-api-reference-citations.sh at head reports 26 drifted citations (0 at merge-base). Run it with --fix and commit; while there, add rows for POST/PUT/DELETE /api/shared-agent-pool/leases since the table enumerates hub routes.
3. CI: TestSleepRatchet will fail. src/crux/agent/shared_pool_test.go:123 adds a time.Sleep(10 * time.Millisecond) poll loop; src/internal/testutil/sleep_ratchet_test.go:24 pins sleepBaseline = 198 over pkg/cmd/crux and head now has 199. Replace the poll with a channel: have recordingSharedSessionPool.Release close/send on a released chan string and select on it with a timeout.
4. Duplicated eligibility predicate and watcher spawn. m.sharedPool != nil && agent.State == StateStopped && !agent.Paused && !agent.OperatorStopped appears at kick_async.go:226 (negated), kick_async.go:260, manager_kick.go:125; the lease-watcher spawn block is repeated verbatim at manager.go:1172-1177 and :1187-1192. Extract func (m *Manager) sharedPoolAutoStartEligibleLocked(a *AgentProcess) bool and func (m *Manager) startSharedPoolLeaseWatcherLocked(a *AgentProcess, leaseID string) so the three sites cannot drift.
Minor: src/pkg/hub/shared_agent_pool.go:234-236 default: branch is unreachable after the method check at :128.
— hive: agent=reviewer-2 backend=copilot model=claude-fable-5.1 copilot=1.0.88
|
hive/check-report: at head
No code is pushed by the hive to a fork branch; this comment is the full report. 🐝 Hive Agent: — hive: agent=scanner backend=copilot model=claude-fable-5.1 copilot=1.0.88 |
Open PR overlap checkBase branch:
Conflicts are checked with every PR merged onto the current tip of
|
…nt runner capacity Introduce an opt-in Hub-coordinated session lease pool so multiple Hive spokes sharing constrained runner hosts can serialize heavyweight CLI processes without sharing filesystem trees, tmux sessions, or credentials. - Add /api/shared-agent-pool/leases on Hub backed by per-Hive HMAC auth, lease persistence across restarts, and a single-waiter FIFO queue. - Add WithSharedSessionPool runtime and Manager option to acquire a lease before launch and evict idle sessions back to stopped state post-turn. - Add HIVE_SHARED_AGENT_POOL_ENABLED and HIVE_SHARED_AGENT_POOL_MAX_SESSIONS environment controls. - Add comprehensive regression and HTTP integration tests across hub and agent. Signed-off-by: James Reilly <jreilly1821@gmail.com>
…bled Signed-off-by: James Reilly <jreilly1821@gmail.com>
…chet, and api-reference citations Signed-off-by: James Reilly <jreilly1821@gmail.com>
14451be to
64e752e
Compare
…w says why and offers Adopt #14524 (hanthor, XXL, stage/072, nudged twice, 26h, hive offline) was never adopted: every sweep logged "author maintains the repository; not adopting" and nothing in the UI said so. Owner decision (2026-10-28): clanker PRs are adoptable on disconnect regardless of the owner's repo role. - config: governor.labels.stage.adoption.maintainer_policy adopt|ask|never, default adopt. adopt never reads the role; ask holds a maintainer's PR for an owner and records author-is-maintainer on the stage record; never is the previous rule (hold + release). Human-authored PRs stay unadopted under all three (the 072 machinery is clanker-only). - engine: maintainerHoldsAdoption / maintainerReleasesAdoption route the five authorMaintainsRepo call sites through the policy; the owner's adoption/supersede request lifts the ask hold. - tracker: every nudged 072 PR row carries adoption{author, due, blocked_by, since, nudges, author_online, author_registered, author_last_seen, author_maintainer, policy, action, text} from the stage record and the same hub presence the adoption uses; caption and an amber pill read "waiting on @hanthor (maintainer, offline 26h) — adopt?" instead of a bare "must act". - decision API: decision "adopt" (PR only, write, adoption enabled) posts an attributed note, sets adoption/supersede, audits repo_item_decision_adopt; SPA row action decide:adopt in the same decide:* vocabulary with an in-app confirm. - Settings → Labels three-way selector with one-line explanations. - tests (config, store, engine policy branches, tracker payload/text/ caption, decision API, governor API, SPA helpers), docs, changelog. Signed-off-by: Andrew Anderson <andy@clubanderson.com>
…w says why and offers Adopt #14524 (hanthor, XXL, stage/072, nudged twice, 26h, hive offline) was never adopted: every sweep logged "author maintains the repository; not adopting" and nothing in the UI said so. Owner decision (2026-10-28): clanker PRs are adoptable on disconnect regardless of the owner's repo role. - config: governor.labels.stage.adoption.maintainer_policy adopt|ask|never, default adopt. adopt never reads the role; ask holds a maintainer's PR for an owner and records author-is-maintainer on the stage record; never is the previous rule (hold + release). Human-authored PRs stay unadopted under all three (the 072 machinery is clanker-only). - engine: maintainerHoldsAdoption / maintainerReleasesAdoption route the five authorMaintainsRepo call sites through the policy; the owner's adoption/supersede request lifts the ask hold. - tracker: every nudged 072 PR row carries adoption{author, due, blocked_by, since, nudges, author_online, author_registered, author_last_seen, author_maintainer, policy, action, text} from the stage record and the same hub presence the adoption uses; caption and an amber pill read "waiting on @hanthor (maintainer, offline 26h) — adopt?" instead of a bare "must act". - decision API: decision "adopt" (PR only, write, adoption enabled) posts an attributed note, sets adoption/supersede, audits repo_item_decision_adopt; SPA row action decide:adopt in the same decide:* vocabulary with an in-app confirm. - Settings → Labels three-way selector with one-line explanations. - tests (config, store, engine policy branches, tracker payload/text/ caption, decision API, governor API, SPA helpers), docs, changelog. Signed-off-by: Andrew Anderson <andy@clubanderson.com>
…w says why and offers Adopt (#15016) #14524 (hanthor, XXL, stage/072, nudged twice, 26h, hive offline) was never adopted: every sweep logged "author maintains the repository; not adopting" and nothing in the UI said so. Owner decision (2026-10-28): clanker PRs are adoptable on disconnect regardless of the owner's repo role. - config: governor.labels.stage.adoption.maintainer_policy adopt|ask|never, default adopt. adopt never reads the role; ask holds a maintainer's PR for an owner and records author-is-maintainer on the stage record; never is the previous rule (hold + release). Human-authored PRs stay unadopted under all three (the 072 machinery is clanker-only). - engine: maintainerHoldsAdoption / maintainerReleasesAdoption route the five authorMaintainsRepo call sites through the policy; the owner's adoption/supersede request lifts the ask hold. - tracker: every nudged 072 PR row carries adoption{author, due, blocked_by, since, nudges, author_online, author_registered, author_last_seen, author_maintainer, policy, action, text} from the stage record and the same hub presence the adoption uses; caption and an amber pill read "waiting on @hanthor (maintainer, offline 26h) — adopt?" instead of a bare "must act". - decision API: decision "adopt" (PR only, write, adoption enabled) posts an attributed note, sets adoption/supersede, audits repo_item_decision_adopt; SPA row action decide:adopt in the same decide:* vocabulary with an in-app confirm. - Settings → Labels three-way selector with one-line explanations. - tests (config, store, engine policy branches, tracker payload/text/ caption, decision API, governor API, SPA helpers), docs, changelog. Signed-off-by: Andrew Anderson <andy@clubanderson.com>
|
@hanthor the hive has adopted this PR ( |
Review-fix attempt 1/6 — ready, but could not be pushed to this forkThe automated fix for the aggregated review findings is committed as @hanthor — to land it on this PR: git fetch https://github.com/hivecommons/hive.git review-fix/14524-attempt-1
git merge --ff-only FETCH_HEAD # on feat/shared-session-pool (based on a9e21b50)
git pushWhat it changes (one commit, 9 files):
Compiled locally ( 🐝 Hive Agent: — hive: agent=ci-maintainer backend=copilot model=claude-fable-5.1 copilot=1.0.88 |
|
Review-fix attempt 2/6 is ready but could not be pushed to this PR branch: the hive bot has no push access to the fork The fix commit is published on upstream as branch git fetch https://github.com/hivecommons/hive.git review-fix/pr-14524-attempt-2
git cherry-pick f09fb8e7e903e4c6643cc3c2c0056eae0a68cdf3 # or: git merge FETCH_HEAD
git pushWhat it addresses:
🐝 Hive Agent: — hive: agent=ci-maintainer backend=copilot model=claude-fable-5.1 copilot=1.0.88 |
Review-fix attempt 3/6 — ready, but could not be pushed to this forkThe hive bot still cannot push to @hanthor — to bring it into this PR: git fetch https://github.com/hivecommons/hive.git review-fix/pr-14524-attempt-3
git cherry-pick cb9e50fe3c259ed1c6d8fa5e92f46008977fe94b
git pushWhat it addresses (all aggregated findings):
Validation on the agent host: 🐝 Hive Agent: — hive: agent=ci-maintainer backend=copilot model=claude-fable-5.1 copilot=1.0.88 |
|
Review-fix attempt 4/6: the fix commit is ready but could not be pushed to this fork branch — the hive bot has no write access to The commit is parked upstream on branch git fetch https://github.com/hivecommons/hive.git fix/shared-session-pool-review-14524
git cherry-pick 5748117b86c2a5a7b1935534890af6359cb413c8
git pushIt addresses every aggregated finding:
Validated locally with 🐝 Hive Agent: — hive: agent=ci-maintainer backend=copilot model=claude-fable-5.1 copilot=1.0.88 |
|
Review-fix attempt 5/6 — commit ready but could not be pushed to this PR branch This PR's head lives on the fork @hanthor — to land it on this PR: git fetch https://github.com/hivecommons/hive.git review-fix/14524-shared-pool-attempt-5
git cherry-pick 7ee6126daa100d8d4c38a5aa2537014ec2c7a5af
git push origin feat/shared-session-poolWhat the commit addresses (all aggregated findings):
Validation: 🐝 Hive Agent: — hive: agent=ci-maintainer backend=copilot model=claude-fable-5.1 copilot=1.0.88 |
|
PR needs rebase. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
What this does
Introduces an opt-in Hub-coordinated session lease pool so multiple Hive spokes sharing constrained runner hosts can serialize heavyweight CLI processes without sharing filesystem trees, tmux sessions, or credentials.
/api/shared-agent-pool/leaseson the Hub backed by per-Hive HMAC authentication, lease persistence across restarts, and a single-waiter FIFO queue.WithSharedSessionPoolruntime and Manager option to acquire a lease before CLI launch and evict idle sessions back to stopped state post-turn.HIVE_SHARED_AGENT_POOL_ENABLEDandHIVE_SHARED_AGENT_POOL_MAX_SESSIONSenvironment controls.