Skip to content

✨ feat(hub,agent): Hub-coordinated live CLI session pool for multi-tenant runner capacity - #14524

Closed
hanthor wants to merge 4 commits into
hivecommons:v5from
tuna-os:feat/shared-session-pool
Closed

hanthor wants to merge 4 commits into
hivecommons:v5from
tuna-os:feat/shared-session-pool

Conversation

@hanthor

@hanthor hanthor commented Oct 27, 2026

Copy link
Copy Markdown
Contributor

What this does

Introduces an opt-in Hub-coordinated session lease pool so multiple Hive spokes sharing constrained runner hosts can serialize heavyweight CLI processes without sharing filesystem trees, tmux sessions, or credentials.

  • Adds /api/shared-agent-pool/leases on the Hub backed by per-Hive HMAC authentication, lease persistence across restarts, and a single-waiter FIFO queue.
  • Adds WithSharedSessionPool runtime and Manager option to acquire a lease before CLI launch and evict idle sessions back to stopped state post-turn.
  • Adds HIVE_SHARED_AGENT_POOL_ENABLED and HIVE_SHARED_AGENT_POOL_MAX_SESSIONS environment controls.
  • Adds comprehensive regression and HTTP integration tests across hub and agent packages.

@kubestellar-prow kubestellar-prow Bot added the dco-signoff: no Indicates the PR's author has not signed the DCO. label Oct 27, 2026
@kubestellar-prow

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign hanthor for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@kubestellar-prow kubestellar-prow Bot added the size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. label Oct 27, 2026
@hivecommons-hive hivecommons-hive Bot added author/clanker A contributor's own hive authored this PR under their login no-dco A commit is missing Signed-off-by — run `git rebase --signoff` role/trusted Author is trusted for unattended merge (trusted-author role or registry/claim tier in require_tiers) stage/072-returned-to-contributor Contributor PR: CI red, changes requested or conflicting; diagnosis posted, author must act labels Oct 27, 2026
@hivecommons-hive

Copy link
Copy Markdown
Contributor

@hanthor this PR is back in your court (stage/072-returned-to-contributor). The hive will not merge it until the head changes.

Review asks:

  • 1 commit is missing the Signed-off-by trailer (DCO) — run git rebase --signoff HEAD~1 && git push --force-with-lease

Reply here or push a new head to this branch (same sign-off); CI and the review swarm re-run automatically and the PR returns to the normal path.

Queued in your relay fix queue (1 waiting): fixes are handed out before new work as soon as you have capacity.

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Security review (head 7985f62)

Per-hive bearer auth is sound: hive_id is bound to the HMAC key via the existing verifyHeartbeatBearer path (src/pkg/hub/hub_keys.go:165-210, constant-time compare), and PUT/DELETE require both lease.ID and lease.HiveID to match (shared_agent_pool.go:213-218), so one tenant cannot renew/release another's lease. The spoke uses hub.SpokeHeartbeatKey(), not the master (src/cmd/hive/main.go:2181).

Finding (medium, in-scope): one authenticated tenant can indefinitely deny pool capacity to every other tenant.

waitingHives (src/pkg/hub/shared_agent_pool.go:40) is an in-memory set with no timestamp, no expiry, and no persistence. A hive is inserted on any 409 (:178-184) and removed only when that same hive later acquires (:170, :194) or sends cancel_wait (:203). The gate at :178 rejects every other hive while any entry exists, even when len(leases) < max.

Consequences:

  1. Stale waiter = permanent fleet-wide 409. A spoke that hits 409 once and then crashes, loses network, or is simply deconfigured never sends cancel_wait (the client only does so on ctx.Done(), src/crux/agent/shared_pool.go:118-124). Its entry stays until Hub restart, and no other tenant can ever acquire a new lease despite free slots.
  2. Two-waiter livelock. If hives A and B both receive 409 while the pool is full, then after a release A is rejected because B is waiting, and B is rejected because A is waiting (hasOtherWaitingHive, :109-116). Neither ever acquires. TestSharedAgentPoolHTTPSerializesTwoHives only ever has a single waiter, so this is untested.

Because this is a multi-tenant control plane and the trigger requires only a valid per-hive key (or an accidental crash), it is a cross-tenant availability regression that the PR's "single-waiter FIFO queue" claim does not account for.

Suggested fix — time-stamp waiters, expire them a little past the client's 3s retry, and only yield to waiters that are older than the requester (true FIFO):

--- a/src/pkg/hub/shared_agent_pool.go
+++ b/src/pkg/hub/shared_agent_pool.go
@@
-	waitingHives map[string]struct{}
+	waitingHives map[string]time.Time // hive_id -> first 409 (FIFO position)
 	loadError    error
 }
+
+// A waiter that has not retried within this window is treated as gone so a
+// crashed spoke cannot starve the fleet.
+const sharedPoolWaiterStaleAfter = 15 * time.Second
@@
-		waitingHives: make(map[string]struct{}),
+		waitingHives: make(map[string]time.Time),
@@
-func (p *sharedAgentPool) hasOtherWaitingHive(hiveID string) bool {
-	for waitingHive := range p.waitingHives {
-		if waitingHive != hiveID {
-			return true
-		}
-	}
-	return false
-}
+// hasEarlierWaitingHive reports whether another hive has been waiting longer
+// than hiveID (which may not be waiting yet). Stale waiters are pruned first.
+func (p *sharedAgentPool) hasEarlierWaitingHive(hiveID string, now time.Time) bool {
+	for h, since := range p.waitingHives {
+		if now.Sub(p.lastSeen[h]) > sharedPoolWaiterStaleAfter {
+			delete(p.waitingHives, h)
+			delete(p.lastSeen, h)
+		}
+		_ = since
+	}
+	mySince, waiting := p.waitingHives[hiveID]
+	if !waiting {
+		mySince = now
+	}
+	for h, since := range p.waitingHives {
+		if h != hiveID && since.Before(mySince) {
+			return true
+		}
+	}
+	return false
+}
@@
-		if len(p.leases) >= p.max || p.hasOtherWaitingHive(req.HiveID) {
-			if p.waitingHives == nil {
-				p.waitingHives = make(map[string]struct{})
-			}
-			p.waitingHives[req.HiveID] = struct{}{}
+		if len(p.leases) >= p.max || p.hasEarlierWaitingHive(req.HiveID, now) {
+			if _, ok := p.waitingHives[req.HiveID]; !ok {
+				p.waitingHives[req.HiveID] = now
+			}
+			p.lastSeen[req.HiveID] = now
 			w.Header().Set("Retry-After", "3")

(add lastSeen map[string]time.Time alongside waitingHives and initialise it in newSharedAgentPool.) Plus a test with two hives both receiving 409 while full, then a release, asserting the earlier waiter acquires and the later one acquires after it — and one asserting a waiter that stops retrying stops blocking after the stale window.

— hive: agent=ci-maintainer backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Style / conventions review (read at head 7985f62).

1. api-reference.md citation drift — this PR turns the docs check red (src/pkg/hub/server.go:1038, :1696-1698)
Inserting the sharedAgentPool field and the three HandleFunc lines above the existing route registrations shifted every pkg/hub/server.go:<line> citation that follows. CI job relative links and anchors resolve already reports it (DRIFT doc:728 POST /api/heartbeat -> pkg/hub/server.go:1691 (line does not register it) and ~20 more). Per AGENTS.md:18, don't hand-edit — run and commit:

bash src/scripts/check-api-reference-citations.sh --fix

While there, api-reference.md is the route reference; the new POST/PUT/DELETE /api/shared-agent-pool/leases endpoints should get an entry alongside /api/heartbeat (then re-run --fix).

2. New time.Sleep poll loop in a crux unit test (src/crux/agent/shared_pool_test.go:114-123)
src/internal/testutil/sleep_ratchet_test.go:36-38 walks src/crux and the header (:20-23) says a new wait must be testutil.Eventually, never a fixed sleep; the baseline (sleepBaseline = 198) only ratchets down, so this is at risk of failing TestSleepRatchet. Use the same helper kick_cancel_test.go:301 uses (import github.com/hivecommons/hive/crux/internal/testutil, see kick_cancel_test.go:15):

-	// Wait briefly for the asynchronous eviction goroutine to complete.
-	deadline := time.Now().Add(2 * time.Second)
-	for time.Now().Before(deadline) {
-		m.mu.RLock()
-		st := a.State
-		lease := a.poolLeaseID
-		m.mu.RUnlock()
-		if st == StateStopped && lease == "" {
-			break
-		}
-		time.Sleep(10 * time.Millisecond)
-	}
+	// The eviction runs on a goroutine spawned by notifyKickOutcome.
+	testutil.EventuallyEvery(t, 2*time.Second, 10*time.Millisecond, func() bool {
+		m.mu.RLock()
+		defer m.mu.RUnlock()
+		return a.State == StateStopped && a.poolLeaseID == ""
+	})

3. Contribution conventions (AGENTS.md:17)
PR title should start with the emoji prefix (✨ feat(hub,agent): …), and the dco check reports commits missing Signed-off-by (git commit -s / git rebase --signoff). Only the author can fix these.

4. Minor duplication (advisory)

  • src/crux/agent/manager.go:1172-1177 and :1187-1192 are the same 5-line "start lease watcher" block; a startSharedPoolLeaseWatcherLocked(agent, name, leaseID) helper would keep the two paths from drifting.
  • src/pkg/hub/shared_agent_pool.go:128 and the default: arm at :234 are unreachable: server.go:1696-1698 registers the handler only for POST/PUT/DELETE method patterns, so ServeMux already answers 405 with Allow.

— hive: agent=reviewer-2 backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Conclusion: the lease API's "other hive is waiting" gate can wedge every participating spoke once two hives have both been told to wait. This needs a fix before it lands; the rest of the change read correctly to me.

Livelock in the waiter gate (high)

src/pkg/hub/shared_agent_pool.go:178-185 returns 409 and adds the caller to waitingHives when either capacity is full or any other hive is waiting. A hive only leaves waitingHives when it successfully acquires (:170, :194) or sends cancel_wait (:200-204). Concrete sequence with MAX_SESSIONS=1, hives A and B, one agent each:

  1. A holds the slot. B POSTs → full → 409, waitingHives={B}.
  2. A's turn ends; evictSharedPoolSessionAfterTurn releases A's lease. Slot free.
  3. A's next kick → Start → POST. len(leases)=0, but hasOtherWaitingHive(A) sees B → 409, waitingHives={A,B}.
  4. B retries (3 s timer, src/crux/agent/shared_pool.go:107) → hasOtherWaitingHive(B) sees A → 409.
  5. Both loop forever: the slot is free and neither can take it. cancel_wait is only sent on ctx.Done() (shared_pool.go:109-115), and the ctx passed to Start from the governor's kick pass is the long-lived process ctx, so nothing ever breaks the cycle.

Consequence beyond the pool: Acquire blocks inside Start at src/crux/agent/manager.go:1149 while agent.launching is held, and the governor's dispatch pass waits on every lane's result (src/cmd/hive/eval_cycle_seams.go:570, :583), so the whole kick cycle on both spokes stalls, not just the one agent. A hub restart doesn't help either — waitingHives is in-memory only, so a restart actually clears it, which is the only recovery I can see.

TestSharedAgentPoolHTTPSerializesTwoHives misses this because reef's 100 ms ctx expires and sends cancel_wait before school re-acquires. A test where school releases and re-acquires while reef is still polling would reproduce it.

The PR body calls this a "single-waiter FIFO queue"; the code is a mutual-exclusion gate with no ordering and no expiry. A fix needs either a real ordered queue (grant the head on the next poll, let others through when the head is served or expires) or at minimum a TTL on waitingHives entries and no gate when len(leases) < max.

Smaller notes

  • src/crux/agent/manager.go:1176-1178: the new if agent.OperatorStopped { return error } runs for every manager, pool or not. Previously a Stop() landing during Phase 2 was overwritten by the launch; now Start fails with "stopped during launch". Probably the better behaviour, but it is a base-path change hidden in a pool PR — worth a sentence in the body or a separate commit.
  • Auth looks right: bearer is verified against the hive_id in the body (shared_agent_pool.go:150), and the cross-tenant test covers the spoof case.
  • CI as of head 7985f62: dco fails (no sign-off), changelog-fragment-guard and relative links and anchors resolve are red — those need attention independently of the above.

I have not approved or requested changes as a review event; this is a comment for the maintainers.

— hive: agent=reviewer backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correctness review (read at head 7985f62).

1. Hub waiter logic deadlocks with two or more waiting hives (src/pkg/hub/shared_agent_pool.go:124-131, :172-180)

hasOtherWaitingHive denies a POST whenever any other hive is in waitingHives, and a denied hive is itself added to the set. Entries only leave on a successful grant or an explicit cancel_wait. So:

  • max=1, A holds. B POSTs → 409, waiting={B}. A releases. A's next kick POSTs → hasOtherWaitingHive(A) sees B → 409, waiting={A,B}. B POSTs → sees A → 409. Nobody is ever granted; both spokes poll every 3s forever, and their governor dispatch lanes (src/cmd/hive/eval_cycle_seams.go:566-575 waits on every lane) stall indefinitely because Acquire only exits on ctx cancel (src/crux/agent/shared_pool.go:124-138).
  • Any spoke that crashes while waiting (no cancel_wait) blocks every other hive until the hub restarts — waitingHives has no expiry.
  • Acquire also skips cancel_wait when ctx is cancelled during the in-flight POST (request returns a non-busy error at :118-120), leaking the waiter entry.

TestSharedAgentPoolHTTPSerializesTwoHives only passes because the 100ms busyCtx triggers cancel_wait. The PR body's "single-waiter FIFO queue" is not what is implemented — this is an unordered set with mutual veto.

Suggested fix — track a single, expiring priority waiter:

 type sharedAgentPool struct {
 	mu           sync.Mutex
 	path         string
 	max          int
 	ttl          time.Duration
 	leases       map[string]sharedAgentLease
-	waitingHives map[string]struct{}
+	waiter       string    // hive with next-slot priority
+	waiterSeen   time.Time // last poll from waiter; stale after sharedPoolWaiterGrace
 	loadError    error
 }
+
+// sharedPoolWaiterGrace is > 3× the spoke poll interval; a waiter that stops
+// polling loses priority instead of blocking everyone until hub restart.
+const sharedPoolWaiterGrace = 10 * time.Second
+
+func (p *sharedAgentPool) otherWaiterActive(hiveID string, now time.Time) bool {
+	return p.waiter != "" && p.waiter != hiveID && now.Sub(p.waiterSeen) < sharedPoolWaiterGrace
+}
+
+func (p *sharedAgentPool) clearWaiter(hiveID string) {
+	if p.waiter == hiveID {
+		p.waiter = ""
+	}
+}
@@ POST
-		if len(p.leases) >= p.max || p.hasOtherWaitingHive(req.HiveID) {
-			if p.waitingHives == nil {
-				p.waitingHives = make(map[string]struct{})
-			}
-			p.waitingHives[req.HiveID] = struct{}{}
+		if len(p.leases) >= p.max || p.otherWaiterActive(req.HiveID, now) {
+			if p.waiter == "" || p.waiter == req.HiveID || now.Sub(p.waiterSeen) >= sharedPoolWaiterGrace {
+				p.waiter, p.waiterSeen = req.HiveID, now
+			}
 			w.Header().Set("Retry-After", "3")

and replace the three delete(p.waitingHives, req.HiveID) sites with p.clearWaiter(req.HiveID). Please add a hub test with three hives (A holds, B and C poll, A releases → B then C get granted) — the current tests cannot catch this.

2. Restart paths launch a CLI without a lease (src/crux/agent/manager_restart.go:938, :787, :506)

Only Manager.Start acquires a lease (manager.go:1147-1153). restartWithReason, RestartMissingSessionWithReason and RestartWithBootstrap call launchInTmux directly. After the pool evicts an agent (State=StateStopped, poolLeaseID="", shared_pool.go:195-203), a dashboard Restart or RestartThenSendKick (manager_restart.go:514) relaunches it with no lease and no watcher, so it runs unbounded and is never evicted — the cap the PR exists to enforce is bypassed. Minimal fix: in those three paths, when m.sharedPool != nil && agent.poolLeaseID == "", refuse and point at Start (which acquires), e.g. in restartWithReason after the not-found check:

if m.sharedPool != nil && agent.poolLeaseID == "" {
	m.mu.Unlock()
	return fmt.Errorf("agent %s has no shared pool lease; use Start", name)
}

(adjusting for the lock state at each site), or route them through the same acquire/watch sequence Start uses.

3. Advisory (in-scope, low)

  • Start → Acquire runs inside the governor's per-agent dispatch lane with the eval-loop ctx and no deadline (main.go:7640, shared_pool.go:112-139), so while another tenant holds the slot this hive's whole eval cycle waits (eval_cycle_seams.go:578). Even after fix #1, consider a bounded acquire timeout so a long turn elsewhere does not freeze sweeps here.
  • Post-turn eviction fires for every settled kind including KickOutcomeQuestion (kick_outcome.go:241-258 → shared_pool.go:160-162), so a session that stopped to ask the operator a question is killed before anyone can answer it.

— hive: agent=adjudicator backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive

Copy link
Copy Markdown
Contributor

hive/check-report (head 7985f62): every red check here is PR-local — v6 is green on the same checks. Fixes, in order:

  1. DCO / dco — the single commit 7985f62 lacks a Signed-off-by: trailer. git commit --amend -s && git push --force-with-lease.
  2. test (rest 2/4) → TestCruxBoundaryRatchet (job 122054560429): cmd/hive and pkg/hub each gain one new agent.<Symbol> reference — agent.NewSharedSessionPoolClient — above their baseline of 0. New agent symbols must be reached through the exported runtime surface (the crux package), not imported from pkg/agent directly; see src/docs/design/crux.md / 📝 RFC: Crux - split the agent runtime into a reusable component (stage 1: in-repo boundary) #13536.
  3. test (rest 2/4) → TestSleepRatchet: 199 time.Sleep( calls in *_test.go under pkg/cmd/crux, above the baseline of 198. Replace the new sleep in the added test with a channel/timeout wait.
  4. relative links and anchors resolve (job 122054090731): the pkg/hub/server.go edits shifted every route line cited by src/docs/api-reference.md:726-733+ (DRIFT ... GET /fleet -> pkg/hub/server.go:1750 (line does not register it), etc.). Run bash src/scripts/check-api-reference-citations.sh --fix and commit the rewritten citations.
  5. coverage is only consequential (rest=failure shard); it clears once (2)–(3) pass.

The hive cannot push to tuna-os/hive:feat/shared-session-pool; this is a report only.


🐝 Hive Agent: scanner | Instance: hosted-available-oke-11-placeholder-r05x | SHA: unknown

— hive: agent=scanner backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive

Copy link
Copy Markdown
Contributor

@hanthor this PR is back in your court (stage/072-returned-to-contributor). The hive will not merge it until the head changes.

Review asks:

  • 2 commits are missing the Signed-off-by trailer (DCO) — run git rebase --signoff HEAD~2 && git push --force-with-lease

Reply here or push a new head to this branch (same sign-off); CI and the review swarm re-run automatically and the PR returns to the normal path.

Queued in your relay fix queue (1 waiting): fixes are handed out before new work as soon as you have capacity.

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Conclusion: the new commit 14451be ("auto-start stopped agents on kick") adds a fatal-crash path in SendKick and does not touch the hub waiter gate that two earlier reviews flagged as a livelock. Needs a fix before it lands. Read at head 14451be against the tree.

1. SendKick double-unlocks m.mu on any Start failure — fatal, not recoverable (critical)

src/crux/agent/manager_kick.go:113 takes defer m.mu.Unlock(). The new branch at :126 unlocks manually before calling m.Start, and if Start returns an error the function returns at :128 without re-locking. The deferred Unlock then runs on an unlocked mutex, which is fatal error: sync: unlock of unlocked mutex — a runtime fatal that recover() cannot catch, so the whole hive process dies.

Start has plenty of ordinary error returns that reach this path: manager.go:961 (not found), :966 (ACMM gate), :972 (already running), :976 (launch already in progress), :1151 (Acquire error), plus the tmux launch failures below. The :976 case is the realistic trigger: two kicks for the same stopped agent (governor pass + dashboard kick, or SendKick and SendKickAsync racing) — the second sees agent.launching and crashes the spoke.

The async twin is correct: src/crux/agent/kick_async.go:252-270 holds no defer and balances every unlock explicitly (the header comment at :249-251 even says so). The fix in SendKick is to re-lock before returning:

 			m.mu.Unlock()
-			if err := m.Start(context.Background(), name); err != nil {
-				return fmt.Errorf("auto-start stopped agent for kick: %w", err)
-			}
+			err := m.Start(context.Background(), name)
 			m.mu.Lock()
+			if err != nil {
+				return fmt.Errorf("auto-start stopped agent for kick: %w", err)
+			}

2. context.Background() makes the kick path unbounded (high)

Both manager_kick.go:127 and kick_async.go:262 call Start(context.Background(), …). Inside Start, sharedPool.Acquire(ctx, name) at manager.go:1149 only exits on a grant or ctx.Done(), and cancel_wait is only sent on ctx.Done() (src/crux/agent/shared_pool.go:109-115). With a Background ctx a kick that cannot get a slot blocks forever, holds agent.launching, and never tells the hub to drop the waiter entry. Combined with the unchanged mutual-veto gate in src/pkg/hub/shared_agent_pool.go:178-185 (see the two reviews at 7985f62 — the hub file is byte-identical at this head), the livelock reported there is now reachable with no escape at all, since this path has no caller-supplied deadline. Please give the auto-start a bounded ctx (or thread the kick's ctx through) and address the hub gate.

3. Unaddressed from 7985f62

The diff 7985f62...14451be touches only kick_async.go, manager_kick.go and shared_pool.go:189-190 (a time.Equal fix — that one is correct and good). Still open:

  • hub waiter livelock (src/pkg/hub/shared_agent_pool.go:124-131, :172-185) — unchanged
  • time.Sleep poll loop in src/crux/agent/shared_pool_test.go:114-123 vs TestSleepRatchet — unchanged
  • api-reference.md citation drift (relative links and anchors resolve still red) — run bash src/scripts/check-api-reference-citations.sh --fix
  • changelog-fragment-guard, test (rest 2/4), coverage, and dco are red at this head

Smaller notes

  • The auto-start commit adds no test; shared_pool_test.go has no case for kicking a StateStopped agent with the pool enabled. A test that makes Start fail (e.g. launching already set) would have caught #1 immediately.
  • The auto-start behaviour is not described in the PR body; a sentence there would help the maintainer, since it changes what "stopped" means for pool-enabled spokes (validatePaneKickLocked now admits stopped agents, kick_async.go:226-228).
  • The same 10-line auto-start block appears in both SendKick and deliverKickAsync; a shared autoStartForKickLocked helper would have made the lock discipline identical in both.

— hive: agent=reviewer-2 backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Security review (head 14451be) — one tenant-isolation regression and one unbounded-wait DoS surface; both agent-fixable.

1. Any single tenant can starve every other tenant indefinitely (medium, in-scope)

src/pkg/hub/shared_agent_pool.go:109-116 (hasOtherWaitingHive) and :177-184: when a POST is refused, the caller's hive_id is added to waitingHives, and from then on every other hive gets 409 even when len(p.leases) < p.max. The entry is only removed when that same hive later acquires or sends cancel_wait (:197-200). Nothing expires it. A spoke that crashes, loses network, or is simply killed mid-wait (the client's cancel at src/crux/agent/shared_pool.go:109-113 is best-effort, 2s, errors ignored) leaves a permanent tombstone that blocks all other tenants until the hub restarts. A tenant with its own valid heartbeat key can also do this on purpose with one request. In a multi-tenant pool this makes one tenant's liveness gate everyone's capacity — the opposite of the isolation the PR body claims.

Fix: make waiting entries expire (the client retries every 3s, so ~15s is generous):

-	waitingHives map[string]struct{}
+	waitingHives map[string]time.Time // hive -> last time it was told to wait
...
 func (p *sharedAgentPool) hasOtherWaitingHive(hiveID string) bool {
-	for waitingHive := range p.waitingHives {
-		if waitingHive != hiveID {
+	cutoff := time.Now().Add(-15 * time.Second)
+	for waitingHive, seen := range p.waitingHives {
+		if seen.Before(cutoff) {
+			delete(p.waitingHives, waitingHive)
+			continue
+		}
+		if waitingHive != hiveID {
 			return true
 		}
 	}
 	return false
 }
...
-			p.waitingHives[req.HiveID] = struct{}{}
+			p.waitingHives[req.HiveID] = now

(and make(map[string]time.Time) at the two construction sites). Please add a test: hive A waits, never cancels, hive B must still acquire after the window.

2. Webhook-driven kick blocks forever on a busy pool (medium, in-scope)

src/crux/agent/manager_kick.go:125-127 auto-starts a stopped agent with m.Start(context.Background(), name). Start calls m.sharedPool.Acquire(ctx, name) (src/crux/agent/manager.go:1149), whose retry loop at src/crux/agent/shared_pool.go:93-115 only exits on success, a non-409 error, or ctx.Done() — and this ctx never completes. SendKick is reached from inbound GitHub webhooks via kickerAdapter.Kick (src/cmd/hive/hookwire.go:302-306), so while another tenant holds the slot (or issue #1 has poisoned the queue) each webhook delivery parks a goroutine forever and the spoke accumulates them without bound.

Fix: bound the auto-start wait:

			startCtx, cancelStart := context.WithTimeout(context.Background(), 2*time.Minute)
			err := m.Start(startCtx, name)
			cancelStart()
			if err != nil {
				return fmt.Errorf("auto-start stopped agent for kick: %w", err)
			}

— hive: agent=ci-maintainer backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correctness review (head 14451be)

1. SendKick auto-start path double-unlocks m.mu → process crash (high).
src/crux/agent/manager_kick.go:113 has defer m.mu.Unlock(). The new branch at :126 calls m.mu.Unlock() and, if m.Start fails (:127-128, e.g. the Hub returns 401/503 or the pool is unavailable), returns without re-locking — the deferred Unlock then fires on an unlocked mutex: fatal error: sync: unlock of unlocked mutex. Any kick to a stopped agent while the pool is enabled and the Hub is unreachable takes the whole spoke down. deliverKickAsync (kick_async.go:259-269) does this correctly; SendKick does not.

			m.mu.Unlock()
			startErr := m.Start(context.Background(), name)
			m.mu.Lock()
			if startErr != nil {
				return fmt.Errorf("auto-start stopped agent for kick: %w", startErr)
			}
			agent, ok = m.agents[name]

Please also add a test for this path (a recordingSharedSessionPool whose acquireFn returns an error, then SendKick) — none of the new tests exercise auto-start-on-kick.

2. Hub waitingHives never expires → one dead waiter starves every other Hive (medium).
src/pkg/hub/shared_agent_pool.go:170 sets waitingHives[hiveID] on every 409 and only removes it on a successful POST (:162,:185) or an explicit cancel_wait DELETE (:193). A spoke that crashes, restarts, or loses connectivity mid-wait never sends cancel_wait, and hasOtherWaitingHive (:113-120) then returns 409 to every other tenant forever — even with zero live leases. The client retries every 3s (crux/agent/shared_pool.go:109), so a waiter entry older than ~10s is stale. Store a timestamp and prune stale entries alongside expired leases (:149-153):

-	waitingHives map[string]struct{}
+	waitingHives map[string]time.Time
...
 	for key, lease := range p.leases {
 		if !lease.ExpiresAt.After(now) {
 			delete(p.leases, key)
 		}
 	}
+	for hive, seen := range p.waitingHives {
+		if now.Sub(seen) > 10*time.Second {
+			delete(p.waitingHives, hive)
+		}
+	}

(and p.waitingHives[req.HiveID] = now at :170).

3. CI is red on three gates this diff trips (medium).

  • TestCruxBoundaryRatchet (job 122059620597): cmd/hive/main.go:2183 and pkg/hub/shared_agent_pool_test.go:14 reference agent.NewSharedSessionPoolClient; baseline is 0 for both packages. Expose the constructor via crux/cruxruntime (e.g. cruxruntime.WithSharedSessionPoolClient(hubURL, hiveID, token)) and have the hub test use a plain net/http client instead of importing crux/agent.
  • TestSleepRatchet: src/crux/agent/shared_pool_test.go:131 adds time.Sleep(10 * time.Millisecond); poll on a channel/time.After select instead.
  • api-reference citation drift from the hub/server.go route edits: run bash src/scripts/check-api-reference-citations.sh --fix and commit.

4. Advisory (low): kick auto-start acquires with context.Background() (manager_kick.go:127, kick_async.go:262), so Acquire's retry loop (crux/agent/shared_pool.go:97-119) has no deadline — a synchronous dashboard kick blocks for as long as another Hive's turn runs. Consider a bounded timeout and a "queued behind shared pool" refusal reason.

— hive: agent=adjudicator backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Read the full diff against v5 plus the surrounding Manager code at head 14451be. The design (per-Hive HMAC bearer, lease persisted atomically, agent-local tmux/UID untouched) is sound, but two defects in the current head need fixing before this can land. Neither is caught by the added tests.

1. SendKick auto-start path unlocks an already-unlocked mutex → process-wide fatal (critical)

src/crux/agent/manager_kick.go:112 takes m.mu.Lock() with defer m.mu.Unlock(). The new branch at manager_kick.go:124-128 does m.mu.Unlock(), calls m.Start(...), and on error returns while the mutex is unlocked — the deferred Unlock then runs on an unlocked sync.Mutex, which is fatal error: sync: unlock of unlocked mutex (not a recoverable panic). Same shape at manager_kick.go:131: if !ok || agent.State != StateRunning { return ... } after re-locking is fine, but the error return at line 126 is not.

Trigger: any Acquire failure — Hub unreachable, 401 from a key rotation, 503 from loadError/persistence — on a kick to a stopped agent takes down the whole spoke. The deliverKickAsync variant at kick_async.go:259-274 is written correctly (explicit unlocks, no defer), so the fix is to mirror that or re-lock before returning.

2. Hub waiter queue deadlocks with two waiters, and a dead waiter starves everyone (high)

src/pkg/hub/shared_agent_pool.go:172-179: a POST is refused with 409 when len(p.leases) >= p.max or p.hasOtherWaitingHive(req.HiveID), and the refused hive is added to waitingHives. Entries are removed only when that same hive later acquires (:166, :193) or sends cancel_wait (:199-203). Consequences I traced:

  • Hives A and B both waiting on a full pool. Slot frees. A retries → hasOtherWaitingHive("A") sees B → 409. B retries → sees A → 409. Neither ever acquires; both Acquire loops (crux/agent/shared_pool.go:98-122) spin forever on a 3s timer. Recovery only via Hub restart (waitingHives is not persisted) or one side's ctx cancelling.
  • A spoke that is SIGKILLed mid-wait never sends cancel_wait, so its entry blocks every other hive permanently, even with free capacity. There is no TTL on waiters, unlike leases.
  • Because SendKick/deliverKickAsync call Start(context.Background(), …), the kick caller blocks indefinitely in that situation.

TestSharedAgentPoolHTTPSerializesTwoHives has a single waiter whose ctx expires (sending cancel_wait) before the slot frees, so it never exercises either case. A waiter timestamp with expiry, or a plain FIFO where only the head-of-queue blocks others, would fix it.

CI red (already visible, listing for completeness)

  • test (rest 2/4): crux boundary ratchet — cmd/hive (main.go:29) and pkg/hub (shared_agent_pool_test.go:14) now import crux/agent; and sleep ratchet +1 from crux/agent/shared_pool_test.go:123.
  • changelog-fragment-guard: pkg/hub/server.go route edits drifted src/docs/api-reference.md citations — bash src/scripts/check-api-reference-citations.sh --fix.
  • DCO: commits lack Signed-off-by.

Docs for the three env vars and the changelog fragment look complete and accurate to the code. I have not approved or requested changes as a review event; a maintainer should decide once the two defects above are addressed.

— hive: agent=reviewer backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed at head 14451be (style/maintainability perspective).

1. SendKick auto-start branch double-unlocks m.mu on Start failure — panic. src/crux/agent/manager_kick.go:112-113 holds defer m.mu.Unlock(); the new branch at :125-129 calls m.mu.Unlock() then returns on error without re-locking, so the deferred unlock fires on an unlocked mutex (sync: unlock of unlocked mutex). Every other unlock/relock in this function re-locks before returning (:171, :175). deliverKickAsync (kick_async.go:260-265) is correct because that function has no defer.

			m.mu.Unlock()
			err := m.Start(context.Background(), name)
			m.mu.Lock()
			if err != nil {
				return fmt.Errorf("auto-start stopped agent for kick: %w", err)
			}

2. CI: api-reference.md citations drifted (changelog-fragment-guard will fail). Adding the three HandleFunc lines at src/pkg/hub/server.go:1693-1699 shifted every hub route below it; src/docs/api-reference.md:728 still cites server.go:1691 for /api/heartbeat (now :1699). bash src/scripts/check-api-reference-citations.sh at head reports 26 drifted citations (0 at merge-base). Run it with --fix and commit; while there, add rows for POST/PUT/DELETE /api/shared-agent-pool/leases since the table enumerates hub routes.

3. CI: TestSleepRatchet will fail. src/crux/agent/shared_pool_test.go:123 adds a time.Sleep(10 * time.Millisecond) poll loop; src/internal/testutil/sleep_ratchet_test.go:24 pins sleepBaseline = 198 over pkg/cmd/crux and head now has 199. Replace the poll with a channel: have recordingSharedSessionPool.Release close/send on a released chan string and select on it with a timeout.

4. Duplicated eligibility predicate and watcher spawn. m.sharedPool != nil && agent.State == StateStopped && !agent.Paused && !agent.OperatorStopped appears at kick_async.go:226 (negated), kick_async.go:260, manager_kick.go:125; the lease-watcher spawn block is repeated verbatim at manager.go:1172-1177 and :1187-1192. Extract func (m *Manager) sharedPoolAutoStartEligibleLocked(a *AgentProcess) bool and func (m *Manager) startSharedPoolLeaseWatcherLocked(a *AgentProcess, leaseID string) so the three sites cannot drift.

Minor: src/pkg/hub/shared_agent_pool.go:234-236 default: branch is unreachable after the method check at :128.

— hive: agent=reviewer-2 backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive

Copy link
Copy Markdown
Contributor

hive/check-report: at head 14451be, beyond the DCO sign-off already requested above, four CI checks fail on this PR's own diff (base v5 is green on all of them). Concrete fixes:

  1. changelog-fragment-guard / relative links and anchors resolve — src/docs/api-reference.md citations into pkg/hub/server.go drifted (e.g. DRIFT doc:726 GET /fleet -> pkg/hub/server.go:1750 (line does not register it), and ~10 more at doc:727-737). Run bash src/scripts/check-api-reference-citations.sh --fix and commit the rewritten citations.

  2. test (rest 2/4) → internal/testutil.TestSleepRatchet — 199 "time.Sleep(" calls in *_test.go under [pkg cmd crux], above the baseline of 198. The new time.Sleep( in a _test.go added by this PR must be replaced with a channel/timeout wait (select on a done channel or require.Eventually).

  3. test (rest 2/4) → internal/testutil.TestCruxBoundaryRatchet — cmd/hive and pkg/hub each now reference agent.NewSharedSessionPoolClient, above their baseline of 0 agent.<Symbol> identifiers. Per src/docs/design/crux.md (📝 RFC: Crux - split the agent runtime into a reusable component (stage 1: in-repo boundary) #13536), expose the pool client on the crux runtime surface (the crux package) and have cmd/hive / pkg/hub import it from there instead of pkg/agent.

  4. coverage is derived — it turns green once the rest shard passes.

No code is pushed by the hive to a fork branch; this comment is the full report.


🐝 Hive Agent: scanner | Instance: hosted-available-oke-11-placeholder-r05x | SHA: unknown

— hive: agent=scanner backend=copilot model=claude-fable-5.1 copilot=1.0.88

@github-actions

github-actions Bot commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

Open PR overlap check

Base branch: v5

This PR conflicts with v5 itself. Rebase it first: until it merges cleanly into the base, conflicts with other open PRs cannot be told apart from base drift, so only same-file overlaps are listed.

Conflicts are checked with every PR merged onto the current tip of v5, so a PR that only needs a rebase is not reported as conflicting with its siblings.

PR Title Relation Files
#14807 ✨ dashboard: stream Job pod logs in the terminal pane for sandbox.runtime: job agents same files src/docs/api-reference.md
#14921 ✨ feature: inline the linked Linear issue into review kicks same files src/cmd/hive/main.go
#15085 ✨ Activity Monitor: Adopt now and Reroute to another clanker for PRs waiting on a contributor's clanker same files src/cmd/hive/main.go
#15081 🐛 fix: explain Activity Monitor merge gates and sweep timing same files src/cmd/hive/main.go

…nt runner capacity

Introduce an opt-in Hub-coordinated session lease pool so multiple Hive
spokes sharing constrained runner hosts can serialize heavyweight CLI
processes without sharing filesystem trees, tmux sessions, or credentials.

- Add /api/shared-agent-pool/leases on Hub backed by per-Hive HMAC auth,
  lease persistence across restarts, and a single-waiter FIFO queue.
- Add WithSharedSessionPool runtime and Manager option to acquire a lease
  before launch and evict idle sessions back to stopped state post-turn.
- Add HIVE_SHARED_AGENT_POOL_ENABLED and HIVE_SHARED_AGENT_POOL_MAX_SESSIONS
  environment controls.
- Add comprehensive regression and HTTP integration tests across hub and agent.

Signed-off-by: James Reilly <jreilly1821@gmail.com>
…bled

Signed-off-by: James Reilly <jreilly1821@gmail.com>
…chet, and api-reference citations

Signed-off-by: James Reilly <jreilly1821@gmail.com>
@hanthor
hanthor force-pushed the feat/shared-session-pool branch from 14451be to 64e752e Compare October 27, 2026 13:00
@kubestellar-prow kubestellar-prow Bot added dco-signoff: yes Indicates the PR's author has signed the DCO. and removed dco-signoff: no Indicates the PR's author has not signed the DCO. labels Oct 27, 2026
@hanthor hanthor changed the title feat(hub,agent): Hub-coordinated live CLI session pool for multi-tenant runner capacity ✨ feat(hub,agent): Hub-coordinated live CLI session pool for multi-tenant runner capacity Oct 27, 2026
@github-actions github-actions Bot removed the conflicts-with-open-pr Open PR has a merge conflict with another open PR label Oct 28, 2026
clubanderson added a commit that referenced this pull request Oct 28, 2026
…w says why and offers Adopt

#14524 (hanthor, XXL, stage/072, nudged twice, 26h, hive
offline) was never adopted: every sweep logged "author maintains the
repository; not adopting" and nothing in the UI said so. Owner decision
(2026-10-28): clanker PRs are adoptable on disconnect regardless of the
owner's repo role.

- config: governor.labels.stage.adoption.maintainer_policy adopt|ask|never,
  default adopt. adopt never reads the role; ask holds a maintainer's PR
  for an owner and records author-is-maintainer on the stage record;
  never is the previous rule (hold + release). Human-authored PRs stay
  unadopted under all three (the 072 machinery is clanker-only).
- engine: maintainerHoldsAdoption / maintainerReleasesAdoption route the
  five authorMaintainsRepo call sites through the policy; the owner's
  adoption/supersede request lifts the ask hold.
- tracker: every nudged 072 PR row carries adoption{author, due,
  blocked_by, since, nudges, author_online, author_registered,
  author_last_seen, author_maintainer, policy, action, text} from the
  stage record and the same hub presence the adoption uses; caption and
  an amber pill read "waiting on @hanthor (maintainer, offline 26h) —
  adopt?" instead of a bare "must act".
- decision API: decision "adopt" (PR only, write, adoption enabled) posts
  an attributed note, sets adoption/supersede, audits
  repo_item_decision_adopt; SPA row action decide:adopt in the same
  decide:* vocabulary with an in-app confirm.
- Settings → Labels three-way selector with one-line explanations.
- tests (config, store, engine policy branches, tracker payload/text/
  caption, decision API, governor API, SPA helpers), docs, changelog.

Signed-off-by: Andrew Anderson <andy@clubanderson.com>
clubanderson added a commit that referenced this pull request Oct 28, 2026
…w says why and offers Adopt

#14524 (hanthor, XXL, stage/072, nudged twice, 26h, hive
offline) was never adopted: every sweep logged "author maintains the
repository; not adopting" and nothing in the UI said so. Owner decision
(2026-10-28): clanker PRs are adoptable on disconnect regardless of the
owner's repo role.

- config: governor.labels.stage.adoption.maintainer_policy adopt|ask|never,
  default adopt. adopt never reads the role; ask holds a maintainer's PR
  for an owner and records author-is-maintainer on the stage record;
  never is the previous rule (hold + release). Human-authored PRs stay
  unadopted under all three (the 072 machinery is clanker-only).
- engine: maintainerHoldsAdoption / maintainerReleasesAdoption route the
  five authorMaintainsRepo call sites through the policy; the owner's
  adoption/supersede request lifts the ask hold.
- tracker: every nudged 072 PR row carries adoption{author, due,
  blocked_by, since, nudges, author_online, author_registered,
  author_last_seen, author_maintainer, policy, action, text} from the
  stage record and the same hub presence the adoption uses; caption and
  an amber pill read "waiting on @hanthor (maintainer, offline 26h) —
  adopt?" instead of a bare "must act".
- decision API: decision "adopt" (PR only, write, adoption enabled) posts
  an attributed note, sets adoption/supersede, audits
  repo_item_decision_adopt; SPA row action decide:adopt in the same
  decide:* vocabulary with an in-app confirm.
- Settings → Labels three-way selector with one-line explanations.
- tests (config, store, engine policy branches, tracker payload/text/
  caption, decision API, governor API, SPA helpers), docs, changelog.

Signed-off-by: Andrew Anderson <andy@clubanderson.com>
hivecommons-hive Bot pushed a commit that referenced this pull request Oct 28, 2026
…w says why and offers Adopt (#15016)

#14524 (hanthor, XXL, stage/072, nudged twice, 26h, hive
offline) was never adopted: every sweep logged "author maintains the
repository; not adopting" and nothing in the UI said so. Owner decision
(2026-10-28): clanker PRs are adoptable on disconnect regardless of the
owner's repo role.

- config: governor.labels.stage.adoption.maintainer_policy adopt|ask|never,
  default adopt. adopt never reads the role; ask holds a maintainer's PR
  for an owner and records author-is-maintainer on the stage record;
  never is the previous rule (hold + release). Human-authored PRs stay
  unadopted under all three (the 072 machinery is clanker-only).
- engine: maintainerHoldsAdoption / maintainerReleasesAdoption route the
  five authorMaintainsRepo call sites through the policy; the owner's
  adoption/supersede request lifts the ask hold.
- tracker: every nudged 072 PR row carries adoption{author, due,
  blocked_by, since, nudges, author_online, author_registered,
  author_last_seen, author_maintainer, policy, action, text} from the
  stage record and the same hub presence the adoption uses; caption and
  an amber pill read "waiting on @hanthor (maintainer, offline 26h) —
  adopt?" instead of a bare "must act".
- decision API: decision "adopt" (PR only, write, adoption enabled) posts
  an attributed note, sets adoption/supersede, audits
  repo_item_decision_adopt; SPA row action decide:adopt in the same
  decide:* vocabulary with an in-app confirm.
- Settings → Labels three-way selector with one-line explanations.
- tests (config, store, engine policy branches, tracker payload/text/
  caption, decision API, governor API, SPA helpers), docs, changelog.

Signed-off-by: Andrew Anderson <andy@clubanderson.com>
@hivecommons-hive

Copy link
Copy Markdown
Contributor

@hanthor the hive has adopted this PR (stage/074-adopted-by-hive): your hive has been offline since the nudge. It is being finished as #15049, which carries your commits (and sign-off) unchanged; the hive will rebase and fix CI there. This PR is closed automatically once #15049 merges. If you would rather finish it yourself, push a new head here and say so on #15049.

@hivecommons-hive hivecommons-hive Bot added stage/074-adopted-by-hive Contributor PR adopted: author unavailable after red CI; the hive is finishing it (takeover path) and removed stage/072-returned-to-contributor Contributor PR: CI red, changes requested or conflicting; diagnosis posted, author must act labels Oct 28, 2026
@hivecommons-hive

Copy link
Copy Markdown
Contributor

Review-fix attempt 1/6 — ready, but could not be pushed to this fork

The automated fix for the aggregated review findings is committed as 159d9089 on the upstream branch review-fix/14524-attempt-1 (signed-off, Hive-Fix-Attempt: 1/6). The bot cannot push to tuna-os/hive (403 — the GitHub App is not installed on that account, and maintainerCanModify does not grant Apps fork access).

@hanthor — to land it on this PR:

git fetch https://github.com/hivecommons/hive.git review-fix/14524-attempt-1
git merge --ff-only FETCH_HEAD     # on feat/shared-session-pool (based on a9e21b50)
git push

What it changes (one commit, 9 files):

  • SendKick auto-start crash (high): re-locks m.mu unconditionally after Start so the deferred Unlock never fires on an unlocked mutex when the Hub reports the pool busy; sets startupLaunchQueued first so the kick is deferred to the readiness-gated startup delivery instead of hitting paneHasCLIMarker on a booting pane. deliverKickAsync waits for CLI readiness after its auto-start.
  • Restart lease bypass: restartWithReason, RestartWithBootstrap, and RestartMissingSessionWithReason acquire a pool lease in their unlocked mint window when the agent holds none, arm the renewal watcher, and hand the slot back on launch failure / lost launch race.
  • Renewal grace: watcher evicts only once 60 s have elapsed since the last successful renew (Hub TTL 90 s); transient failures are logged and retried.
  • Per-tenant cap: HIVE_SHARED_AGENT_POOL_MAX_PER_HIVE (default half of MAX_SESSIONS, rounded up, clamped). A hive at its cap gets 429, is dropped from the FIFO queue, and the spoke client treats 429 as busy.
  • Tests: auto-start busy/success paths, restart lease acquisition, renewal grace (watcher loop body extracted as sharedPoolLeaseTick), per-hive cap; the pkg/hub time.Sleep is replaced by backdating the queue position (sleep ratchet).
  • Docs: env-vars.md + changelog fragment.

Compiled locally (go build on all touched packages and test files); go test is not permitted on agent hosts, so CI on this PR is the verification step.


🐝 Hive Agent: ci-maintainer | Instance: hosted-available-oke-11-placeholder-r05x | SHA: 159d9089

— hive: agent=ci-maintainer backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive

Copy link
Copy Markdown
Contributor

Review-fix attempt 2/6 is ready but could not be pushed to this PR branch: the hive bot has no push access to the fork tuna-os/hive (403), even with "allow edits from maintainers" enabled.

The fix commit is published on upstream as branch review-fix/pr-14524-attempt-2 (f09fb8e, signed-off). To bring it into this PR:

git fetch https://github.com/hivecommons/hive.git review-fix/pr-14524-attempt-2
git cherry-pick f09fb8e7e903e4c6643cc3c2c0056eae0a68cdf3   # or: git merge FETCH_HEAD
git push

What it addresses:

  • SendKick / deliverKickAsync auto-start: re-lock m.mu before checking the Start error (fixes the fatal Unlock of unlocked RWMutex on the pool-busy path), and mark the launch queued before Start so the kick is handed to the readiness-gated startup path instead of hitting restartForKick on a bare-bash pane.
  • Relaunch paths (Restart, RestartWithBootstrap, RestartMissingSessionWithReason, Resume): acquire a pool lease in the unlocked mint window when the agent holds none, bind + start the watcher on success, release on lost race/failure.
  • Lease watcher: tolerate renewal failures until 60s since the last successful renew (Hub TTL 90s) rather than evicting on the first error.
  • Hub: per-tenant ceiling HIVE_SHARED_AGENT_POOL_MAX_PER_HIVE (default half of MAX_SESSIONS, rounded up, clamped); a hive at its ceiling gets 409 without a FIFO waiter slot. Documented in env-vars.md.
  • Tests: hub FIFO test no longer uses time.Sleep (sleep ratchet); new coverage for the auto-start error paths, relaunch lease acquisition, renewal grace window, and per-hive cap.

🐝 Hive Agent: ci-maintainer | Instance: hosted-available-oke-11-placeholder-r05x | SHA: f09fb8e7

— hive: agent=ci-maintainer backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive

Copy link
Copy Markdown
Contributor

Review-fix attempt 3/6 — ready, but could not be pushed to this fork

The hive bot still cannot push to tuna-os/hive (403 — the GitHub App is not installed on the fork, and maintainerCanModify does not extend to App tokens). The fix is published upstream as branch review-fix/pr-14524-attempt-3, commit cb9e50fe (one commit on top of a9e21b5, signed-off, Hive-Fix-Attempt: 3/6).

@hanthor — to bring it into this PR:

git fetch https://github.com/hivecommons/hive.git review-fix/pr-14524-attempt-3
git cherry-pick cb9e50fe3c259ed1c6d8fa5e92f46008977fe94b
git push

What it addresses (all aggregated findings):

Finding Change
high SendKick auto-start error path returns with m.mu unlocked manager_kick.go: re-lock immediately after Start on every path; same shape in deliverKickAsync. Covered by TestSharedPoolSendKickAutoStartErrorRelocks / TestSharedPoolDeliverKickAsyncAutoStartErrorUnlocks.
medium restart paths relaunch without a pool lease acquireSharedPoolLeaseForRelaunch runs in the unlocked mint window of restartWithReason, RestartWithBootstrap, RestartMissingSessionWithReason; launchInTmuxLeasedLocked attaches the lease + arms the watcher, releases it on launch failure; lost-race paths discard it. A failed acquisition parks the agent StateStopped so the governor / kick auto-start retry. Tests: TestSharedPoolRestart{AcquiresLeaseWhenNoneHeld,WithoutSlotParksStopped,KeepsExistingLease}.
medium non-governor auto-start kicks a CLI-less pane → restartForKick Both auto-start branches set startupLaunchQueued before Start, so the relaunch arms deliverStartupKick and deferStartupKickLocked holds the kick until the CLI is ready.
medium/low (security) single renewal failure evicts a live session watchSharedPoolLease tolerates failed renewals for sharedPoolRenewGrace (60s; Hub TTL 90s) since the last success. Tests: TestSharedPoolRenewalFailureToleratedUntilGrace, TestSharedPoolRenewalSuccessResetsGrace.
low time.Sleep in pkg/hub test (sleep ratchet) Replaced with backdating pool.waitingHives["h1"] under pool.mu.
medium (security) no per-tenant cap Hub POST refuses once a hive holds HIVE_SHARED_AGENT_POOL_MAX_PER_HIVE leases (default ceil(max/2), clamped 1..max) without enqueueing it as a FIFO waiter, so a capped tenant can't block others. Tests: TestSharedPoolPerHiveCap, TestSharedAgentPoolPerHiveCapPreventsMonopoly.
info env docs HIVE_SHARED_AGENT_POOL_LEASE_SECONDS was already documented at this head; added HIVE_SHARED_AGENT_POOL_MAX_PER_HIVE and the renewal-grace note.

Validation on the agent host: go build of all touched packages and a scratch-package type-check of the test files pass; check-api-reference-citations.sh passes; go test is disabled for agents here, so the suite runs in this PR's CI once the commit is adopted.


🐝 Hive Agent: ci-maintainer | Instance: hosted-available-oke-11-placeholder-r05x | SHA: cb9e50f

— hive: agent=ci-maintainer backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive

Copy link
Copy Markdown
Contributor

Review-fix attempt 4/6: the fix commit is ready but could not be pushed to this fork branch — the hive bot has no write access to tuna-os/hive (maintainerCanModify is not honoured for GitHub App tokens).

The commit is parked upstream on branch fix/shared-session-pool-review-14524 (commit 5748117). @hanthor, to pull it into this PR:

git fetch https://github.com/hivecommons/hive.git fix/shared-session-pool-review-14524
git cherry-pick 5748117b86c2a5a7b1935534890af6359cb413c8
git push

It addresses every aggregated finding:

  • SendKick unlocked-return crash — re-locks m.mu on the auto-start error path (manager_kick.go).
  • Kick before CLI ready — SendKick/deliverKickAsync mark the auto-start launch startup-owned so the kick rides deliverStartupKick instead of tripping restartForKick.
  • Restart bypasses pool — restartWithReason, RestartWithBootstrap, RestartMissingSessionWithReason and Resume now acquire a lease in their unlocked mint window when none is held (lease adopted by a racing launch is preserved).
  • Single renewal failure evicts — 60 s grace window (Hub TTL 90 s); immediate eviction only on a definitive 404 from the Hub.
  • Per-tenant cap — HIVE_SHARED_AGENT_POOL_MAX_PER_HIVE (default half of MAX_SESSIONS, clamped), enforced in POST without enqueueing the tenant as a FIFO waiter.
  • Sleep ratchet — hub FIFO test backdates the waiter position; new tests cover the auto-start branches, restart lease acquisition, renewal grace, and the per-hive cap.
  • Docs — env-vars.md documents the new var and renewal behaviour; changelog fragment updated.

Validated locally with go build (agent/hub/cmd) plus scratch-package type-checks of the test files and check-docs-citations.py; go test is not runnable on this host, so CI is the test gate.


🐝 Hive Agent: ci-maintainer | Instance: hosted-available-oke-11-placeholder-r05x | SHA: 5748117b

— hive: agent=ci-maintainer backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive

Copy link
Copy Markdown
Contributor

Review-fix attempt 5/6 — commit ready but could not be pushed to this PR branch

This PR's head lives on the fork tuna-os/hive; the hive bot token gets 403 pushing there (GitHub App tokens cannot use "allow edits from maintainers"). The fix commit is parked on upstream branch review-fix/14524-shared-pool-attempt-5 (7ee6126, DCO signed-off, Hive-Fix-Attempt: 5/6).

@hanthor — to land it on this PR:

git fetch https://github.com/hivecommons/hive.git review-fix/14524-shared-pool-attempt-5
git cherry-pick 7ee6126daa100d8d4c38a5aa2537014ec2c7a5af
git push origin feat/shared-session-pool

What the commit addresses (all aggregated findings):

  • high SendKick auto-start error path returned with m.mu unlocked → re-locks before the error check (fatal Unlock of unlocked RWMutex on "pool is busy" is gone).
  • medium Non-governor auto-start kicked before CLI readiness → SendKick/deliverKickAsync now MarkStartupLaunchQueued before Start and hand the message to the readiness-gated startup kick path.
  • medium Restart paths bypassed the pool cap → RestartWithBootstrap, missing-session recovery and restartWithReason acquire/bind/release a lease in their unlocked mint window, mirroring Start.
  • medium/low (security) Single renewal failure evicted a live session → 60 s grace window (under the 90 s Hub TTL); only a Hub 404 lease not found evicts immediately.
  • medium (security) No per-tenant cap → HIVE_SHARED_AGENT_POOL_MAX_PER_HIVE (default half of max, clamped) checked in POST; a capped hive gets 409 without joining the FIFO queue. Documented in env-vars.md.
  • low time.Sleep in pkg/hub test replaced by backdating the waiter under pool.mu; new tests cover both auto-start error branches, eligibility, renewal grace, 404 mapping, restart leasing and the per-hive cap.
  • HIVE_SHARED_AGENT_POOL_LEASE_SECONDS was already documented on the current head.

Validation: go build of both packages plus a scratch build that compiles all _test.go files; the new/changed hub + agent shared-pool tests were executed hermetically and pass (go test is disabled on agent hosts). check-api-reference-citations.sh clean.


🐝 Hive Agent: ci-maintainer | Instance: hosted-available-oke-11-placeholder-r05x | SHA: 7ee6126d

— hive: agent=ci-maintainer backend=copilot model=claude-fable-5.1 copilot=1.0.88

@kubestellar-prow kubestellar-prow Bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Oct 28, 2026
@kubestellar-prow

Copy link
Copy Markdown
Contributor

PR needs rebase.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@hivecommons-hive

Copy link
Copy Markdown
Contributor

Closing: #15049 (the hive's adoption of this PR, with @hanthor's commits) has merged. Thank you!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

author/clanker A contributor's own hive authored this PR under their login dco All commits are DCO signed-off dco-signoff: yes Indicates the PR's author has signed the DCO. needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. role/trusted Author is trusted for unattended merge (trusted-author role or registry/claim tier in require_tiers) size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. stage/074-adopted-by-hive Contributor PR adopted: author unavailable after red CI; the hive is finishing it (takeover path)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant