Repository navigation
✨ feat(hub,agent): Hub-coordinated live CLI session pool for multi-tenant runner capacity - #15049
Conversation
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
Open PR overlap checkBase branch: Conflicts are checked with every PR merged onto the current tip of
|
…nt runner capacity Introduce an opt-in Hub-coordinated session lease pool so multiple Hive spokes sharing constrained runner hosts can serialize heavyweight CLI processes without sharing filesystem trees, tmux sessions, or credentials. - Add /api/shared-agent-pool/leases on Hub backed by per-Hive HMAC auth, lease persistence across restarts, and a single-waiter FIFO queue. - Add WithSharedSessionPool runtime and Manager option to acquire a lease before launch and evict idle sessions back to stopped state post-turn. - Add HIVE_SHARED_AGENT_POOL_ENABLED and HIVE_SHARED_AGENT_POOL_MAX_SESSIONS environment controls. - Add comprehensive regression and HTTP integration tests across hub and agent. Signed-off-by: James Reilly <jreilly1821@gmail.com>
…bled Signed-off-by: James Reilly <jreilly1821@gmail.com>
…chet, and api-reference citations Signed-off-by: James Reilly <jreilly1821@gmail.com>
…is full Signed-off-by: James Reilly <jreilly1821@gmail.com>
…st sleep cmd/hive named crux/agent directly for NewSharedSessionPoolClient, which TestCruxBoundaryRatchet counts as new consumer coupling. Re-export the SharedSessionPool contract and constructor from crux/runtime so the host configures the pool through the runtime seam. The hub FIFO-waiter test slept 10ms to order two 409s; back-date the first waiter's recorded timestamp under the pool mutex instead, keeping TestSleepRatchet at its baseline. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: scanner <280983584+hivecommons-hive[bot]@users.noreply.github.com>
pkg/hub's pool test named crux/agent directly; the boundary ratchet counts test references too. Use the crux/runtime re-export instead. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: scanner <280983584+hivecommons-hive[bot]@users.noreply.github.com>
… blips, cap per-hive leases Review fixes for the shared CLI session pool: - SendKick's auto-start branch unlocked m.mu, called Start, and returned on error without re-locking, so the deferred Unlock hit an unlocked mutex (runtime fatal) whenever Acquire timed out on a busy pool. Re-lock before every return. - restartWithReason, RestartMissingSessionWithReason, RestartWithBootstrap and Resume relaunched via launchInTmux without a pool lease, so an evicted agent restarted from the dashboard or recovery ran unleased and uncapped. Acquire in the existing unlocked mint window, bind the lease and start its watcher on success, and return a fresh lease to the Hub when the launch does not happen. - watchSharedPoolLease evicted a live turn on the first Renew error. Tolerate transient failures for 60s after the last successful renewal (three missed ticks, still inside the 90s Hub TTL); a 404 from the Hub (lease reclaimed) still evicts immediately. - Add HIVE_SHARED_AGENT_POOL_MAX_PER_HIVE so a single tenant cannot hold every slot and renew it forever; a Hive at its cap gets 409 without taking a FIFO waiter position that would block other tenants. Hive-Fix-Attempt: 1/6 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: ci-maintainer <280983584+hivecommons-hive[bot]@users.noreply.github.com>
9c4744d to
80a8f65
Compare
|
hive/awaiting-ci: rebased onto current 🐝 Hive Agent: — hive: agent=scanner backend=copilot model=claude-fable-5.1 copilot=1.0.88 |
There was a problem hiding this comment.
Read against v5 at head 80a8f65: this looks correct to me, and the adoption commit does fix what it says it fixes. Not approving — that is a human's signature — but nothing here needs a change from my read.
What I verified (hive-authored supersede of #14524; the body carries #14524's changes_requested forward and 80a8f65 is the fix commit):
- SendKick unlock/return bug —
src/crux/agent/manager_kick.go:131-140: the auto-start branch now re-locksm.mubefore checkingerr, so the deferredUnlockalways pairs with a held mutex. On v5-era #14524 the error return happened while unlocked. OperatorStoppedguard inStart—manager.go:1199returns "stopped during launch", butStartclears the flag at entry (manager.go:991), so a dashboard Start of an operator-stopped agent still works; the guard only catches aStopracing the unlocked tmux-launch window.- Renew blips —
src/crux/agent/shared_pool.go:151-197: renew every 20 s, eviction only whensharedPoolRenewFailureEvictssays so (60 s since last success, or a 404). Fits inside the Hub's 90 s TTL (src/pkg/hub/shared_agent_pool.go:20). - Hub side —
shared_agent_pool.go:164-285: bearer is verified against the request'shive_id(verifyHeartbeatBearer), body is capped at 4 KB, lease IDs are 16 random bytes; a tenant atHIVE_SHARED_AGENT_POOL_MAX_PER_HIVEgets 409 without taking a FIFO waiter slot (:220-227); stale waiters are pruned after 15 s of silence (:139-150), which closes the "waiter never expires" finding from the #14524 review. - Routes are registered as method-scoped patterns on the existing mux (
src/pkg/hub/server.go:1704-1710); no change to existing endpoints.
Housekeeping for a maintainer: #14524 (fork tuna-os/hive, changes_requested) is the superseded original. Merging this one makes #14524 a no-op; it should be closed then, not merged alongside.
— hive: agent=reviewer-2 backend=copilot model=claude-fable-5.1 copilot=1.0.88
There was a problem hiding this comment.
The carried-over finding from #14524 is addressed at this head; the two red checks are the diff's own and are small.
Verified (read src/pkg/hub/shared_agent_pool.go at 80a8f65):
- The swarm's
changes_requestedon #14524 —waitingHivesnever expired, so one stale tenant 409'd every other hive until Hub restart — is fixed.hasEarlierWaitingHivenow prunes any waiter whoselastSeenis older thansharedPoolWaiterStaleAfterbefore computing FIFO position, andhandlerefresheslastSeenon every 409.TestSharedAgentPool…covers a staleh2being pruned soh3acquires.
Two things keeping CI red, both in this PR's own code:
- golangci-lint —
src/crux/agent/shared_pool_test.go:196:m.mu.Lock(); m.mu.Unlock()isSA2001: empty critical section. The intent (proveSendKick's error path releasedm.mu) is sound;if !m.mu.TryLock() { t.Fatal("m.mu still held after SendKick error") }; m.mu.Unlock()expresses it without the lint hit and gives a real failure instead of a hang. - coverage —
crux/agent: 88.6% (floor 89%), patch coverage 67.0% against the 80% threshold. The newshared_pool.golease/renew/evict paths insrc/crux/agentare where the uncovered lines are; the floor is per-package so a couple of tests on the renewal-failure branches should clear it.
Scope note for the maintainer: this is the hive's supersede of #14524 (same head lineage, @hanthor's commits preserved), so it is not a duplicate to close — #14524 is the one that goes away when this lands.
I read the Hub pool and the agent-side test; I did not re-read the full 1.9k-line diff (manager_*.go, runtime.go) beyond the lease ordering, so treat this as a targeted follow-up to the #14524 review rather than a fresh full pass.
— hive: agent=reviewer backend=copilot model=claude-fable-5.1 copilot=1.0.88
…balance probe Replace the empty critical section (SA2001) in the SendKick lock-balance test with a TryLock assertion so the test fails instead of hanging when the lock leaks, and golangci-lint passes. Make the lease watcher tick interval a package var (same pattern as agyTurnStatusInterval) so tests can drive watchSharedPoolLease: successful renewals mid-turn, transient renewal failures tolerated within the grace window, immediate eviction on ErrSharedSessionLeaseNotFound, exit when the lease is superseded or the agent is gone, and idle agents without a visible prompt. Also cover the eviction outcome guards, releaseSharedPoolLease, acquireSharedPoolLeaseUnlocked/abandonFreshSharedPoolLeaseLocked, and the Hub client's Acquire busy-retry, deadline cancel_wait, error, Renew and Release branches. Hive-Fix-Attempt: 2/6 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: ci-maintainer <280983584+hivecommons-hive[bot]@users.noreply.github.com>
There was a problem hiding this comment.
Re-reviewed at head 5a9b263. The four findings the swarm raised on #14524 / the previous head a9e21b5 are each fixed in the tree; this looks correct to me now.
What I verified (read, not inferred):
- SendKick lock balance —
src/crux/agent/manager_kick.go:125-134now re-locksm.mubefore every return after the auto-startStart()call, so the deferredUnlockat :113 always pairs with a held mutex. - Restart paths are leased — all three relaunch sites in
src/crux/agent/manager_restart.go(:499, :794, :953) go throughacquireSharedPoolLeaseUnlocked→launchInTmuxLeasedLocked; a dashboard/recovery restart can no longer run uncapped. - Renewal blips —
src/crux/agent/shared_pool.go:18-26,204-212: a failed renew only evicts aftersharedPoolRenewGrace(60s, three missed 20s ticks) since the last success, or immediately on a definitive 404. One hub blip no longer kills an in-flight turn. - Per-tenant cap —
src/pkg/hub/shared_agent_pool.go:124-134,219-227:HIVE_SHARED_AGENT_POOL_MAX_PER_HIVEcaps leases perhive_id, and a hive at its cap gets 409 without taking a FIFO slot. Stale waiters are pruned inhasEarlierWaitingHive(:146-150) vialastSeen, which also closes the "waiter never expires" hole. - Sleep ratchet — zero
time.Sleep(in the diff's_test.gofiles;sharedPoolRenewIntervalis a var for fast ticking instead. - Docs —
src/docs/env-vars.mddocuments all four pool env vars including the new per-hive cap and the renewal-grace semantics.
Open item (not a code finding): build-hub (linux/arm64) failed at the "Build and push hub image by digest" step on this head while every Go test shard, lint, coverage and the amd64 hub build passed. That step has no Go source input beyond what amd64 already built, so it reads as a runner/registry failure — a re-run should clear it. I have not approved or merged anything.
— hive: agent=reviewer-2 backend=copilot model=claude-fable-5.1 copilot=1.0.88
|
Thank you for your contribution! Your PR has been merged. We'd love to hear how your experience was: share feedback |
Supersedes #14524 (adopted after CI went red and the contributor was unavailable; original commits by @hanthor preserved).
This branch is #14524's head
a9e21b50ce65as pushed by @hanthor (forktuna-os/hive), unchanged. Last lane reason on the original:conflicting.Hive fix agent: rebase this branch on
v5, fix the failing checks / review asks, and push to THIS branch. Keep the original author's commits and theirSigned-off-by; add your own sign-off on your commits. Do not open another PR.Open review finding carried over from #14524: the review swarm's latest verdict on
a9e21b50ce65requested changes (changes_requested); read the swarm's review on #14524, address it here, and let the swarm re-review this PR. Green CI alone is not done.