Skip to content

✨ feat(hub,agent): Hub-coordinated live CLI session pool for multi-tenant runner capacity - #15049

Merged
hivecommons-hive[bot] merged 8 commits into
v5from
hive/adopt-14524
Oct 28, 2026
Merged

hivecommons-hive[bot] merged 8 commits into
v5from
hive/adopt-14524

Conversation

@hivecommons-hive

Copy link
Copy Markdown
Contributor

Supersedes #14524 (adopted after CI went red and the contributor was unavailable; original commits by @hanthor preserved).

This branch is #14524's head a9e21b50ce65 as pushed by @hanthor (fork tuna-os/hive), unchanged. Last lane reason on the original: conflicting.

Hive fix agent: rebase this branch on v5, fix the failing checks / review asks, and push to THIS branch. Keep the original author's commits and their Signed-off-by; add your own sign-off on your commits. Do not open another PR.

Open review finding carried over from #14524: the review swarm's latest verdict on a9e21b50ce65 requested changes (changes_requested); read the swarm's review on #14524, address it here, and let the swarm re-review this PR. Green CI alone is not done.

@kubestellar-prow

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign hanthor for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@kubestellar-prow kubestellar-prow Bot added needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Oct 28, 2026
@hivecommons-hive hivecommons-hive Bot added author/hive This repository's own hive (the App or one of its agent lanes) authored this PR dco All commits are DCO signed-off role/trusted Author is trusted for unattended merge (trusted-author role or registry/claim tier in require_tiers) stage/070-fixing Hive-authored PR: CI red, changes requested or needs rebase; a hive agent is fixing it labels Oct 28, 2026
@github-actions

github-actions Bot commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

Open PR overlap check

Base branch: v5

Conflicts are checked with every PR merged onto the current tip of v5, so a PR that only needs a rebase is not reported as conflicting with its siblings.

PR Title Relation Files
#15081 🐛 fix: explain Activity Monitor merge gates and sweep timing same files src/cmd/hive/main.go
#15085 ✨ Activity Monitor: Adopt now and Reroute to another clanker for PRs waiting on a contributor's clanker same files src/cmd/hive/main.go
#14524 ✨ feat(hub,agent): Hub-coordinated live CLI session pool for multi-tenant runner capacity same files changelog.d/added-shared-agent-session-pool.md
src/cmd/hive/eval_cycle_seams.go
src/cmd/hive/main.go
src/cmd/hive/shared_pool.go
src/cmd/hive/shared_pool_test.go
src/crux/agent/kick_async.go
src/crux/agent/kick_outcome.go
src/crux/agent/manager.go
src/crux/agent/manager_kick.go
src/crux/agent/shared_pool.go
src/crux/agent/shared_pool_test.go
src/crux/runtime/runtime.go
src/crux/runtime/runtime_test.go
src/docs/api-reference.md
src/docs/env-vars.md
src/pkg/hub/hub_testing_helpers_test.go
src/pkg/hub/server.go
src/pkg/hub/shared_agent_pool.go
src/pkg/hub/shared_agent_pool_test.go
#14807 ✨ dashboard: stream Job pod logs in the terminal pane for sandbox.runtime: job agents same files src/docs/api-reference.md
#14921 ✨ feature: inline the linked Linear issue into review kicks same files src/cmd/hive/main.go

hanthor and others added 7 commits October 28, 2026 15:31
…nt runner capacity

Introduce an opt-in Hub-coordinated session lease pool so multiple Hive
spokes sharing constrained runner hosts can serialize heavyweight CLI
processes without sharing filesystem trees, tmux sessions, or credentials.

- Add /api/shared-agent-pool/leases on Hub backed by per-Hive HMAC auth,
  lease persistence across restarts, and a single-waiter FIFO queue.
- Add WithSharedSessionPool runtime and Manager option to acquire a lease
  before launch and evict idle sessions back to stopped state post-turn.
- Add HIVE_SHARED_AGENT_POOL_ENABLED and HIVE_SHARED_AGENT_POOL_MAX_SESSIONS
  environment controls.
- Add comprehensive regression and HTTP integration tests across hub and agent.

Signed-off-by: James Reilly <jreilly1821@gmail.com>
…bled

Signed-off-by: James Reilly <jreilly1821@gmail.com>
…chet, and api-reference citations

Signed-off-by: James Reilly <jreilly1821@gmail.com>
…is full

Signed-off-by: James Reilly <jreilly1821@gmail.com>
…st sleep

cmd/hive named crux/agent directly for NewSharedSessionPoolClient, which
TestCruxBoundaryRatchet counts as new consumer coupling. Re-export the
SharedSessionPool contract and constructor from crux/runtime so the host
configures the pool through the runtime seam.

The hub FIFO-waiter test slept 10ms to order two 409s; back-date the first
waiter's recorded timestamp under the pool mutex instead, keeping
TestSleepRatchet at its baseline.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: scanner <280983584+hivecommons-hive[bot]@users.noreply.github.com>
pkg/hub's pool test named crux/agent directly; the boundary ratchet counts
test references too. Use the crux/runtime re-export instead.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: scanner <280983584+hivecommons-hive[bot]@users.noreply.github.com>
… blips, cap per-hive leases

Review fixes for the shared CLI session pool:

- SendKick's auto-start branch unlocked m.mu, called Start, and returned
  on error without re-locking, so the deferred Unlock hit an unlocked
  mutex (runtime fatal) whenever Acquire timed out on a busy pool.
  Re-lock before every return.
- restartWithReason, RestartMissingSessionWithReason, RestartWithBootstrap
  and Resume relaunched via launchInTmux without a pool lease, so an
  evicted agent restarted from the dashboard or recovery ran unleased and
  uncapped. Acquire in the existing unlocked mint window, bind the lease
  and start its watcher on success, and return a fresh lease to the Hub
  when the launch does not happen.
- watchSharedPoolLease evicted a live turn on the first Renew error.
  Tolerate transient failures for 60s after the last successful renewal
  (three missed ticks, still inside the 90s Hub TTL); a 404 from the Hub
  (lease reclaimed) still evicts immediately.
- Add HIVE_SHARED_AGENT_POOL_MAX_PER_HIVE so a single tenant cannot hold
  every slot and renew it forever; a Hive at its cap gets 409 without
  taking a FIFO waiter position that would block other tenants.

Hive-Fix-Attempt: 1/6
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: ci-maintainer <280983584+hivecommons-hive[bot]@users.noreply.github.com>
@kubestellar-prow kubestellar-prow Bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Oct 28, 2026
@hivecommons-hive

Copy link
Copy Markdown
Contributor Author

hive/awaiting-ci: rebased onto current v5 (only conflict was citation line-drift in src/docs/api-reference.md; resolved by keeping v5 rows, re-adding the shared-agent-pool routes and re-running check-api-reference-citations.sh --fix). CI pending — sweep will merge when green.


🐝 Hive Agent: scanner | Instance: hosted-available-oke-11-placeholder-r05x | SHA: 9be2fee41

— hive: agent=scanner backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive hivecommons-hive Bot added stage/050-review-pending Review swarm requested; awaiting a verdict for the current head stage/070-fixing Hive-authored PR: CI red, changes requested or needs rebase; a hive agent is fixing it and removed stage/070-fixing Hive-authored PR: CI red, changes requested or needs rebase; a hive agent is fixing it stage/050-review-pending Review swarm requested; awaiting a verdict for the current head labels Oct 28, 2026
@hivecommons-hive hivecommons-hive Bot added stage/050-review-pending Review swarm requested; awaiting a verdict for the current head stage/070-fixing Hive-authored PR: CI red, changes requested or needs rebase; a hive agent is fixing it and removed stage/050-review-pending Review swarm requested; awaiting a verdict for the current head stage/070-fixing Hive-authored PR: CI red, changes requested or needs rebase; a hive agent is fixing it labels Oct 28, 2026

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Read against v5 at head 80a8f65: this looks correct to me, and the adoption commit does fix what it says it fixes. Not approving — that is a human's signature — but nothing here needs a change from my read.

What I verified (hive-authored supersede of #14524; the body carries #14524's changes_requested forward and 80a8f65 is the fix commit):

  • SendKick unlock/return bug — src/crux/agent/manager_kick.go:131-140: the auto-start branch now re-locks m.mu before checking err, so the deferred Unlock always pairs with a held mutex. On v5-era #14524 the error return happened while unlocked.
  • OperatorStopped guard in Start — manager.go:1199 returns "stopped during launch", but Start clears the flag at entry (manager.go:991), so a dashboard Start of an operator-stopped agent still works; the guard only catches a Stop racing the unlocked tmux-launch window.
  • Renew blips — src/crux/agent/shared_pool.go:151-197: renew every 20 s, eviction only when sharedPoolRenewFailureEvicts says so (60 s since last success, or a 404). Fits inside the Hub's 90 s TTL (src/pkg/hub/shared_agent_pool.go:20).
  • Hub side — shared_agent_pool.go:164-285: bearer is verified against the request's hive_id (verifyHeartbeatBearer), body is capped at 4 KB, lease IDs are 16 random bytes; a tenant at HIVE_SHARED_AGENT_POOL_MAX_PER_HIVE gets 409 without taking a FIFO waiter slot (:220-227); stale waiters are pruned after 15 s of silence (:139-150), which closes the "waiter never expires" finding from the #14524 review.
  • Routes are registered as method-scoped patterns on the existing mux (src/pkg/hub/server.go:1704-1710); no change to existing endpoints.

Housekeeping for a maintainer: #14524 (fork tuna-os/hive, changes_requested) is the superseded original. Merging this one makes #14524 a no-op; it should be closed then, not merged alongside.

— hive: agent=reviewer-2 backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The carried-over finding from #14524 is addressed at this head; the two red checks are the diff's own and are small.

Verified (read src/pkg/hub/shared_agent_pool.go at 80a8f65):

  • The swarm's changes_requested on #14524 — waitingHives never expired, so one stale tenant 409'd every other hive until Hub restart — is fixed. hasEarlierWaitingHive now prunes any waiter whose lastSeen is older than sharedPoolWaiterStaleAfter before computing FIFO position, and handle refreshes lastSeen on every 409. TestSharedAgentPool… covers a stale h2 being pruned so h3 acquires.

Two things keeping CI red, both in this PR's own code:

  1. golangci-lint — src/crux/agent/shared_pool_test.go:196: m.mu.Lock(); m.mu.Unlock() is SA2001: empty critical section. The intent (prove SendKick's error path released m.mu) is sound; if !m.mu.TryLock() { t.Fatal("m.mu still held after SendKick error") }; m.mu.Unlock() expresses it without the lint hit and gives a real failure instead of a hang.
  2. coverage — crux/agent: 88.6% (floor 89%), patch coverage 67.0% against the 80% threshold. The new shared_pool.go lease/renew/evict paths in src/crux/agent are where the uncovered lines are; the floor is per-package so a couple of tests on the renewal-failure branches should clear it.

Scope note for the maintainer: this is the hive's supersede of #14524 (same head lineage, @hanthor's commits preserved), so it is not a duplicate to close — #14524 is the one that goes away when this lands.

I read the Hub pool and the agent-side test; I did not re-read the full 1.9k-line diff (manager_*.go, runtime.go) beyond the lease ordering, so treat this as a targeted follow-up to the #14524 review rather than a fresh full pass.

— hive: agent=reviewer backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive hivecommons-hive Bot added stage/070-fixing Hive-authored PR: CI red, changes requested or needs rebase; a hive agent is fixing it stage/050-review-pending Review swarm requested; awaiting a verdict for the current head and removed stage/050-review-pending Review swarm requested; awaiting a verdict for the current head stage/070-fixing Hive-authored PR: CI red, changes requested or needs rebase; a hive agent is fixing it labels Oct 28, 2026
@hivecommons-hive hivecommons-hive Bot added stage/070-fixing Hive-authored PR: CI red, changes requested or needs rebase; a hive agent is fixing it and removed stage/050-review-pending Review swarm requested; awaiting a verdict for the current head labels Oct 28, 2026
…balance probe

Replace the empty critical section (SA2001) in the SendKick lock-balance
test with a TryLock assertion so the test fails instead of hanging when
the lock leaks, and golangci-lint passes.

Make the lease watcher tick interval a package var (same pattern as
agyTurnStatusInterval) so tests can drive watchSharedPoolLease: successful
renewals mid-turn, transient renewal failures tolerated within the grace
window, immediate eviction on ErrSharedSessionLeaseNotFound, exit when the
lease is superseded or the agent is gone, and idle agents without a visible
prompt. Also cover the eviction outcome guards, releaseSharedPoolLease,
acquireSharedPoolLeaseUnlocked/abandonFreshSharedPoolLeaseLocked, and the
Hub client's Acquire busy-retry, deadline cancel_wait, error, Renew and
Release branches.

Hive-Fix-Attempt: 2/6
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: ci-maintainer <280983584+hivecommons-hive[bot]@users.noreply.github.com>
@hivecommons-hive hivecommons-hive Bot added stage/050-review-pending Review swarm requested; awaiting a verdict for the current head stage/070-fixing Hive-authored PR: CI red, changes requested or needs rebase; a hive agent is fixing it and removed stage/070-fixing Hive-authored PR: CI red, changes requested or needs rebase; a hive agent is fixing it stage/050-review-pending Review swarm requested; awaiting a verdict for the current head labels Oct 28, 2026

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed at head 5a9b263. The four findings the swarm raised on #14524 / the previous head a9e21b5 are each fixed in the tree; this looks correct to me now.

What I verified (read, not inferred):

  • SendKick lock balance — src/crux/agent/manager_kick.go:125-134 now re-locks m.mu before every return after the auto-start Start() call, so the deferred Unlock at :113 always pairs with a held mutex.
  • Restart paths are leased — all three relaunch sites in src/crux/agent/manager_restart.go (:499, :794, :953) go through acquireSharedPoolLeaseUnlocked → launchInTmuxLeasedLocked; a dashboard/recovery restart can no longer run uncapped.
  • Renewal blips — src/crux/agent/shared_pool.go:18-26,204-212: a failed renew only evicts after sharedPoolRenewGrace (60s, three missed 20s ticks) since the last success, or immediately on a definitive 404. One hub blip no longer kills an in-flight turn.
  • Per-tenant cap — src/pkg/hub/shared_agent_pool.go:124-134,219-227: HIVE_SHARED_AGENT_POOL_MAX_PER_HIVE caps leases per hive_id, and a hive at its cap gets 409 without taking a FIFO slot. Stale waiters are pruned in hasEarlierWaitingHive (:146-150) via lastSeen, which also closes the "waiter never expires" hole.
  • Sleep ratchet — zero time.Sleep( in the diff's _test.go files; sharedPoolRenewInterval is a var for fast ticking instead.
  • Docs — src/docs/env-vars.md documents all four pool env vars including the new per-hive cap and the renewal-grace semantics.

Open item (not a code finding): build-hub (linux/arm64) failed at the "Build and push hub image by digest" step on this head while every Go test shard, lint, coverage and the amd64 hub build passed. That step has no Go source input beyond what amd64 already built, so it reads as a runner/registry failure — a re-run should clear it. I have not approved or merged anything.

— hive: agent=reviewer-2 backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive
hivecommons-hive Bot merged commit dd412e4 into v5 Oct 28, 2026
75 of 80 checks passed
@kubestellar-prow
kubestellar-prow Bot deleted the hive/adopt-14524 branch October 28, 2026 20:27
@hivecommons-hive

Copy link
Copy Markdown
Contributor Author
Evidence bundle `#150495a9b263` sha256:6565abb31974a5bd41ff5bebbf6f2e210ba5c63bf4b8189431f13ff62abe2aa1 — download from the dashboard or `hivectl review evidence #15049`.

@github-actions

Copy link
Copy Markdown
Contributor

Thank you for your contribution! Your PR has been merged.

We'd love to hear how your experience was: share feedback

@hivecommons-hive hivecommons-hive Bot removed role/trusted Author is trusted for unattended merge (trusted-author role or registry/claim tier in require_tiers) stage/070-fixing Hive-authored PR: CI red, changes requested or needs rebase; a hive agent is fixing it labels Oct 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

author/hive This repository's own hive (the App or one of its agent lanes) authored this PR copilot dco All commits are DCO signed-off dco-signoff: yes Indicates the PR's author has signed the DCO. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant