Repository navigation
✨ Betty runs several model calls per meeting at once - #18623
clubanderson wants to merge 2 commits into
Conversation
A meeting may now have up to meetings.llm.per_meeting_in_flight (default 3, never above max_in_flight) model calls in flight, so a second request no longer waits for the first answer. A per-meeting Gate bounds the calls and reserves each in-flight call's expected cost against the token budget, so concurrent calls cannot all pass the check on the same ledger total. On a warm CLI backend the meeting keeps a pool of warm CLIs (one turn per process, since stream results carry no request id) that grows only when calls overlap. Part of #18590. Signed-off-by: Andy Anderson <andy@clubanderson.com>
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
Open PR overlap checkBase branch: Conflicts are checked with every PR merged onto the current tip of
|
Mutation testinggremlins found 9 surviving mutant(s) in changed files: the code ran under tests, but changing it did not fail any test.
3 mutant(s) in changed files are not reached by any test (NOT COVERED). A maintainer may apply the
ℹ️ 3 mutant(s) on added lines were SKIPPED because gremlins v0.6.0 only honours the first
Verdict: |
|
Hive status — updated in place; edits do not notify. Full history is in the Activity Monitor. @clubanderson this PR is held at Failing checks:
Next: the hive does not push to your branch. Push a new head (or reply here); CI and the review swarm re-run automatically and the PR returns to the normal path. |
The Settings parity guard needs every hive.yaml leaf referenced by the Settings UI; the meetings settings' effective values now include it. Part of #18590. Signed-off-by: Andy Anderson <andy@clubanderson.com>
|
Combined into #18635 to get through CI and review together. |
Part 1 of the concurrent-requests work on #18590: parallel model answers.
Before: on the warm CLI backend (the live-test backend) every model call from one meeting was serialised by the session's one-turn semaphore, so a second question waited for the first answer. On the API backends Betty's calls already overlapped, bounded only by the hive-wide
max_in_flight, but with no per-meeting cap and no budget reservation: concurrent calls could all pass the budget check on the same ledger total.Now:
meetings.llm.per_meeting_in_flight(default3, envHIVE_MEETINGS_LLM_PER_MEETING_IN_FLIGHT, clamped tomeetings.llm.max_in_flight) bounds one meeting's concurrent model calls. The existingmax_in_flightkey already meant "across meetings", so the per-meeting cap gets its own key instead of changing that meaning.meetllm.Gate(per meeting): admission semaphore plus a token reservation. A call in flight holds its expected cost (largest observed call, 2000 tokens until one is seen); the next call is refused withErrBudgetwhenledger + reserved >= budget. Sequential behaviour is unchanged (reservation is zero), so the ledger stays the single record of spend and parallel answers cannot overspend.meetllm.CLIPool(warm CLI backends): a warmclaudestream-json process answers one turn at a time and its result events carry no request id, so turns cannot be interleaved on one process safely. The pool runs up to N processes; only the first is warmed at meeting start. When calls overlap and no idle warm CLI is left, the next one is warmed in the background, so a burst pays at most one cold start and a meeting that never overlaps runs one CLI.Expected latency (not measured live here: running hives and the meetings namespace are off limits for this work): with 3 questions asked together on the API backends, answer N now arrives at about one model call (~1 to 2.5 s on Haiku) instead of N calls. On the
claudeCLI backend the first overlapping burst pays one cold start (a few seconds) for the second answer; later bursts answer warm in parallel.Tests:
gate_test.go(peak concurrency equals gate size for 1 and 3, budget reservation refuses an overlapping call, settle never goes negative, busy deadline),pool_test.go(overlapping turns land on 2 distinct processes and the pool stops at its size, sequential turns keep one CLI, closed pool refuses), config resolve clamps tomax_in_flight.🤖 Generated with Claude Code