Skip to content

✨ Betty runs several model calls per meeting at once - #18623

Closed
clubanderson wants to merge 2 commits into
v5from
betty-parallel-answers
Closed

clubanderson wants to merge 2 commits into
v5from
betty-parallel-answers

Conversation

@clubanderson

@clubanderson clubanderson commented Nov 17, 2026 •

Copy link
Copy Markdown
Member

Part 1 of the concurrent-requests work on #18590: parallel model answers.

Before: on the warm CLI backend (the live-test backend) every model call from one meeting was serialised by the session's one-turn semaphore, so a second question waited for the first answer. On the API backends Betty's calls already overlapped, bounded only by the hive-wide max_in_flight, but with no per-meeting cap and no budget reservation: concurrent calls could all pass the budget check on the same ledger total.

Now:

  • meetings.llm.per_meeting_in_flight (default 3, env HIVE_MEETINGS_LLM_PER_MEETING_IN_FLIGHT, clamped to meetings.llm.max_in_flight) bounds one meeting's concurrent model calls. The existing max_in_flight key already meant "across meetings", so the per-meeting cap gets its own key instead of changing that meaning.
  • meetllm.Gate (per meeting): admission semaphore plus a token reservation. A call in flight holds its expected cost (largest observed call, 2000 tokens until one is seen); the next call is refused with ErrBudget when ledger + reserved >= budget. Sequential behaviour is unchanged (reservation is zero), so the ledger stays the single record of spend and parallel answers cannot overspend.
  • meetllm.CLIPool (warm CLI backends): a warm claude stream-json process answers one turn at a time and its result events carry no request id, so turns cannot be interleaved on one process safely. The pool runs up to N processes; only the first is warmed at meeting start. When calls overlap and no idle warm CLI is left, the next one is warmed in the background, so a burst pays at most one cold start and a meeting that never overlaps runs one CLI.

Expected latency (not measured live here: running hives and the meetings namespace are off limits for this work): with 3 questions asked together on the API backends, answer N now arrives at about one model call (~1 to 2.5 s on Haiku) instead of N calls. On the claude CLI backend the first overlapping burst pays one cold start (a few seconds) for the second answer; later bursts answer warm in parallel.

Tests: gate_test.go (peak concurrency equals gate size for 1 and 3, budget reservation refuses an overlapping call, settle never goes negative, busy deadline), pool_test.go (overlapping turns land on 2 distinct processes and the pool stops at its size, sequential turns keep one CLI, closed pool refuses), config resolve clamps to max_in_flight.

🤖 Generated with Claude Code

A meeting may now have up to meetings.llm.per_meeting_in_flight (default
3, never above max_in_flight) model calls in flight, so a second request
no longer waits for the first answer. A per-meeting Gate bounds the calls
and reserves each in-flight call's expected cost against the token
budget, so concurrent calls cannot all pass the check on the same ledger
total. On a warm CLI backend the meeting keeps a pool of warm CLIs (one
turn per process, since stream results carry no request id) that grows
only when calls overlap.

Part of #18590.

Signed-off-by: Andy Anderson <andy@clubanderson.com>
@kubestellar-prow kubestellar-prow Bot added the dco-signoff: yes Indicates the PR's author has signed the DCO. label Nov 17, 2026
@kubestellar-prow

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign danathar for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@kubestellar-prow kubestellar-prow Bot added the size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. label Nov 17, 2026
@github-actions

github-actions Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Open PR overlap check

Base branch: v5

Conflicts are checked with every PR merged onto the current tip of v5, so a PR that only needs a rebase is not reported as conflicting with its siblings.

PR Title Relation Files
#18615 ✨ Meetings: delete finished meetings and stale join requests same files src/docs/meetings.md
src/pkg/dashboard/api_meetings.go
#18625 ✨ Betty names who and what each answer is for when requests overlap same files src/docs/meetings.md
src/pkg/meetings/betty/answer.go
#18633 🐛 Meetings: don't restart-loop the Spektacular stream child when live mode isn't available same files src/cmd/hive/meetingswire.go
#18627 ✨ Betty speaks ready answers shortest first and moves a long voice queue to chat same files src/cmd/hive/meetingswire.go
src/docs/meetings.md
src/pkg/config/meetings_config.go
src/pkg/dashboard/api_meetings.go
#18631 ✨ Betty cites the sources of chat answers and says when she is not sure same files src/docs/meetings.md
#18611 ✨ meetings: desktop companion meetings (/api/companion/*) same files src/docs/meetings.md
#18629 ✨ Betty times every answer stage and answers live-state questions without waiting for search same files src/docs/meetings.md
#18602 📖 docs: Betty on Zoom without a Zoom app, options and recommendation same files src/docs/meetings.md

@github-actions

Copy link
Copy Markdown
Contributor

Mutation testing

gremlins found 9 surviving mutant(s) in changed files: the code ran under tests, but changing it did not fail any test.

File Line Mutation
src/pkg/meetings/meetllm/gate.go 85 CONDITIONALS_BOUNDARY
src/pkg/meetings/meetllm/gate.go 88 CONDITIONALS_BOUNDARY
src/pkg/meetings/meetllm/meetllm.go 207 CONDITIONALS_BOUNDARY
src/pkg/meetings/meetllm/meetllm.go 207 CONDITIONALS_NEGATION
src/pkg/meetings/meetllm/meetllm.go 301 CONDITIONALS_BOUNDARY
src/pkg/meetings/meetllm/meetllm.go 328 CONDITIONALS_BOUNDARY
src/pkg/meetings/meetllm/pool.go 35 CONDITIONALS_BOUNDARY
src/pkg/meetings/meetllm/pool.go 144 CONDITIONALS_BOUNDARY
src/pkg/meetings/meetllm/pool.go 174 CONDITIONALS_NEGATION

3 mutant(s) in changed files are not reached by any test (NOT COVERED). A maintainer may apply the mutation/acknowledged label to accept this.

File Line Mutation
src/cmd/hive/meetingswire.go 347 CONDITIONALS_NEGATION
src/pkg/meetings/meetllm/cli.go 546 CONDITIONALS_NEGATION
src/pkg/meetings/meetllm/pool.go 153 CONDITIONALS_NEGATION

ℹ️ 3 mutant(s) on added lines were SKIPPED because gremlins v0.6.0 only honours the first lines added lines of each default-context hunk, so non-contiguous edits in one hunk are partly missed. This does not affect the verdict.

File Line Mutation
src/cmd/hive/meetingswire.go 363 CONDITIONALS_NEGATION
src/pkg/meetings/meetllm/meetllm.go 336 CONDITIONALS_NEGATION
src/pkg/meetings/meetllm/meetllm.go 336 CONDITIONALS_NEGATION

Verdict: lived (#17349).

@hivecommons-hive

Copy link
Copy Markdown
Contributor

Hive status — updated in place; edits do not notify. Full history is in the Activity Monitor.

@clubanderson this PR is held at stage/071-needs-author and the hive will not merge it yet.

Failing checks:

  • test (dashboard shuffle 3/3)

Next: the hive does not push to your branch. Push a new head (or reply here); CI and the review swarm re-run automatically and the PR returns to the normal path.

The Settings parity guard needs every hive.yaml leaf referenced by the
Settings UI; the meetings settings' effective values now include it.

Part of #18590.

Signed-off-by: Andy Anderson <andy@clubanderson.com>
@clubanderson

Copy link
Copy Markdown
Member Author

Combined into #18635 to get through CI and review together.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

dco-signoff: yes Indicates the PR's author has signed the DCO. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant