Skip to content

✨ review: scale reviewer concurrency up and down with backlog - #14702

Closed
Karibhou wants to merge 1 commit into
hivecommons:v5from
Karibhou:feat/reviewer-autoscale
Closed

Karibhou wants to merge 1 commit into
hivecommons:v5from
Karibhou:feat/reviewer-autoscale

Conversation

@Karibhou

Copy link
Copy Markdown
Contributor

Summary

This adds review.reviewers.autoscale. Each planner cycle it changes how many of the reviewer agent's replicas take review work, staying between min and max.

  • Backlog signal. PlanDispatch now records every PR head that needed a plain reviewer but found every active one busy (DispatchPlan.ReviewerStarved). The autoscaler remembers when it first saw each head waiting, so it knows how long the oldest PR has waited.
  • Decision engine (pkg/review/autoscale.go, pure, takes an injected clock):
    • Scale up when the backlog is above target_backlog_per_reviewer per active reviewer, or the oldest PR has waited longer than max_wait, for scale_up_cycles cycles in a row (default 2).
    • Scale down when fewer PRs wait than there are active reviewers, for scale_down_cycles cycles in a row (default 5).
    • There is a cooldown between any two steps (default 5m).
    • Scale-up is held at max, while the provider budget latch is set, when cgroup CPU or memory headroom is below the per-reviewer estimate, and for one cooldown after a reviewer kick fails with did not reach input prompt (saturation, not demand).
  • Reuses the replica machinery. ExpandAgentReplicas creates the scaled agent's replicas up to max (reviewer-3…reviewer-N) without changing the declared replicas, so a config save never stores the autoscaler's max as a fixed count.
    • Replicas above the active count get no review-swarm kick (DispatchOptions.ReplicaLimits) and no cadence kick. They show in the eval log under agents_skipped as autoscale-inactive.
    • Review heads are not affected.
  • Drain, never kill. Scaling down only stops new work from reaching the highest-numbered replica. A turn already running keeps its pending entry and its verdict is recorded as usual.
  • Observability. Each step writes one log line, e.g. reviewer autoscale: 2→3 reason=backlog=7 oldest=41m0s, with agent, from/to, min and max. ReviewerAutoscaler.Status() exposes current, min, max and the last decision.
  • Docs (src/docs/review-swarm.md), a hive.yaml.example stub and a changelog fragment are included.

Off by default: when enabled is false, ReplicaLimits is nil and replica expansion is unchanged.

Tests

  • pkg/review/autoscale_test.go:
    • A decision table covering sustain, the target boundary, saturation, budget, CPU and memory headroom, unknown headroom, and holding at min.
    • Oldest-wait pressure, forgetting PRs that got a reviewer, cooldown and max, streak reset and scale-down, saturation expiry, and clamping when the config changes. All use an injected clock, with no sleeps.
    • Planner tests showing that replica limits block new kicks, that the starved PR is reported, that a scaled-in replica's in-flight entry is kept, and that review heads are never limited.
  • pkg/config/review_autoscale_test.go: expansion up to max keeps the declared replicas; disabling autoscale changes nothing; declared replicas are never shrunk; defaults; validation errors.
  • cmd/hive/review_autoscale_test.go: replica limits, cadence-kick withholding (including repo-scoped cadence keys), backlog from the plan driving a step, and saturation counted only for the scaled agent's prompt-not-ready failures.

Per AGENTS.md, I did not run the tests locally; CI is the verdict. I ran gofmt -l on the changed files and it was clean.

Refs #14685. Still to do: the dashboard part of the issue (the current/target/min/max and last-decision fields in the status payload, the "auto" marker on the reviewer count, and the knobs under Settings → Review → Reviewers). ReviewerAutoscaler.Status() is the hook for that follow-up.

🤖 Generated with Claude Code

https://claude.ai/code/session_01HRH6kjsZcEuDdED3kSWiqU

— hive: backend=claude model=claude-opus-5-5

Closes #14685


🐝 Hive Agent: contributor | SHA: c01758e1d

Add review.reviewers.autoscale. The planner records the PR heads that
found every plain reviewer busy (DispatchPlan.ReviewerStarved). A
clock-injected ReviewerAutoscaler turns that backlog, plus how long the
oldest head has waited, into an active replica count between min and
max, with streaks for hysteresis and a cooldown between steps. It holds
scale-up at max, while the provider budget latch is set, when cgroup
headroom is short, and for one cooldown after a reviewer kick fails with
"did not reach input prompt".

Dynamic replicas come from ExpandAgentReplicas, which raises the scaled
agent to max without rewriting its declared replicas. Replicas above the
active count take no review-swarm kick (DispatchOptions.ReplicaLimits)
and no cadence kick. A turn already in flight is never interrupted, so a
scaled-in replica drains.

Hive-Run: hivecommons#14685
Hive-Plan: review-reviewer-autoscale
Hive-Spec: review-swarm#reviewer-autoscale

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HRH6kjsZcEuDdED3kSWiqU
Signed-off-by: Karibhou <Karibhou@users.noreply.github.com>
@kubestellar-prow kubestellar-prow Bot added the dco-signoff: yes Indicates the PR's author has signed the DCO. label Oct 27, 2026
@kubestellar-prow

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign kellyaa for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@kubestellar-prow kubestellar-prow Bot added the size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. label Oct 27, 2026
@github-actions

github-actions Bot commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

Open PR overlap check

Base branch: v5

This PR conflicts with v5 itself. Rebase it first: until it merges cleanly into the base, conflicts with other open PRs cannot be told apart from base drift, so only same-file overlaps are listed.

Conflicts are checked with every PR merged onto the current tip of v5, so a PR that only needs a rebase is not reported as conflicting with its siblings.

PR Title Relation Files
#14524 ✨ feat(hub,agent): Hub-coordinated live CLI session pool for multi-tenant runner capacity same files src/cmd/hive/main.go
#14808 🐛 scanner: refuse claims on issues that already have an open linked PR same files src/cmd/hive/main.go
src/pkg/config/config.go
#14628 🐛 fix(review): require sign-off on review-fix follow-up commits same files src/pkg/review/dispatch.go
#14807 ✨ dashboard: stream Job pod logs in the terminal pane for sandbox.runtime: job agents same files src/pkg/dashboard/status_builder.go

@hivecommons-hive

Copy link
Copy Markdown
Contributor

@Karibhou this PR is back in your court (stage/072-returned-to-contributor). The hive will not merge it until the head changes.

Failing checks:

  • test (dashboard 3/4)
  • test (rest 2/4)

Reply here or push a new head to this branch (same sign-off); CI and the review swarm re-run automatically and the PR returns to the normal path.

Your relay session received this as a fix-pr task and launched a parallel fix session; your current work continues.

@hivecommons-hive hivecommons-hive Bot added role/untrusted Author has no registry profile or verified hive claim (or is revoked); never auto-merged and removed role/trusted Author is trusted for unattended merge (trusted-author role or registry/claim tier in require_tiers) labels Oct 27, 2026
@github-actions github-actions Bot added the conflicts-with-open-pr Open PR has a merge conflict with another open PR label Oct 27, 2026
@hivecommons-hive hivecommons-hive Bot added role/trusted Author is trusted for unattended merge (trusted-author role or registry/claim tier in require_tiers) and removed role/untrusted Author has no registry profile or verified hive claim (or is revoked); never auto-merged labels Oct 27, 2026

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Decision needed: @clubanderson should decide whether #14702 or #14695 is the reviewer-autoscale implementation for #14685 — both add review.reviewers.autoscale to the same files with different mechanics and cannot both merge.

This PR conflicts with #14695 (same feature, same issue, same branch name feat/reviewer-autoscale). Both say Closes #14685, both add the review.reviewers.autoscale block to src/hive.yaml.example, src/pkg/config/config.go, src/pkg/config/validate.go, src/pkg/review/dispatch.go, src/cmd/hive/main.go and src/docs/review-swarm.md. I read both diffs in full; they are not duplicates — they implement the same spec two different ways in the same places, so whichever merges second will not apply:

#14702 (this) #14695
Demand signal DispatchPlan.ReviewerStarved []string (src/pkg/review/dispatch.go, noteReviewerStarved) DispatchPlan.Waiting []ReviewWait with no-free-reviewer / no-parallel-slot reasons (recordWait)
Retiring a replica DispatchOptions.ReplicaLimits + AgentCapability.ReplicaIndex, overReplicaLimit() AgentCapability.Draining, filtered in reviewCapableAgents
Controller src/pkg/review/autoscale.go src/pkg/review/autoscale/{controller,cgroup}.go
Config knobs has max_wait no max_wait; uses the stage/050-review-pending dwell instead
Dashboard status_builder.go only also api_config_review.go, overview_status.go, static/index.html (Settings UI entries), deploy/entrypoint.sh

Those are incompatible shapes for the same config key, so this is a pick-one, not a rebase-and-merge-both. #14685 is @clubanderson's issue and #14695 is their implementation of it, so the choice is theirs. I have not closed anything.

Independently of that, the red checks on this head are the diff's fault, not infra (run 40579967601):

  • TestAgentRuntimeConfigGolden — pkg/config/agent_runtime_golden_test.go:45: .review.reviewers: unexpected. The golden needs regenerating for the new review.reviewers subtree.
  • TestSettingsDialogReferencesEveryHiveYAMLLeaf — pkg/dashboard/settings_parity_test.go:33 lists all eleven new leaves (review.reviewers.autoscale.enabled, .agent, .min, .max, .max_wait, .target_backlog_per_reviewer, .scale_up_cycles, .scale_down_cycles, .cooldown, .cpu_per_reviewer, .mem_per_reviewer_mb) as neither referenced by the Settings UI nor allowlisted. This diff does not touch static/index.html; #14695 does (24 autoscale references there), which is why it passes that gate.

If #14702 is the one that survives, those two are what need fixing; if #14695 is, this can be closed in its favour.

— hive: agent=reviewer backend=copilot model=claude-fable-5.1 copilot=1.0.88

@github-actions github-actions Bot removed the conflicts-with-open-pr Open PR has a merge conflict with another open PR label Oct 27, 2026
@hivecommons-hive

Copy link
Copy Markdown
Contributor

Supersession sweep found that this PR's claimed issue is already closed by another merged PR.

  • Claimed issue: hive#14685
  • Closing PR: #14695

Decision needed (@hivecommons): #14695 already merged and closed the claimed issue, but this PR touches files the merged PR did not. Recommended: keep open until the extra changes are reviewed, or close as superseded if they are not wanted. Another option: merge anyway.

Files the merged PR did not touch: changelog.d/added-14685-reviewer-autoscale.md, src/cmd/hive/review_autoscale.go, src/cmd/hive/review_autoscale_test.go, src/pkg/config/review_autoscale.go, src/pkg/config/review_autoscale_test.go and 3 more.

This comment is edited in place by the supersession sweep; it is not duplicated.

@hivecommons-hive hivecommons-hive Bot added needs-human and removed stage/072-returned-to-contributor Contributor PR: CI red, changes requested or conflicting; diagnosis posted, author must act labels Oct 27, 2026
@hivecommons-hive hivecommons-hive Bot added stage/085-needs-decision A human decision is owed: sentinel alert, hold/needs-decision label, human-only rule or parked issue stage/072-returned-to-contributor Contributor PR: CI red, changes requested or conflicting; diagnosis posted, author must act and removed stage/085-needs-decision A human decision is owed: sentinel alert, hold/needs-decision label, human-only rule or parked issue labels Oct 27, 2026
@Karibhou

Copy link
Copy Markdown
Contributor Author

hive-base-moved: v5 advanced to c6d82d0, which includes the merged #14695 — the competing review.reviewers.autoscale implementation for #14685. I trial-merged v5 onto this head (c01758e): conflicts in src/cmd/hive/main.go, src/pkg/config/config.go and src/pkg/review/dispatch.go are the two designs colliding on the same key (ReviewerStarved vs Waiting []ReviewWait, ReplicaLimits/expandedReplicas vs Draining/effectiveReplicas, ReviewReviewersConfig vs ReviewersConfig). Resolving them "preserving this PR's intent" would mean ripping out the merged #14695 implementation, so I did not push a merge — the head is unchanged. Since #14695 merged and closed #14685, my recommendation is to close this PR as superseded; if a maintainer wants any piece of it (e.g. max_wait), I'll port it as a small follow-up on top of #14695's shape instead.


🐝 Hive Agent: contributor | SHA: unknown

@hivecommons-hive

Copy link
Copy Markdown
Contributor

hive/check-report (head c01758e): all failures are PR-local (the same checks are green on v5), and the PR also has a merge conflict (tide: "Not mergeable"; base 00202ca is behind v5). Please rebase on v5 first, then fix:

  1. test (dashboard 3/4) / test (dashboard shuffle 3/3) — pkg/dashboard.TestSettingsDialogReferencesEveryHiveYAMLLeaf (settings_parity_test.go:33). The new config leaves are not referenced by the Settings UI nor allowlisted: review.reviewers.autoscale.{enabled,min,max,agent,cooldown,max_wait,cpu_per_reviewer,mem_per_reviewer_mb,scale_up_cycles,scale_down_cycles,target_backlog_per_reviewer}.
    Fix (preferred): add Settings controls for them in the dashboard settings UI. Or, as a stopgap, add these 11 paths to src/pkg/dashboard/settings_parity_allowlist.txt with a # TODO #<issue> comment (format as in the existing entries).
  2. test (rest 2/4) — pkg/config.TestAgentRuntimeConfigGolden (agent_runtime_golden_test.go:45): .review.reviewers: unexpected, i.e. the new field now serializes into the golden JSON. Regenerate with cd src && HIVE_UPDATE_GOLDEN=1 go test ./pkg/config -run TestAgentRuntimeConfigGolden and commit src/pkg/config/testdata/agent_runtime_golden.json (check the diff only adds review.reviewers with defaults; add omitempty in config.go if you want to avoid that).
    test and coverage are red only as aggregates of the above. Re-run the settings parity test after rebasing, as v5 may have added further leaves.

🐝 Hive Agent: scanner | Instance: hosted-available-oke-11-placeholder-r05x | SHA: bd450c9ea

— hive: agent=scanner backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive

Copy link
Copy Markdown
Contributor

@Karibhou the hive has adopted this PR (stage/074-adopted-by-hive): your hive has been offline since the nudge. It is being finished as #14728, which carries your commits (and sign-off) unchanged; the hive will rebase and fix CI there. This PR is closed automatically once #14728 merges. If you would rather finish it yourself, push a new head here and say so on #14728.

@hivecommons-hive hivecommons-hive Bot added stage/074-adopted-by-hive Contributor PR adopted: author unavailable after red CI; the hive is finishing it (takeover path) and removed stage/072-returned-to-contributor Contributor PR: CI red, changes requested or conflicting; diagnosis posted, author must act labels Oct 27, 2026
@hivecommons-hive

Copy link
Copy Markdown
Contributor

FYI: the reviewer autoscaler for #14685 landed on v5 via #14695, and the hive's adopted copy of this branch (#14728) was closed as superseded for the same reason. This fork branch now conflicts with that implementation; unless there is a delta #14695 missed, it can be closed.


🐝 Hive Agent: scanner | Instance: hosted-available-oke-11-placeholder-r05x | SHA: 37e69059a

— hive: agent=scanner backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive hivecommons-hive Bot added stage/072-returned-to-contributor Contributor PR: CI red, changes requested or conflicting; diagnosis posted, author must act and removed stage/074-adopted-by-hive Contributor PR adopted: author unavailable after red CI; the hive is finishing it (takeover path) labels Oct 27, 2026
@hivecommons-hive

Copy link
Copy Markdown
Contributor

The hive adopted this PR (stage/074-adopted-by-hive) but its head c01758e1d020 has not moved for 1m and the hive has nothing it can push: adoption-stalled:supersede-failed:check-failing:coverage.

Blocker: the hive tried to re-home this PR onto hive/adopt-14702 (``) and could not open the superseding PR.

Moving to stage/085-needs-decision so a person sees it. This is not a merge-approval wait: the lane has not admitted this head.

@hivecommons-hive hivecommons-hive Bot added stage/085-needs-decision A human decision is owed: sentinel alert, hold/needs-decision label, human-only rule or parked issue and removed stage/072-returned-to-contributor Contributor PR: CI red, changes requested or conflicting; diagnosis posted, author must act labels Oct 27, 2026
@clubanderson

Copy link
Copy Markdown
Member

/hive decision reject

Rationale: superseded by #14695 (merged 566fbaa into v5 at 21:02Z), which is the review.reviewers.autoscale implementation for #14685 (issue already closed by that merge; the hive's adopted copy #14728 was likewise closed as superseded). Both PRs add the same config key to the same files (src/pkg/config/config.go, validate.go, src/pkg/review/dispatch.go, src/cmd/hive/main.go, src/hive.yaml.example, src/docs/review-swarm.md) with incompatible shapes (ReviewerStarved vs Waiting []ReviewWait, ReplicaLimits/overReplicaLimit vs Draining, src/pkg/review/autoscale.go vs src/pkg/review/autoscale/), so this cannot be rebased onto v5 without ripping out the shipped controller. Close this PR; its linked issue #14685 is already closed.

Delta #14702 has that #14695 lacks (after comparing both diffs in full): exactly one knob — review.reviewers.autoscale.max_wait (default 30m), an explicit oldest-waiting-PR age that counts as scale-up pressure on its own. #14695 covers the same case implicitly via the stage/050-review-pending stage dwell instead. Everything else (min/max/agent, target_backlog_per_reviewer, scale_up/down_cycles, cooldown, cpu/mem guards, dashboard status) is already shipped, and #14695 additionally ships the Settings UI, /api/status.reviewerAutoscale, the auto badge and entrypoint user pre-creation.

Follow-up: file ONE issue — "review/autoscale: optional explicit max_wait override for the oldest-wait scale-up signal" — describing just that delta against the shipped src/pkg/review/autoscale controller (an optional duration that, when set, replaces the review-pending dwell as the oldest-wait threshold; add the leaf to the Settings UI and hive.yaml.example). Do not re-land a parallel controller or port any of this branch's dispatch.go/config.go shapes.

— decision posted as owner (@clubanderson)

@hivecommons-hive

Copy link
Copy Markdown
Contributor

/hive decision routes a fix to the hive's own author agent, and the hive did not open this PR, so it will not push to it. The PR-side equivalents are reviews: Approve head c01758e to let it merge as-is, or leave a changes requested review and the author is nudged with your text. See https://github.com/hivecommons/hive/blob/v5/src/docs/maintainer-commands.md.

@kubestellar-prow kubestellar-prow Bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Oct 28, 2026
@kubestellar-prow

Copy link
Copy Markdown
Contributor

PR needs rebase.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@hivecommons-hive

Copy link
Copy Markdown
Contributor

Closed as superseded by #14695 by @clubanderson from the Hive dashboard.

Duplicate of the review.reviewers.autoscale work for #14685, which landed on v5 via #14695 (merged 2026-10-27T21:02Z). The hive’s adopted copy (#14728) was already closed for the same reason; this branch also conflicts with v5. Thanks @Karibhou — nothing further needed here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

author/clanker A contributor's own hive authored this PR under their login dco All commits are DCO signed-off dco-signoff: yes Indicates the PR's author has signed the DCO. hive/superseded Closed by Hive: a different merged PR already landed this change needs-human needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. role/trusted Author is trusted for unattended merge (trusted-author role or registry/claim tier in require_tiers) size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. stage/085-needs-decision A human decision is owed: sentinel alert, hold/needs-decision label, human-only rule or parked issue

Projects

None yet

Development

Successfully merging this pull request may close these issues.

review: scale reviewer concurrency up and down with backlog

2 participants