Skip to content

πŸ› fix(packs): give gateway-moved pack agents a usable model - #16731

Merged
hivecommons-hive[bot] merged 1 commit into
v5from
scanner/fix-14678
Nov 5, 2026
Merged

hivecommons-hive[bot] merged 1 commit into
v5from
scanner/fix-14678

Conversation

@hivecommons-hive

Copy link
Copy Markdown
Contributor

Why

On crux/v0.2.0-449-g55318dc61, which already includes #14777, #15130 and #16301, the reporter's reviewer card shows backend litellm ⚑ but the model dropdown is empty. The card also reads "Paused β€” session down Β· scheduling paused Β· next kick never". scanner, quality and guide run fine on the same IBM LiteLLM gateway. So the backend move worked, but the reviewer was left with no model it can use.

Root cause

When the pack moved a pack-owned agent onto the hive's gateway, it cleared the agent's model and expected the gateway's default_model to fill it in. In this hive the gateway has no default_model. The operator chose a model for each agent on the dashboard instead (a live ModelOverride restored from hive-state.json, or an operator-owned model:). DefaultInferenceModel returned "", so the reviewer launched Claude Code through the gateway with no model and failed every time. The watchdog then paused it with a crash-loop pause. Nothing ever resumed it, because the pack apply only resumes agents the pack itself paused.

There was also a smaller issue. After an apply moves an operator agent's pack-owned config onto the gateway that its live override already names, the override matches the config. On the next restart that agent was no longer counted as an operator pick.

Fix (src/pkg/dashboard/api_packs.go)

  • hiveGatewayModel: if the gateway has no default model, a pack agent placed on it gets the model the operator's agents on that gateway run (their live ModelOverride, or a config model the pack did not write; the most common one if they differ). If the gateway does have a default_model, behaviour is unchanged (πŸ› bug: Mysterious failure to do inferenceΒ #9945): the model stays unset.
  • This applies when a level creates an agent, when it moves an agent off its pack CLI, and when it repairs an agent an earlier apply left on the gateway with no model.
  • resumeGatewayModelCrashLoops: an agent that this apply gives a model it was missing, and that is paused with the watchdog's crash-loop trigger, is resumed. Other pauses are left alone.
  • A gateway override on top of a pack-owned config backend equal to it still counts as an operator pick, so the result holds across restarts.

Tests in pack_gateway_model_test.go cover the override and config shapes for create, move and repair, with two applies each. They also check that a gateway default_model still takes precedence, and that only a crash-loop pause is resumed.

Closes #14678

β€” hive: agent=scanner backend=copilot model=claude-fable-5.1 copilot=1.0.88 requested_by=@MikeSpreitzer

A pack agent moved onto an inference gateway had its model cleared so the
gateway's default_model would apply. When the gateway has none (the operator
picked a model per agent on the dashboard), the agent launched with no model,
crash-looped and was paused by the watchdog. It now takes the model the
operator's agents on that gateway run, and a crash-loop pause on an agent
this repairs is resumed.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: scanner <280983584+hivecommons-hive[bot]@users.noreply.github.com>
Co-authored-by: MikeSpreitzer <14296719+MikeSpreitzer@users.noreply.github.com>
@kubestellar-prow kubestellar-prow Bot added the dco-signoff: yes Indicates the PR's author has signed the DCO. label Nov 5, 2026
@kubestellar-prow

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign kellyaa for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@kubestellar-prow kubestellar-prow Bot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Nov 5, 2026
@github-actions

github-actions Bot commented Nov 5, 2026

Copy link
Copy Markdown
Contributor

Open PR overlap check

Base branch: v5

Conflicts are checked with every PR merged onto the current tip of v5, so a PR that only needs a rebase is not reported as conflicting with its siblings.

No conflicting open PRs or same-file overlaps were found.

@hivecommons-hive

hivecommons-hive Bot commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor Author

Hive status β€” updated in place; edits do not notify. Full history is in the Activity Monitor.

hive/awaiting-ci: CI pending β€” sweep will merge when green.


🐝 Hive Agent: scanner | Instance: hosted-available-oke-11-placeholder-r05x | SHA: unknown

β€” hive: agent=scanner backend=copilot model=claude-fable-5.1 copilot=1.0.88

Hive filed review findings that do not block this PR as backlog items:

Cap: 10 backlog item(s) per PR per day.

Review findings from reviewer (folded here because this head already carries its hive review):

Read the diff against the v5 tree at b3b75b2, plus the callers it depends on. This looks correct to me.

What I verified

  • Ordering is right for the resume path: each changed agent goes through AgentMgr.UpdateConfig inside the loop (src/pkg/dashboard/api_packs.go:341) before resumeGatewayModelCrashLoops runs after it (api_packs.go:469), and Manager.Resume relaunches a StatePaused agent from agent.Config (src/crux/agent/manager_pause.go:138-196), so the relaunch sees the new model.
  • Only watchdog.CrashLoopTrigger pauses are touched (api_packs.go:1393); operator and rotation pauses are left alone.
  • Gateway-with-default_model behaviour is unchanged: hiveGatewayModel returns "" there (api_packs.go:1359), so the repair branch at api_packs.go:257-268 reduces to the previous "clear the pack model" path (πŸ› bug: Mysterious failure to do inferenceΒ #9945). TestApplyPackGatewayDefaultModelOutranksOperatorsModel covers it.
  • No self-feedback across restarts: the model written is stamped ModelOwner=pack (api_packs.go:234,266), and hiveAgentBackends only exports a config model when ModelOwner != pack (api_packs.go:1219-1221), so moved pack agents never vote for their own copied model.
  • The new "override equals pack-owned gateway config" operator-pick rule (api_packs.go:1223-1226) cannot be triggered by the rotation sweep, which only sets an override to a different backend (src/cmd/hive/main_sweeps.go:834).

One low-severity note, not a blocker

api_packs.go:1225 reassigns the whole vote, which drops the config-derived vote.model set at :1220. It is restored at :1228 only when a live ModelOverride exists. An agent with a live BackendOverride but an operator-owned model: in config and no live ModelOverride would therefore not contribute its model. In practice the dashboard sets both together (api_agent_config.go:846,851), so this is an edge the current callers do not hit; keeping vote.model across the reassignment would close it if you want to.

Evidence bundle `#16731b3b75b2` sha256:a8bf29948d52d99001ba8470d3e97c6fd0906b82723628e7d7f89c87078cc3e4 β€” download from the dashboard or `hivectl review evidence #16731`.

@hivecommons-hive hivecommons-hive Bot added author/hive This repository's own hive (the App or one of its agent lanes) authored this PR dco All commits are DCO signed-off role/trusted Author is trusted for unattended merge (trusted-author role or registry/claim tier in require_tiers) stage/050-review-pending Review swarm requested; awaiting a verdict for the current head labels Nov 5, 2026

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Read the apply loop and the resume path at b3b75b2; this looks correct to me.

  • Ordering is right for the resume. UpdateConfig is called per agent inside the loop (src/pkg/dashboard/api_packs.go:341), and resumeGatewayModelCrashLoops runs only after the loop (:469), so when Manager.Resume relaunches (src/crux/agent/manager_pause.go:138, forceRelaunch on StatePaused) the manager already holds the config with the filled model.
  • #9945 behaviour is preserved. hiveGatewayModel returns "" whenever DefaultInferenceModel(gateway) != "" (api_packs.go:1359), so a gateway with a default_model still gets an unset model; the repair branch (:255) then compares "" != "" and is a no-op, so no churn on restarts. TestApplyPackGatewayDefaultModelOutranksOperatorsModel pins this.
  • Only crash-loop pauses are touched. :1395 checks proc.PausedTrigger != watchdog.CrashLoopTrigger and skips everything else; TestApplyPackResumesGatewayAgentCrashLoopedWithoutModel covers both the positive and the dashboard-api negative case.
  • The moved-agent branch (:230) and repair branch both use the pre-apply hiveBackends snapshot, so a second apply converges (tests run each shape twice).

One note, not a defect: the reviewer's model: is written pack-owned, so it will track whatever model the operator's agents on that gateway run (most common, ties by name) on every restart. That is what the docs change describes, so it is intended.

CI: all dashboard/agent/scheduler test shards green at this head; tide is the only pending gate.

β€” hive: agent=reviewer-2 backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive hivecommons-hive Bot added stage/060-reviewed Review swarm verdict approve; a gate is still pending (CI running, soak not started) and removed stage/050-review-pending Review swarm requested; awaiting a verdict for the current head labels Nov 5, 2026
@hivecommons-hive
hivecommons-hive Bot merged commit 14d65c0 into v5 Nov 5, 2026
91 of 98 checks passed
@kubestellar-prow
kubestellar-prow Bot deleted the scanner/fix-14678 branch November 5, 2026 08:19
@github-actions

github-actions Bot commented Nov 5, 2026

Copy link
Copy Markdown
Contributor

Thank you for your contribution! Your PR has been merged.

We'd love to hear how your experience was: share feedback

@hivecommons-hive hivecommons-hive Bot removed role/trusted Author is trusted for unattended merge (trusted-author role or registry/claim tier in require_tiers) stage/060-reviewed Review swarm verdict approve; a gate is still pending (CI running, soak not started) labels Nov 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

author/hive This repository's own hive (the App or one of its agent lanes) authored this PR copilot dco All commits are DCO signed-off dco-signoff: yes Indicates the PR's author has signed the DCO. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

πŸ› bug: reviewer agent has mysteriously inconsistent configuration for inference

0 participants