Repository navigation
π fix(packs): give gateway-moved pack agents a usable model - #16731
Conversation
A pack agent moved onto an inference gateway had its model cleared so the gateway's default_model would apply. When the gateway has none (the operator picked a model per agent on the dashboard), the agent launched with no model, crash-looped and was paused by the watchdog. It now takes the model the operator's agents on that gateway run, and a crash-loop pause on an agent this repairs is resumed. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: scanner <280983584+hivecommons-hive[bot]@users.noreply.github.com> Co-authored-by: MikeSpreitzer <14296719+MikeSpreitzer@users.noreply.github.com>
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
Open PR overlap checkBase branch: Conflicts are checked with every PR merged onto the current tip of No conflicting open PRs or same-file overlaps were found. |
|
Hive status β updated in place; edits do not notify. Full history is in the Activity Monitor. hive/awaiting-ci: CI pending β sweep will merge when green. π Hive Agent: β hive: agent=scanner backend=copilot model=claude-fable-5.1 copilot=1.0.88 Hive filed review findings that do not block this PR as backlog items:
Cap: 10 backlog item(s) per PR per day. Review findings from Read the diff against the v5 tree at What I verified
One low-severity note, not a blocker
|
There was a problem hiding this comment.
Read the apply loop and the resume path at b3b75b2; this looks correct to me.
- Ordering is right for the resume.
UpdateConfigis called per agent inside the loop (src/pkg/dashboard/api_packs.go:341), andresumeGatewayModelCrashLoopsruns only after the loop (:469), so whenManager.Resumerelaunches (src/crux/agent/manager_pause.go:138,forceRelaunchonStatePaused) the manager already holds the config with the filled model. - #9945 behaviour is preserved.
hiveGatewayModelreturns""wheneverDefaultInferenceModel(gateway) != ""(api_packs.go:1359), so a gateway with adefault_modelstill gets an unset model; the repair branch (:255) then compares"" != ""and is a no-op, so no churn on restarts.TestApplyPackGatewayDefaultModelOutranksOperatorsModelpins this. - Only crash-loop pauses are touched.
:1395checksproc.PausedTrigger != watchdog.CrashLoopTriggerand skips everything else;TestApplyPackResumesGatewayAgentCrashLoopedWithoutModelcovers both the positive and thedashboard-apinegative case. - The moved-agent branch (
:230) and repair branch both use the pre-applyhiveBackendssnapshot, so a second apply converges (tests run each shape twice).
One note, not a defect: the reviewer's model: is written pack-owned, so it will track whatever model the operator's agents on that gateway run (most common, ties by name) on every restart. That is what the docs change describes, so it is intended.
CI: all dashboard/agent/scheduler test shards green at this head; tide is the only pending gate.
β hive: agent=reviewer-2 backend=copilot model=claude-fable-5.1 copilot=1.0.88
|
Thank you for your contribution! Your PR has been merged. We'd love to hear how your experience was: share feedback |
Why
On
crux/v0.2.0-449-g55318dc61, which already includes #14777, #15130 and #16301, the reporter's reviewer card shows backendlitellm β‘but the model dropdown is empty. The card also reads "Paused β session down Β· scheduling paused Β· next kick never". scanner, quality and guide run fine on the same IBM LiteLLM gateway. So the backend move worked, but the reviewer was left with no model it can use.Root cause
When the pack moved a pack-owned agent onto the hive's gateway, it cleared the agent's model and expected the gateway's
default_modelto fill it in. In this hive the gateway has nodefault_model. The operator chose a model for each agent on the dashboard instead (a liveModelOverriderestored fromhive-state.json, or an operator-ownedmodel:).DefaultInferenceModelreturned"", so the reviewer launched Claude Code through the gateway with no model and failed every time. The watchdog then paused it with a crash-loop pause. Nothing ever resumed it, because the pack apply only resumes agents the pack itself paused.There was also a smaller issue. After an apply moves an operator agent's pack-owned config onto the gateway that its live override already names, the override matches the config. On the next restart that agent was no longer counted as an operator pick.
Fix (
src/pkg/dashboard/api_packs.go)hiveGatewayModel: if the gateway has no default model, a pack agent placed on it gets the model the operator's agents on that gateway run (their liveModelOverride, or a config model the pack did not write; the most common one if they differ). If the gateway does have adefault_model, behaviour is unchanged (π bug: Mysterious failure to do inferenceΒ #9945): the model stays unset.resumeGatewayModelCrashLoops: an agent that this apply gives a model it was missing, and that is paused with the watchdog's crash-loop trigger, is resumed. Other pauses are left alone.Tests in
pack_gateway_model_test.gocover the override and config shapes for create, move and repair, with two applies each. They also check that a gatewaydefault_modelstill takes precedence, and that only a crash-loop pause is resumed.Closes #14678
β hive: agent=scanner backend=copilot model=claude-fable-5.1 copilot=1.0.88 requested_by=@MikeSpreitzer