Repository navigation
π Reconcile launch_cmd on backend switch and degrade instead of crash-looping on a base-config contradiction - #15750
Conversation
β¦op on a base-config contradiction The dashboard backend switch persisted backend: codex next to a stale launch_cmd: /usr/bin/bob β¦ (hive.yaml and the per-agent overlay). The running process kept its in-memory state, so nothing failed until the next pod restart, when config validation rejected the base config and the hive crash-looped for 18 hours. - claimAgentFieldOwnership, PUT β¦/models and ACMM pack backend moves now clear a launch_cmd that launches a different CLI than the new backend (GovernorConfig.ReconcileLaunchCmdForBackend, reusing the validator's detection) and log an INFO audit line; a matching launch_cmd is kept. - Config.Save (validateSaveGuard) and SaveAgentFile refuse a backend/launch_cmd contradiction so the loader never meets one written by the dashboard; handleAgentConfigModels now surfaces save errors as a system alert instead of log-only. - The loader (loadWithOverrides) drops a contradicting launch_cmd from the BASE config, keeps backend, logs ERROR in the #6024 overlay-rejection style and records the drop in Config.LaunchCmdDegradations; the dashboard raises a per-agent system alert at RegisterAPI. The #6024 'fell back to base config' path is no help when base has the same defect. - Tests for the handler, save refusal and loader degradation; docs and changelog fragment. Signed-off-by: Andy Anderson <andy@clubanderson.com>
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
Open PR overlap checkBase branch: Conflicts are checked with every PR merged onto the current tip of
|
|
@clubanderson this PR is held at Failing checks:
Next: the hive does not push to your branch. Push a new head (or reply here); CI and the review swarm re-run automatically and the PR returns to the normal path. |
|
@clubanderson this PR is held at Failing checks:
Next: the hive does not push to your branch. Push a new head (or reply here); CI and the review swarm re-run automatically and the PR returns to the normal path. |
Signed-off-by: Andy Anderson <andy@clubanderson.com>
|
stage 080-awaiting-merge-admission β 050-review-pending: check-pending |
There was a problem hiding this comment.
Reviewed (correctness) β no in-scope blockers at 478e194. The three gaps in #15749 are closed where the issue says they are: claimAgentFieldOwnership (api_agent_control.go:208), handleAgentConfigModels (api_agent_config.go:994) and applyPack (api_packs.go:327) all reconcile before persisting; validateSaveGuard (config.go:7054) and SaveAgentFile (agent_overlay.go:279) refuse the contradiction; loadWithOverrides degrades before ExpandAgentReplicas/validation (config.go:5459), so replicas inherit the cleared launch_cmd. The pre-existing PUT /config path already validates before mutating in-memory state (api_agent_config.go:750), so a bad request cannot poison every later Save.
Two adjacent, non-blocking notes for follow-up:
hive validatenow printsconfig OKfor a base-config contradiction.src/cmd/hive/config_check.go:49-58only reports load failure and surviving provenance; the degradation is visible only in the slog ERROR line. Since this command exists precisely to answer "will this config boot as written?", consider surfacingcfg.LaunchCmdDegradations:fmt.Fprintf(stdout, "config OK: %s\n", *configPath) + for _, d := range cfg.LaunchCmdDegradations { + fmt.Fprintf(stderr, " WARNING: agent %s: launch_cmd %q dropped (contradicts backend %q)\n", d.Agent, d.LaunchCmd, d.Backend) + }
- Hot reload does not raise the banner.
surfaceLaunchCmdDegradationsruns only fromRegisterAPI(api.go:31); the watcher'sLoadWithDashboardOverlay(pkg/config/watcher.go:137) will degrade and log, but the per-agent alert is not re-raised for a config hand-edited into contradiction after boot.
β hive: agent=adjudicator-4 backend=copilot model=claude-fable-5.1 copilot=1.0.88
|
Hive filed review findings that do not block this PR as backlog items:
Cap: 10 backlog item(s) per PR per day. |
|
stage 080-awaiting-merge-admission β 050-review-pending: check-pending |
There was a problem hiding this comment.
Reviewed (correctness) β no blocker; one advisory gap between the docs/alert text and the code.
launch-cmd-degraded-<agent> alert is never cleared (low). src/pkg/dashboard/api_agent_control.go:246-259 raises the alert in surfaceLaunchCmdDegradations, and both the alert message (api_agent_control.go:257, "Save the agent once from the dashboard β¦ to make this permanent") and src/docs/agent-configuration.md:727 ("save the agent once from the dashboard (or fix the file) to clear the alert") say a save clears it. Nothing calls ClearSystemAlert(launchCmdDegradedAlertPrefix+β¦) β launchCmdDegradedAlertPrefix has exactly one use (the AddSystemAlert). The banner stays until the next pod restart even after the operator does what it asks. Since a successful Config.Save() writes every agent (with the already-cleared launch_cmd) to hive.yaml and the dashboard overlay, the simplest honest fix is to drop all degraded banners after a save succeeds:
--- a/src/pkg/dashboard/api.go
+++ b/src/pkg/dashboard/api.go
@@ func (s *Server) saveConfig() error {
if err := s.deps.Config.Save(); err != nil {
return err
}
+ // The file on disk now carries the degraded (valid) launch_cmds, so the
+ // boot-time "dropped launch_cmd" banners no longer describe a drift.
+ for _, d := range s.deps.Config.LaunchCmdDegradations {
+ s.ClearSystemAlert(launchCmdDegradedAlertPrefix + d.Agent)
+ }(or, if you prefer per-agent clearing, call it in claimAgentFieldOwnership/handleAgentConfigModels/handleAgentConfigGeneral after their saves, and reword the doc line). Either way the doc sentence should match what the code does.
Test note (info): the ACMM pack-apply reconcile at src/pkg/dashboard/api_packs.go:325-327 has no test; the other two backend-mutating paths do. It is the same helper, so low risk.
Verified: DegradeContradictingLaunchCmds runs after overlay merge and before validateWithOptions (src/pkg/config/config.go:5450-5471) so an accepted overlay cannot reintroduce the contradiction; handleAgentConfigGeneral's cliPinValue backend edit (api_agent_config.go:514) is already gated by the pre-existing 400 at api_agent_config.go:748-752; saveLocked β validateSaveGuard guards both hive.yaml and the dashboard overlay (config.go:7140,7158). go build ./pkg/config/ ./pkg/dashboard/ passes at 9320180; go test is unavailable on this host, CI pending.
β hive: agent=adjudicator-4 backend=copilot model=claude-fable-5.1 copilot=1.0.88
There was a problem hiding this comment.
Read the three layers against the tree rather than the diff alone; the mechanism holds together.
- Reconcile is wired at every backend-writing site I could find.
claimAgentFieldOwnership(api_agent_control.go:208, serving/switch),handleAgentConfigModels(api_agent_config.go:994) and the pack-apply move (api_packs.go:325) all callreconcileLaunchCmdForBackendbeforeConfig.Agents[name]is written back. The only otherLaunchCmdwriter in the dashboard, the full agent PUT atapi_agent_config.go:444, already runsValidateLaunchCmdBackendat:750and 400s before saving β so the newvalidateSaveGuardβValidateAgentLaunchCmdsgate cannot be tripped by an in-memory contradiction a dashboard path introduced. That matters becauseSave()is shared: a stray contradiction would now block persistence for every agent, and I verified no current writer leaves one behind. SaveAgentFilewithGovernorConfig{}only loses the gateway-name half of the gate (config.go:6640-6646,isGatewayNameon a zero config); it cannot produce a false refusal, and the doc comment says so. Fine.- Boot degrade sits in the right slot:
loadWithOverridesrunsDegradeContradictingLaunchCmdsafter overlay merge +ApplyAgentDefaultsand beforevalidateWithOptions(config.go:5461-5471), and bails out of a drop when clearing would not satisfy the gate, so the fatal path survives for the cases it should. The dashboard-overlay path (loadWithDashboardOverlay,:5590) still goes throughRejectInvalidAgentOverlays, so a contradiction persisted there is rejected rather than validated against. go build ./pkg/config/ ./pkg/dashboard/passes on the head; the api-reference churn is the citation rewrite the doc check requires.
One thing worth a follow-up, low severity:
launch-cmd-degraded-<agent>is never cleared.surfaceLaunchCmdDegradationsraises it (api_agent_control.go:256) and nothing callsClearSystemAlert(launchCmdDegradedAlertPrefix+name)β the only reference to the prefix is the raise. The banner tells the operator to "save the agent once from the dashboard to make this permanent"; when they do,Config.Agents[name].LaunchCmdis already""so the save succeeds, yet the error banner stays up until the next restart. AClearSystemAlertinreconcileLaunchCmdForBackend's caller success paths (or wherever the agent is next persisted cleanly) would close the loop the banner itself describes.
Looks correct to me otherwise.
β hive: agent=reviewer backend=copilot model=claude-fable-5.1 copilot=1.0.88
There was a problem hiding this comment.
Reviewed (intent-alignment) β the diff closes all three gaps in #15749: reconcileLaunchCmdForBackend is wired into claimAgentFieldOwnership (src/pkg/dashboard/api_agent_control.go:208), handleAgentConfigModels (api_agent_config.go:994) and applyPack (api_packs.go:327); validateSaveGuard/SaveAgentFile gate the pair (config.go:7057, agent_overlay.go:279); and DegradeContradictingLaunchCmds runs before validateWithOptions in loadWithOverrides (config.go:5461). The 36 api-reference/*.md files are citation line-number refreshes only. One doc-accuracy nit:
Low β docs/changelog claim a 400 that this diff does not add. changelog.d/fixed-backend-switch-clears-launch-cmd.md:1 and src/docs/agent-configuration.md ("Saves refuse the contradiction β¦ surfaced as a 400 and a dashboard alert") describe the new save guards as returning HTTP 400. Neither new refusal path does: claimAgentFieldOwnership (api_agent_control.go:212-224) and handleAgentConfigModels (api_agent_config.go:1029-1042) log, raise agent-field-save-failed, and still return 200. The only 400 is the pre-existing #5921 check in handleAgentConfigUpdate (api_agent_config.go:750). Suggest rewording so the operator-facing doc matches behavior:
- **Saves refuse the contradiction.** `hive.yaml` and per-agent overlay writes validate the pair and return an error (logged and surfaced as an `agent-field-save-failed` dashboard alert; `PUT /api/config/agent/{name}` additionally rejects it with a 400) rather than persisting a config the next boot would reject.
β hive: agent=adjudicator-4 backend=copilot model=claude-fable-5.1 copilot=1.0.88
There was a problem hiding this comment.
Style/maintainability review (head 9320180)
1. The launch-cmd-degraded-<agent> alert is never cleared in-process, but the docs promise it is β src/pkg/dashboard/api_agent_control.go:244-259 is the only reference to launchCmdDegradedAlertPrefix; no ClearSystemAlert/ClearSystemAlertsExcept call uses it. src/docs/agent-configuration.md (new "launch_cmd and backend" section) says "save the agent once from the dashboard (or fix the file) to clear the alert", and the alert text itself says "Save the agent once from the dashboard β¦ to make this permanent". After a successful save the banner stays until the next restart. Since Config.Save writes every agent, the cheapest correct place is saveConfig (src/pkg/dashboard/api.go:740), which already fans out post-save side effects:
--- a/src/pkg/dashboard/api.go
+++ b/src/pkg/dashboard/api.go
@@ func (s *Server) saveConfig() error {
if err := s.deps.Config.Save(); err != nil {
return err
}
+ // hive.yaml now matches the in-memory agents, so any launch_cmd the
+ // loader dropped at boot is permanently gone from disk.
+ s.ClearSystemAlertsExcept(launchCmdDegradedAlertPrefix, nil)
+ s.deps.Config.LaunchCmdDegradations = nil
if s.deps.Governor != nil {(or reword the doc/alert to say the banner clears on the next restart).
2. Stale doc comment on ValidateAgentLaunchCmds β src/pkg/config/config.go:7063-7066 says "the dashboard's persistence paths (Save, SaveAgentFile) call it", but SaveAgentFile (src/pkg/config/agent_overlay.go:280) calls (GovernorConfig{}).ValidateLaunchCmdBackend directly and its own comment (agent_overlay.go:273-275) explains why it cannot run the gateway-aware half. Only validateSaveGuard calls it.
// ValidateAgentLaunchCmds runs the backend/launch_cmd gate over every agent.
// It is the write-side twin of the loader's check: Config.Save calls it (via
// validateSaveGuard) so a contradiction is refused at save time rather than
// discovered as a fatal error at boot. SaveAgentFile runs only the
// governor-independent half of the gate; see its doc comment.
Advisory (no change requested): reconcileLaunchCmdForBackend's bool return is unused at all three call sites (api_agent_config.go:994, api_agent_control.go:208, api_packs.go:327); DegradeContradictingLaunchCmds sorts its result but logs in map order β iterating sorted names (as ValidateAgentLaunchCmds does) would make the boot log deterministic. The 13 api-reference/*.md files are pure line-number citation churn (verified: every changed line pairs with its :N-normalised twin).
β hive: agent=ci-maintainer backend=copilot model=claude-fable-5.1 copilot=1.0.88
|
merged by Hive trusted-author auto-merge: author @clubanderson holds role owner and GitHub permission admin on repo; CI green at 80615b2. |
|
Thank you for your contribution! Your PR has been merged. We'd love to hear how your experience was: share feedback |
Fixes #15749
Summary
A dashboard backend switch persisted
backend: codexnext to a stalelaunch_cmd: /usr/bin/bob β¦; the next pod restart failed config validation on the base config and the hive crash-looped for 18 hours (hosted hivehosted-available-oke-08-placeholder-xf07). This closes all three gaps:launch_cmd. NewGovernorConfig.ReconcileLaunchCmdForBackendreuses the validator'sValidateLaunchCmdBackendgate;Server.reconcileLaunchCmdForBackendcalls it fromclaimAgentFieldOwnership(the/switchpath),handleAgentConfigModels(PUT β¦/modelswith a backend) and the ACMM pack-apply backend moves. Alaunch_cmdthat launches a different CLI is cleared with an INFO audit line (launch_cmd cleared on backend switch, agent, previous_launch_cmd, backend); a matching one, an empty one, or a wrapper with no opinion is kept.Config.Save(validateSaveGuardβ newConfig.ValidateAgentLaunchCmds) andconfig.SaveAgentFilerefuse a backend/launch_cmd contradiction.claimAgentFieldOwnershipalready turned a save error into anagent-field-save-failedalert;handleAgentConfigModelsnow does too instead of log-only.loadWithOverridescalls newConfig.DegradeContradictingLaunchCmds()before validation: a contradicting base-configlaunch_cmdis dropped,backendkept, logged at ERROR in the π A config contradiction bricks a hosted spoke with no supported recovery pathΒ #6024 overlay-rejection style, and recorded inConfig.LaunchCmdDegradations.RegisterAPIsurfaces onelaunch-cmd-degraded-<agent>system alert per entry. The π A config contradiction bricks a hosted spoke with no supported recovery pathΒ #6024 "fell back to base config" path remains for overlays but is no help when base has the same defect β this is the self-heal for that case. The fatal path is kept for any contradiction that clearinglaunch_cmdwould not resolve.Also: docs (
agent-configuration.md, new "launch_cmdandbackend" section) and a changelog fragment.Test evidence
New tests (no
time.Sleep):pkg/config:TestReconcileLaunchCmdForBackend,TestLoadWithOverrides_BaseConfigContradictionDegradesInsteadOfFailing,TestDegradeContradictingLaunchCmds_SortedAndIdempotent,TestSave_RefusesBackendLaunchCmdContradiction,TestValidateAgentLaunchCmds,TestSaveAgentFile_RefusesBackendLaunchCmdContradictionpkg/dashboard:TestClaimAgentFieldOwnership_{ClearsContradictingLaunchCmd,KeepsMatchingLaunchCmd,ModelOnlyLeavesLaunchCmd,SaveRefusalRaisesAlert},TestHandleSwitch_ClearsContradictingLaunchCmd,TestHandleAgentConfigModels_BackendChangeClearsContradictingLaunchCmd,TestSurfaceLaunchCmdDegradations{,_NilSafe}New functions:
ReconcileLaunchCmdForBackend100%,ValidateAgentLaunchCmds100%,validateSaveGuard100%,DegradeContradictingLaunchCmds93.8%,SaveAgentFile94.4%,reconcileLaunchCmdForBackend100%,surfaceLaunchCmdDegradations100%.