Skip to content

πŸ› Reconcile launch_cmd on backend switch and degrade instead of crash-looping on a base-config contradiction - #15750

Merged
hivecommons-hive[bot] merged 7 commits into
v5from
fix/backend-switch-clears-launch-cmd
Oct 30, 2026
Merged

hivecommons-hive[bot] merged 7 commits into
v5from
fix/backend-switch-clears-launch-cmd

Conversation

@clubanderson

Copy link
Copy Markdown
Member

Fixes #15749

Summary

A dashboard backend switch persisted backend: codex next to a stale launch_cmd: /usr/bin/bob …; the next pod restart failed config validation on the base config and the hive crash-looped for 18 hours (hosted hive hosted-available-oke-08-placeholder-xf07). This closes all three gaps:

  1. Backend switch reconciles launch_cmd. New GovernorConfig.ReconcileLaunchCmdForBackend reuses the validator's ValidateLaunchCmdBackend gate; Server.reconcileLaunchCmdForBackend calls it from claimAgentFieldOwnership (the /switch path), handleAgentConfigModels (PUT …/models with a backend) and the ACMM pack-apply backend moves. A launch_cmd that launches a different CLI is cleared with an INFO audit line (launch_cmd cleared on backend switch, agent, previous_launch_cmd, backend); a matching one, an empty one, or a wrapper with no opinion is kept.
  2. Save paths validate. Config.Save (validateSaveGuard β†’ new Config.ValidateAgentLaunchCmds) and config.SaveAgentFile refuse a backend/launch_cmd contradiction. claimAgentFieldOwnership already turned a save error into an agent-field-save-failed alert; handleAgentConfigModels now does too instead of log-only.
  3. Boot degrades instead of failing. loadWithOverrides calls new Config.DegradeContradictingLaunchCmds() before validation: a contradicting base-config launch_cmd is dropped, backend kept, logged at ERROR in the πŸ› A config contradiction bricks a hosted spoke with no supported recovery pathΒ #6024 overlay-rejection style, and recorded in Config.LaunchCmdDegradations. RegisterAPI surfaces one launch-cmd-degraded-<agent> system alert per entry. The πŸ› A config contradiction bricks a hosted spoke with no supported recovery pathΒ #6024 "fell back to base config" path remains for overlays but is no help when base has the same defect β€” this is the self-heal for that case. The fatal path is kept for any contradiction that clearing launch_cmd would not resolve.

Also: docs (agent-configuration.md, new "launch_cmd and backend" section) and a changelog fragment.

Test evidence

New tests (no time.Sleep):

  • pkg/config: TestReconcileLaunchCmdForBackend, TestLoadWithOverrides_BaseConfigContradictionDegradesInsteadOfFailing, TestDegradeContradictingLaunchCmds_SortedAndIdempotent, TestSave_RefusesBackendLaunchCmdContradiction, TestValidateAgentLaunchCmds, TestSaveAgentFile_RefusesBackendLaunchCmdContradiction
  • pkg/dashboard: TestClaimAgentFieldOwnership_{ClearsContradictingLaunchCmd,KeepsMatchingLaunchCmd,ModelOnlyLeavesLaunchCmd,SaveRefusalRaisesAlert}, TestHandleSwitch_ClearsContradictingLaunchCmd, TestHandleAgentConfigModels_BackendChangeClearsContradictingLaunchCmd, TestSurfaceLaunchCmdDegradations{,_NilSafe}
$ cd src && go test -cover ./pkg/config/ ./pkg/dashboard/
ok  github.com/hivecommons/hive/pkg/config     coverage: 90.1% of statements   (v5 baseline: 90.0%)
ok  github.com/hivecommons/hive/pkg/dashboard  coverage: 88.1% of statements   (v5 baseline: 88.1%)

New functions: ReconcileLaunchCmdForBackend 100%, ValidateAgentLaunchCmds 100%, validateSaveGuard 100%, DegradeContradictingLaunchCmds 93.8%, SaveAgentFile 94.4%, reconcileLaunchCmdForBackend 100%, surfaceLaunchCmdDegradations 100%.

…op on a base-config contradiction

The dashboard backend switch persisted backend: codex next to a stale
launch_cmd: /usr/bin/bob … (hive.yaml and the per-agent overlay). The
running process kept its in-memory state, so nothing failed until the next
pod restart, when config validation rejected the base config and the hive
crash-looped for 18 hours.

- claimAgentFieldOwnership, PUT …/models and ACMM pack backend moves now
  clear a launch_cmd that launches a different CLI than the new backend
  (GovernorConfig.ReconcileLaunchCmdForBackend, reusing the validator's
  detection) and log an INFO audit line; a matching launch_cmd is kept.
- Config.Save (validateSaveGuard) and SaveAgentFile refuse a
  backend/launch_cmd contradiction so the loader never meets one written by
  the dashboard; handleAgentConfigModels now surfaces save errors as a
  system alert instead of log-only.
- The loader (loadWithOverrides) drops a contradicting launch_cmd from the
  BASE config, keeps backend, logs ERROR in the #6024 overlay-rejection
  style and records the drop in Config.LaunchCmdDegradations; the dashboard
  raises a per-agent system alert at RegisterAPI. The #6024 'fell back to
  base config' path is no help when base has the same defect.
- Tests for the handler, save refusal and loader degradation; docs and
  changelog fragment.

Signed-off-by: Andy Anderson <andy@clubanderson.com>
@kubestellar-prow kubestellar-prow Bot added the dco-signoff: yes Indicates the PR's author has signed the DCO. label Oct 30, 2026
@kubestellar-prow

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign danathar for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@kubestellar-prow kubestellar-prow Bot added the size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. label Oct 30, 2026
@github-actions

github-actions Bot commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

Open PR overlap check

Base branch: v5

Conflicts are checked with every PR merged onto the current tip of v5, so a PR that only needs a rebase is not reported as conflicting with its siblings.

PR Title Relation Files
#15757 ✨ Auto-merge hive-authored superseding PRs without operator decision same files src/pkg/config/config.go
src/pkg/dashboard/api.go

@hivecommons-hive

Copy link
Copy Markdown
Contributor

@clubanderson this PR is held at stage/071-needs-author and the hive will not merge it yet.

Failing checks:

  • relative links and anchors resolve

Next: the hive does not push to your branch. Push a new head (or reply here); CI and the review swarm re-run automatically and the PR returns to the normal path.

@hivecommons-hive

Copy link
Copy Markdown
Contributor

@clubanderson this PR is held at stage/071-needs-author and the hive will not merge it yet.

Failing checks:

  • relative links and anchors resolve

Next: the hive does not push to your branch. Push a new head (or reply here); CI and the review swarm re-run automatically and the PR returns to the normal path.

Signed-off-by: Andy Anderson <andy@clubanderson.com>
@kubestellar-prow kubestellar-prow Bot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Oct 30, 2026
@hivecommons-hive

Copy link
Copy Markdown
Contributor

stage 080-awaiting-merge-admission β†’ 050-review-pending: check-pending

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed β€” no findings from this perspective.

β€” hive: agent=ci-maintainer backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed (correctness) β€” no in-scope blockers at 478e194. The three gaps in #15749 are closed where the issue says they are: claimAgentFieldOwnership (api_agent_control.go:208), handleAgentConfigModels (api_agent_config.go:994) and applyPack (api_packs.go:327) all reconcile before persisting; validateSaveGuard (config.go:7054) and SaveAgentFile (agent_overlay.go:279) refuse the contradiction; loadWithOverrides degrades before ExpandAgentReplicas/validation (config.go:5459), so replicas inherit the cleared launch_cmd. The pre-existing PUT /config path already validates before mutating in-memory state (api_agent_config.go:750), so a bad request cannot poison every later Save.

Two adjacent, non-blocking notes for follow-up:

  1. hive validate now prints config OK for a base-config contradiction. src/cmd/hive/config_check.go:49-58 only reports load failure and surviving provenance; the degradation is visible only in the slog ERROR line. Since this command exists precisely to answer "will this config boot as written?", consider surfacing cfg.LaunchCmdDegradations:
    	fmt.Fprintf(stdout, "config OK: %s\n", *configPath)
    +	for _, d := range cfg.LaunchCmdDegradations {
    +		fmt.Fprintf(stderr, "  WARNING: agent %s: launch_cmd %q dropped (contradicts backend %q)\n", d.Agent, d.LaunchCmd, d.Backend)
    +	}
  2. Hot reload does not raise the banner. surfaceLaunchCmdDegradations runs only from RegisterAPI (api.go:31); the watcher's LoadWithDashboardOverlay (pkg/config/watcher.go:137) will degrade and log, but the per-agent alert is not re-raised for a config hand-edited into contradiction after boot.

β€” hive: agent=adjudicator-4 backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive

Copy link
Copy Markdown
Contributor

Hive filed review findings that do not block this PR as backlog items:

Cap: 10 backlog item(s) per PR per day.

@hivecommons-hive

Copy link
Copy Markdown
Contributor

stage 080-awaiting-merge-admission β†’ 050-review-pending: check-pending

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed β€” no findings from this perspective.

β€” hive: agent=ci-maintainer backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed (correctness) β€” no blocker; one advisory gap between the docs/alert text and the code.

launch-cmd-degraded-<agent> alert is never cleared (low). src/pkg/dashboard/api_agent_control.go:246-259 raises the alert in surfaceLaunchCmdDegradations, and both the alert message (api_agent_control.go:257, "Save the agent once from the dashboard … to make this permanent") and src/docs/agent-configuration.md:727 ("save the agent once from the dashboard (or fix the file) to clear the alert") say a save clears it. Nothing calls ClearSystemAlert(launchCmdDegradedAlertPrefix+…) β€” launchCmdDegradedAlertPrefix has exactly one use (the AddSystemAlert). The banner stays until the next pod restart even after the operator does what it asks. Since a successful Config.Save() writes every agent (with the already-cleared launch_cmd) to hive.yaml and the dashboard overlay, the simplest honest fix is to drop all degraded banners after a save succeeds:

--- a/src/pkg/dashboard/api.go
+++ b/src/pkg/dashboard/api.go
@@ func (s *Server) saveConfig() error {
 	if err := s.deps.Config.Save(); err != nil {
 		return err
 	}
+	// The file on disk now carries the degraded (valid) launch_cmds, so the
+	// boot-time "dropped launch_cmd" banners no longer describe a drift.
+	for _, d := range s.deps.Config.LaunchCmdDegradations {
+		s.ClearSystemAlert(launchCmdDegradedAlertPrefix + d.Agent)
+	}

(or, if you prefer per-agent clearing, call it in claimAgentFieldOwnership/handleAgentConfigModels/handleAgentConfigGeneral after their saves, and reword the doc line). Either way the doc sentence should match what the code does.

Test note (info): the ACMM pack-apply reconcile at src/pkg/dashboard/api_packs.go:325-327 has no test; the other two backend-mutating paths do. It is the same helper, so low risk.

Verified: DegradeContradictingLaunchCmds runs after overlay merge and before validateWithOptions (src/pkg/config/config.go:5450-5471) so an accepted overlay cannot reintroduce the contradiction; handleAgentConfigGeneral's cliPinValue backend edit (api_agent_config.go:514) is already gated by the pre-existing 400 at api_agent_config.go:748-752; saveLocked β†’ validateSaveGuard guards both hive.yaml and the dashboard overlay (config.go:7140,7158). go build ./pkg/config/ ./pkg/dashboard/ passes at 9320180; go test is unavailable on this host, CI pending.

β€” hive: agent=adjudicator-4 backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Read the three layers against the tree rather than the diff alone; the mechanism holds together.

  • Reconcile is wired at every backend-writing site I could find. claimAgentFieldOwnership (api_agent_control.go:208, serving /switch), handleAgentConfigModels (api_agent_config.go:994) and the pack-apply move (api_packs.go:325) all call reconcileLaunchCmdForBackend before Config.Agents[name] is written back. The only other LaunchCmd writer in the dashboard, the full agent PUT at api_agent_config.go:444, already runs ValidateLaunchCmdBackend at :750 and 400s before saving β€” so the new validateSaveGuard β†’ ValidateAgentLaunchCmds gate cannot be tripped by an in-memory contradiction a dashboard path introduced. That matters because Save() is shared: a stray contradiction would now block persistence for every agent, and I verified no current writer leaves one behind.
  • SaveAgentFile with GovernorConfig{} only loses the gateway-name half of the gate (config.go:6640-6646, isGatewayName on a zero config); it cannot produce a false refusal, and the doc comment says so. Fine.
  • Boot degrade sits in the right slot: loadWithOverrides runs DegradeContradictingLaunchCmds after overlay merge + ApplyAgentDefaults and before validateWithOptions (config.go:5461-5471), and bails out of a drop when clearing would not satisfy the gate, so the fatal path survives for the cases it should. The dashboard-overlay path (loadWithDashboardOverlay, :5590) still goes through RejectInvalidAgentOverlays, so a contradiction persisted there is rejected rather than validated against.
  • go build ./pkg/config/ ./pkg/dashboard/ passes on the head; the api-reference churn is the citation rewrite the doc check requires.

One thing worth a follow-up, low severity:

  • launch-cmd-degraded-<agent> is never cleared. surfaceLaunchCmdDegradations raises it (api_agent_control.go:256) and nothing calls ClearSystemAlert(launchCmdDegradedAlertPrefix+name) β€” the only reference to the prefix is the raise. The banner tells the operator to "save the agent once from the dashboard to make this permanent"; when they do, Config.Agents[name].LaunchCmd is already "" so the save succeeds, yet the error banner stays up until the next restart. A ClearSystemAlert in reconcileLaunchCmdForBackend's caller success paths (or wherever the agent is next persisted cleanly) would close the loop the banner itself describes.

Looks correct to me otherwise.

β€” hive: agent=reviewer backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed (intent-alignment) β€” the diff closes all three gaps in #15749: reconcileLaunchCmdForBackend is wired into claimAgentFieldOwnership (src/pkg/dashboard/api_agent_control.go:208), handleAgentConfigModels (api_agent_config.go:994) and applyPack (api_packs.go:327); validateSaveGuard/SaveAgentFile gate the pair (config.go:7057, agent_overlay.go:279); and DegradeContradictingLaunchCmds runs before validateWithOptions in loadWithOverrides (config.go:5461). The 36 api-reference/*.md files are citation line-number refreshes only. One doc-accuracy nit:

Low β€” docs/changelog claim a 400 that this diff does not add. changelog.d/fixed-backend-switch-clears-launch-cmd.md:1 and src/docs/agent-configuration.md ("Saves refuse the contradiction … surfaced as a 400 and a dashboard alert") describe the new save guards as returning HTTP 400. Neither new refusal path does: claimAgentFieldOwnership (api_agent_control.go:212-224) and handleAgentConfigModels (api_agent_config.go:1029-1042) log, raise agent-field-save-failed, and still return 200. The only 400 is the pre-existing #5921 check in handleAgentConfigUpdate (api_agent_config.go:750). Suggest rewording so the operator-facing doc matches behavior:

- **Saves refuse the contradiction.** `hive.yaml` and per-agent overlay writes validate the pair and return an error (logged and surfaced as an `agent-field-save-failed` dashboard alert; `PUT /api/config/agent/{name}` additionally rejects it with a 400) rather than persisting a config the next boot would reject.

β€” hive: agent=adjudicator-4 backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Style/maintainability review (head 9320180)

1. The launch-cmd-degraded-<agent> alert is never cleared in-process, but the docs promise it is β€” src/pkg/dashboard/api_agent_control.go:244-259 is the only reference to launchCmdDegradedAlertPrefix; no ClearSystemAlert/ClearSystemAlertsExcept call uses it. src/docs/agent-configuration.md (new "launch_cmd and backend" section) says "save the agent once from the dashboard (or fix the file) to clear the alert", and the alert text itself says "Save the agent once from the dashboard … to make this permanent". After a successful save the banner stays until the next restart. Since Config.Save writes every agent, the cheapest correct place is saveConfig (src/pkg/dashboard/api.go:740), which already fans out post-save side effects:

--- a/src/pkg/dashboard/api.go
+++ b/src/pkg/dashboard/api.go
@@ func (s *Server) saveConfig() error {
 	if err := s.deps.Config.Save(); err != nil {
 		return err
 	}
+	// hive.yaml now matches the in-memory agents, so any launch_cmd the
+	// loader dropped at boot is permanently gone from disk.
+	s.ClearSystemAlertsExcept(launchCmdDegradedAlertPrefix, nil)
+	s.deps.Config.LaunchCmdDegradations = nil
 	if s.deps.Governor != nil {

(or reword the doc/alert to say the banner clears on the next restart).

2. Stale doc comment on ValidateAgentLaunchCmds β€” src/pkg/config/config.go:7063-7066 says "the dashboard's persistence paths (Save, SaveAgentFile) call it", but SaveAgentFile (src/pkg/config/agent_overlay.go:280) calls (GovernorConfig{}).ValidateLaunchCmdBackend directly and its own comment (agent_overlay.go:273-275) explains why it cannot run the gateway-aware half. Only validateSaveGuard calls it.

// ValidateAgentLaunchCmds runs the backend/launch_cmd gate over every agent.
// It is the write-side twin of the loader's check: Config.Save calls it (via
// validateSaveGuard) so a contradiction is refused at save time rather than
// discovered as a fatal error at boot. SaveAgentFile runs only the
// governor-independent half of the gate; see its doc comment.

Advisory (no change requested): reconcileLaunchCmdForBackend's bool return is unused at all three call sites (api_agent_config.go:994, api_agent_control.go:208, api_packs.go:327); DegradeContradictingLaunchCmds sorts its result but logs in map order β€” iterating sorted names (as ValidateAgentLaunchCmds does) would make the boot log deterministic. The 13 api-reference/*.md files are pure line-number citation churn (verified: every changed line pairs with its :N-normalised twin).

β€” hive: agent=ci-maintainer backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive
hivecommons-hive Bot merged commit d736590 into v5 Oct 30, 2026
73 of 74 checks passed
@kubestellar-prow
kubestellar-prow Bot deleted the fix/backend-switch-clears-launch-cmd branch October 30, 2026 14:18
@hivecommons-hive

Copy link
Copy Markdown
Contributor
Evidence bundle `#1575080615b2` sha256:3abc53bc2ce3d803fe3753bf159c7c857a7ae661d0d6f4caaa1ee51ee91f65e7 β€” download from the dashboard or `hivectl review evidence #15750`.

@hivecommons-hive

Copy link
Copy Markdown
Contributor

merged by Hive trusted-author auto-merge: author @clubanderson holds role owner and GitHub permission admin on repo; CI green at 80615b2.

@github-actions

Copy link
Copy Markdown
Contributor

Thank you for your contribution! Your PR has been merged.

We'd love to hear how your experience was: share feedback

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

dco-signoff: yes Indicates the PR's author has signed the DCO. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

πŸ› dashboard backend switch leaves a contradicting launch_cmd; next boot fails config validation and the hive crash-loops

1 participant