Skip to content

πŸ› fix: warn before boot recovery drops safety gates - #16446

Closed
thatguywob wants to merge 1 commit into
hivecommons:v5from
thatguywob:fix/16438-config-safety-gates
Closed

thatguywob wants to merge 1 commit into
hivecommons:v5from
thatguywob:fix/16438-config-safety-gates

Conversation

@thatguywob

Copy link
Copy Markdown
Contributor

Fixes #16438

Boot recovery could replace the runtime snapshot with a ConfigMap seed before reporting that safety gates had disappeared. Compare the selected boot config against surviving runtime, .runtime.bak, legacy .bak, and dashboard files before that write, and log each removed label gate, repository pause, or relaxed per-repo writeback mode.

The boot warnings remain visible in /api/status (configSafetyWarnings) and the Settings banner across config reloads and the entrypoint's privilege-drop re-exec. config.refuse_gate_regression: true refuses startup before snapshot replacement; unreadable or malformed prior files also produce a diagnostic and refuse startup in strict mode. hive config diff --config <path> [previous paths...] exposes the same comparison without starting services. Gate enforcement is unchanged.

Added Go regression coverage for all prior file names, partial loss, preserved/stronger gates, malformed baselines, strict-mode persistence and CLI dispatch, and status visibility across reloads. Extended the existing entrypoint recovery suite to assert strict refusal happens before the snapshot write and normal recovery preserves its warning. gofmt -l is clean for every changed Go file. Tests were not run locally, as required by AGENTS.md; CI supplies the test verdict. This has not been deployed.

β€” hive: backend=codex


🐝 Hive Agent: contributor | SHA: 96d5ad5db

Compare the selected boot config against surviving runtime, backup and
dashboard layers before replacing the runtime snapshot. Keep gate deltas
visible in status and settings, expose hive config diff, and support strict
startup refusal through config.refuse_gate_regression.

Hive-Run: hivecommons#16438
Hive-Plan: boot-safety-gate-regression
Hive-Spec: issue-16438#proposed-behaviour
Co-authored-by: clubanderson <407614+clubanderson@users.noreply.github.com>
Signed-off-by: thatguywob <thatguywob@users.noreply.github.com>
@kubestellar-prow kubestellar-prow Bot added the dco-signoff: yes Indicates the PR's author has signed the DCO. label Nov 3, 2026
@kubestellar-prow

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign kellyaa for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@kubestellar-prow kubestellar-prow Bot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Nov 3, 2026
@hivecommons-hive

Copy link
Copy Markdown
Contributor

@thatguywob this PR is back in your court (stage/072-returned-to-contributor). The hive will not merge it until the head changes.

Failing checks:

  • relative links and anchors resolve

Reply here or push a new head to this branch (same sign-off); CI and the review swarm re-run automatically and the PR returns to the normal path.

Your relay could not take this as a fix-pr task (Hub is not the active polling slot); the hive will retry it from your relay fix queue.

@hivecommons-hive

Copy link
Copy Markdown
Contributor
96d5ad5

⚠️ Sentinel alert β€” maintainer review required

Hive flagged this PR (author @thatguywob, head 96d5ad5db1cf) because it matches behaviors that can override security controls, escalate privileges or damage the codebase. This is a heuristic, not an accusation β€” a maintainer should confirm the change is intended before it merges.

  • sensitive_path β€” changes 2 sensitive path(s) (Touches a sensitive path (OWNERS, workflows, policies, hive config, security docs, …))
    • src/deploy/entrypoint.sh
    • src/deploy/test_entrypoint_runtime_config.sh

Hive added the sentinel-alert label. While it is present, Hive will not approve this PR, apply LGTM/approval labels, or merge it through any auto-merge lane. Remove the label once reviewed; Hive will not re-apply it unless new commits are pushed. Tune paths and behaviors under sentinel in hive.yaml or the dashboard Security tab.

@github-actions

github-actions Bot commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

Open PR overlap check

Base branch: v5

This PR conflicts with v5 itself. Rebase it first: until it merges cleanly into the base, conflicts with other open PRs cannot be told apart from base drift, so only same-file overlaps are listed.

Conflicts are checked with every PR merged onto the current tip of v5, so a PR that only needs a rebase is not reported as conflicting with its siblings.

PR Title Relation Files
#16443 πŸ› fix: keep Matrix overlays above terminal rendering same files src/pkg/dashboard/static/index.html
#16456 ✨ feat(github): enforce pin-only exception in merge-request relay same files src/pkg/config/config.go
#16454 πŸ› Restore activity monitor layout and compact chevrons same files src/pkg/dashboard/static/index.html

@github-actions github-actions Bot added the conflicts-with-open-pr Open PR has a merge conflict with another open PR label Nov 3, 2026

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Decision needed: @clubanderson should decide whether #16446 (entrypoint hive config diff before the snapshot write) or #16449 (Go-side boot check) is the implementation of #16438 that lands β€” they add the same config.refuse_gate_regression key with different Go fields, status shapes and prior-file sets, so both cannot merge.

Conclusion: the mechanism is sound and does what the body says, but this PR conflicts with #16449 (same issue, same YAML key, incompatible shapes), it carries sentinel-alert for touching src/deploy/entrypoint.sh (routing to a maintainer is required for that alone), and consulting the legacy /data/hive.yaml.bak makes the warning β€” and strict mode β€” fire on every boot of a migrated hive.

Conflict with #16449 (both Fixes #16438)

Read both diffs in full. Same intent, different change β€” not a duplicate, a conflict:

#16446 (this) #16449
config: block Go field Config.Config *ConfigLoadPolicy (config.go:77) Config.ConfigGuard ConfigGuardConfig
/api/status shape configSafetyWarnings: []string (server.go:554) config.gate_regressions: [{gate,previous,effective,source}]
where the compare runs entrypoint, before the cp at entrypoint.sh:1157, via new hive config diff Go boot, after that cp
prior files runtime, runtime.bak, legacy .bak, dashboard runtime, runtime.bak, dashboard

Whichever lands second will not apply (both are labelled conflicts-with-open-pr).

Finding β€” legacy .bak makes this fire forever on migrated hives (medium)

  • src/cmd/hive/config_diff.go:29 includes config.RuntimeConfigFileLegacy (/data/hive.yaml.bak) in the default prior set.
  • src/deploy/entrypoint.sh:1217 calls hive_check_safety_gates unconditionally on every boot, including the steady-state "boot from runtime config" path at entrypoint.sh:992.
  • The legacy file is never removed or refreshed: migration leaves it in place (entrypoint.sh:1189, "legacy left in place"), and the seed branch comment at :1155 says the same. It is frozen at the moment of the rename.

Consequence: on any hive that migrated from .bak and whose operator later removed a gate on purpose (e.g. un-paused a repo via the dashboard), every subsequent boot logs safety gate dropped … (previous: /data/hive.yaml.bak), and with config.refuse_gate_regression: true the pod refuses to start until someone deletes the legacy file. #16449 excludes the legacy file for exactly this reason. Dropping RuntimeConfigFileLegacy from the default list in config_diff.go:29 (keeping it available as a positional arg) would fix it.

What holds up

  • Warnings are exported as HIVE_BOOT_SAFETY_WARNINGS and read at status-build time (safety_gates.go:26, status_builder.go:395), so they survive the rootβ†’dev re-exec and config reloads β€” #16449 loses its result on the first reload.
  • hive_check_safety_gates is idempotent via the -z guard, so the second call at :1217 is a no-op after the seed-branch call at :1154.
  • Parser errors are deliberately not echoed (safety_gates.go:80) β€” good, overlays carry credentials.

Scoping caveat (same in #16449)

The seed branch is only reached when hive_runtime_config_read returned empty (entrypoint.sh:991), so the a1 scenario is caught only if the moved-aside overlay sits at hive.yaml.runtime.bak or the legacy name. "Before the snapshot write" does not by itself rescue a runtime file that is already gone.

CI is red on this head (docker, dashboard/rest test shards, coverage, changelog-fragment-guard); not triaged here. Fork PR β€” I cannot push fixes here.

β€” hive: agent=reviewer-2 backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive

hivecommons-hive Bot commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

Hive status β€” updated in place; edits do not notify. Full history is in the Activity Monitor.

Review findings from adjudicator-3 (folded here because this head already carries its hive review):

Security perspective (read at head 96d5ad5).

Strict mode is itself a safety gate the recovery path drops silently β€” src/pkg/config/safety_gates.go:66 and :122: SafetyGateDiff returns effective.Config.RefuseGateRegression, i.e. only the selected boot config's value. Because Config.Config is now part of the persisted struct (src/pkg/config/config.go:77) and dashboardOverlayBytes() marshals the whole Config (config.go:7609-7631), an operator who enables config.refuse_gate_regression: true via the dashboard/runtime has it written to hive.yaml.runtime / .dashboard β€” but not to the frozen ConfigMap seed. In precisely the #16438 scenario (runtime moved aside, boot from seed), the seed has no config: block, so strict is false, the helper in src/deploy/entrypoint.sh:968-973 only warns, and cp "$HIVE_CONFIG_PATH" "$HIVE_CONFIG_RUNTIME" at :1157 overwrites the layer that carried the refusal. The PR body's claim that strict mode "refuses startup before snapshot replacement" holds only when the seed itself opts in, which is the one layer the issue says operators cannot edit at runtime. No warning is emitted for the dropped flag either (it is not in the compared keys at :93-121).

Suggested fix β€” honour strict mode from any surviving prior layer, and report its loss like the other gates:

--- a/src/pkg/config/safety_gates.go
+++ b/src/pkg/config/safety_gates.go
@@ func SafetyGateDiff(path string, previous []string) ([]string, bool, error) {
 	warnings := []string{}
+	strict := effective.Config.RefuseGateRegression
 	for _, priorPath := range previous {
 		prior, err := readGateSnapshot(priorPath)
 		if os.IsNotExist(err) {
 			continue
 		}
 		if err != nil {
 			// Do not print parser errors: they can contain unrelated secret values.
 			warnings = append(warnings, fmt.Sprintf("safety gate comparison unavailable: %s (unreadable or invalid YAML)", priorPath))
 			continue
 		}
 		dropped := func(key string, was any) {
 			value, _ := json.Marshal(was)
 			warnings = append(warnings, fmt.Sprintf("safety gate dropped: %s (was %s) β€” effective config has none or a weaker restriction (previous: %s)", key, value, priorPath))
 		}
+		if prior.Config.RefuseGateRegression && !effective.Config.RefuseGateRegression {
+			dropped("config.refuse_gate_regression", true)
+			// A prior layer asked for refusal; the seed cannot silently waive it.
+			strict = true
+		}
 		if len(nonemptyLabels(prior.Review.RequireLabels)) > 0 && len(nonemptyLabels(effective.Review.RequireLabels)) == 0 {
@@
-	return warnings, effective.Config.RefuseGateRegression, nil
+	return warnings, strict, nil
 }

Plus a case in TestSafetyGateDiffRecovery where only the prior carries config: {refuse_gate_regression: true} and asserting strict == true.

Checked and found no issue: banner output is routed through escapeHtml (static-src/js/config-tabs.js:39); HIVE_BOOT_SAFETY_WARNINGS carries only gate keys/label/repo names, parser errors are not echoed (safety_gates.go:78-80); hive config diff is read-only and the helper is skipped on the privilege-drop re-exec once the env var is set.

Review findings from ci-maintainer (folded here because this head already carries its hive review):

Intent alignment vs #16438 (read at head 96d5ad5)

The three proposed behaviours land: per-gate WARN safety gate dropped: … on the boot log before the runtime snapshot copy (src/deploy/entrypoint.sh:1154 precedes the cp at :1157, and set -e at :2 makes the strict return 1 fatal), configSafetyWarnings in /api/status + Settings banner (status_builder.go:395, config-tabs.js:38-40), and hive config diff (main.go:7116). Gate enforcement is untouched. One in-scope gap:

Strict mode is read only from the effective config, so it cannot guard the scenario the issue describes β€” src/pkg/config/safety_gates.go:132 returns effective.Config.RefuseGateRegression; prior.Config is parsed (:34) but never consulted. In the K8s recovery branch the effective config is the ConfigMap seed, which the entrypoint itself documents as unwritable by spoke operators (entrypoint.sh:988-993: "spoke RBAC omits them entirely … The runtime config on the PVC is the only layer any component can actually write"). An operator who enables config.refuse_gate_regression: true via the dashboard gets it persisted to hive.yaml.runtime/.dashboard β€” exactly the layers that are moved aside or ignored in #16438 β€” so the flag silently vanishes along with the gates it was meant to protect, and startup proceeds with a warning only. The flag's own loss is also not reported.

Suggested fix: honor strict mode from any surviving prior layer and flag its removal like the other gates:

--- a/src/pkg/config/safety_gates.go
+++ b/src/pkg/config/safety_gates.go
@@ func SafetyGateDiff(path string, previous []string) ([]string, bool, error) {
 	warnings := []string{}
+	strict := effective.Config.RefuseGateRegression
 	for _, priorPath := range previous {
 		prior, err := readGateSnapshot(priorPath)
@@
 			warnings = append(warnings, fmt.Sprintf("safety gate dropped: %s (was %s) β€” effective config has none or a weaker restriction (previous: %s)", key, value, priorPath))
 		}
+		if prior.Config.RefuseGateRegression && !effective.Config.RefuseGateRegression {
+			dropped("config.refuse_gate_regression", true)
+			strict = true
+		}
 		if len(nonemptyLabels(prior.Review.RequireLabels)) > 0 && len(nonemptyLabels(effective.Review.RequireLabels)) == 0 {
@@
-	return warnings, effective.Config.RefuseGateRegression, nil
+	return warnings, strict, nil
 }

and adjust the doc comment at :52 ("The bool is the effective config's strict-mode setting") plus the hive.yaml.example:1-5 note accordingly. A test asserting strict == true when only the prior file sets the flag would cover it.

Review findings from adjudicator-2 (folded here because this head already carries its hive review):

Correctness review (head 96d5ad5, read against merge-base db58e1a). Two in-scope issues, both small to fix.

1. Default prior list includes the frozen legacy /data/hive.yaml.bak β†’ permanent false warnings, and strict mode can never boot on migrated hives

src/cmd/hive/config_diff.go:29 always compares against config.RuntimeConfigFileLegacy. At merge-base that file is documented as "READ as a fallback and never written, renamed or removed" (src/pkg/config/config.go:7446-7454), and the entrypoint only consults it when hive.yaml.runtime is absent (src/deploy/entrypoint.sh:750-756). On any migrated hive it is a snapshot frozen at migration day. Every routine unpause (POST /api/repos/resume, src/pkg/dashboard/api.go:335-338), per-repo writeback relaxation, or label-gate removal made since then will be reported on every boot as safety gate dropped … (previous: /data/hive.yaml.bak), the Settings banner will say "Safety gates dropped at startup" forever, and with config.refuse_gate_regression: true the hive refuses to start until the operator deletes a file the codebase says must not be touched. That turns the signal #16438 asks for into noise and makes strict mode unusable on ~51 hives.

Mirror hive_runtime_config_read: consult the legacy name only when the new runtime file does not exist.

	previous := fs.Args()
	if len(previous) == 0 {
		previous = []string{config.RuntimeConfigFile, config.RuntimeConfigFile + ".bak", config.DashboardOverlayFile}
		// The legacy name is frozen at migration and never rewritten; only
		// treat it as "prior config" when it is the file the hive would boot from.
		if _, err := os.Stat(config.RuntimeConfigFile); os.IsNotExist(err) {
			previous = append(previous, config.RuntimeConfigFileLegacy)
		}
	}

(Aside: nothing in the tree writes hive.yaml.runtime.bak; keeping it in the list is harmless but it is a name this PR invents, not one the issue or Save() produces.)

2. Strict mode is read only from the effective config, so a ConfigMap reseed silently drops the strict flag itself

src/pkg/config/safety_gates.go:118 returns effective.Config.RefuseGateRegression. An operator who sets config.refuse_gate_regression: true on a running hive persists it to hive.yaml.runtime / .dashboard β€” the ConfigMap seed is frozen at provisioning and will not carry it. In exactly the recovery path the issue describes (runtime moved aside, boot from seed), the comparator sees the prior file had strict mode but the seed does not, logs the dropped gates, and then starts anyway because strict is false. The guard defeats itself in its target scenario.

	strict := effective.Config.RefuseGateRegression
	warnings := []string{}
	for _, priorPath := range previous {
		prior, err := readGateSnapshot(priorPath)
		if os.IsNotExist(err) {
			continue
		}
		if err != nil {
			// Do not print parser errors: they can contain unrelated secret values.
			warnings = append(warnings, fmt.Sprintf("safety gate comparison unavailable: %s (unreadable or invalid YAML)", priorPath))
			continue
		}
		// Strict mode set in a prior layer must survive the reseed it guards against.
		strict = strict || prior.Config.RefuseGateRegression

and return warnings, strict, nil at the end of SafetyGateDiff. Add a case to TestSafetyGateDiffInvalidBaselineAndStrictPersistence where only the prior file sets the flag and assert strict == true.

Everything else I traced checks out: the first hive_check_safety_gates call (entrypoint.sh:1154) does run before the cp … "$HIVE_CONFIG_RUNTIME" snapshot in the recovery branch; gosu/setpriv re-exec preserves the exported env so the guard short-circuits the second pass; gateSnapshot field shapes match RepoPause, writeback.RepoOverride and the require_labels slices; escapeHtml/window._lastStatus are in scope for config-tabs.js. Tests not run locally (CI supplies the verdict).

Review findings from reviewer (folded here because this head already carries its hive review):

Conclusion: this PR and #16449 are the same fix for #16438 and conflict rather than stack: both add config.refuse_gate_regression, both log WARN safety gate dropped: … for the same four gates, both add a Settings banner, and both add a config field to config.Config under different Go names (Config *ConfigLoadPolicy here, ConfigGuard ConfigGuardConfig there) and different /api/status shapes (configSafetyWarnings[] vs config.gate_regressions[]). Only one can land. I have not closed anything; this carries sentinel-alert and edits src/deploy/entrypoint.sh from a fork, so the trust call is a maintainer's.

What this PR gets right that #16449 does not: the comparison runs before cp "$HIVE_CONFIG_PATH" "$HIVE_CONFIG_RUNTIME" (src/deploy/entrypoint.sh:1154 β†’ :1157). That is the only point at which a moved-aside or truncated runtime overlay can still be compared; #16449 compares in Go after that copy has already replaced hive.yaml.runtime with the seed.

Verified defects (CI already shows the first):

  1. hive_check_safety_gates turns any hive config diff failure into a boot abort, strict mode or not β€” src/deploy/entrypoint.sh:969-974. The file runs under set -e (entrypoint.sh:2), and HIVE_BOOT_SAFETY_WARNINGS="$(hive config diff …)" || return 1 propagates exit 1 for every non-zero exit, not just the strict refusal: unreadable effective config (config_diff.go:33-35), flag-parse error, or no hive on PATH. The existing entrypoint boot suite shows exactly this: TestK8sFirstBootUsesTheSeed and TestEntrypointHardensRuntimeConfigItRecreates/k8s_seeds_runtime_from_ConfigMap fail on this head with "the prelude did not create the runtime config" / "first boot did not fall back to the seed" (entrypoint_boot_test.go:182, :233, job 126061225390) β€” the hook aborted before the snapshot write. Severity high: a non-strict operator who hits any diff error gets a crash-looping spoke instead of a warning. The non-strict path should swallow errors (|| true, log, continue) and only the explicit strict refusal should be fatal.

  2. Legacy /data/hive.yaml.bak is in the default baseline list β€” src/cmd/hive/config_diff.go:28 and the unconditional second call at entrypoint.sh:1217. hive_runtime_config_read (entrypoint.sh:750-756) only falls back to the legacy file when hive.yaml.runtime is absent, and the entrypoint comment at the snapshot write says the legacy file is "left untouched on the PVC". So on every steady-state boot the live runtime config is compared against a file frozen at the rename, and any gate an operator has removed on purpose since then warns on every boot β€” and with refuse_gate_regression: true refuses every boot. Severity medium. πŸ› fix(config): warn, report and optionally refuse when a seed boot drops a safety gate the previous runtime overlay carriedΒ #16449 deliberately excludes this file for that reason (gate_regression.go:75-77).

Also red on this head: changelog-fragment-guard is the api-reference citation drift from editing pkg/dashboard/server.go (bash src/scripts/check-api-reference-citations.sh --fix), and coverage/test carry the two entrypoint test failures above.

One question I could not settle: whether a maintainer wants the comparison in the entrypoint (this PR, earlier in the boot, extra hive config diff subcommand) or in bootConfigWith (#16449, no shell surface, but blind to the overwritten overlay). A combination β€” #16449's detector/docs with this PR's pre-snapshot placement β€” would cover the incident as #16438 describes it.

@hivecommons-hive

Copy link
Copy Markdown
Contributor

⚠️ Hold-gate mergeability: this PR has a real conflict with its base

This PR is hold-gated and has fallen behind its base branch. The hive tried to merge the base in to keep it mergeable while it waits on review, but that merge conflicts and the hive cannot resolve it automatically.

It will stay held and will NOT merge on its own even once reviewed β€” removing hold alone will not be enough. Someone will need to merge or rebase the base branch in by hand (or resolve the conflict another way) before this PR can merge.

@github-actions github-actions Bot removed the conflicts-with-open-pr Open PR has a merge conflict with another open PR label Nov 3, 2026
@hivecommons-hive

Copy link
Copy Markdown
Contributor

@thatguywob the hive has adopted this PR (stage/074-adopted-by-hive): CI stayed red for 2h after the nudge and the relay nudge went unanswered. The hive will rebase this branch on v5 and fix CI here; your commits and sign-off stay. Push a new head yourself at any time and the hive yields it back to you.

@hivecommons-hive

Copy link
Copy Markdown
Contributor

Closing: #16438 was closed by the maintainer after #16449 (merged) landed the same boot-time safety-gate regression warning on v5 (gate_regressions in status + renderGateRegressionBanner() in the Settings tab, plus config.refuse_gate_regression). Rebasing this branch onto v5 conflicts with that merged change in config-tabs.js/index.html, so there is nothing left for this PR to add.


🐝 Hive Agent: scanner | Instance: hosted-available-oke-11-placeholder-r05x | SHA: 55318dc61

β€” hive: agent=scanner backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive

Copy link
Copy Markdown
Contributor

Superseded by hive: re-authored as #16458 with the same commits and sign-off because the hive could not finish it on the contributor's branch. This PR is closed; if #16458 fails, hive reopens it.

@hivecommons-hive hivecommons-hive Bot added the hive/superseded Closed by Hive: a different merged PR already landed this change label Nov 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

dco-signoff: yes Indicates the PR's author has signed the DCO. hive/superseded Closed by Hive: a different merged PR already landed this change sentinel-alert size/L Denotes a PR that changes 100-499 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Boot from ConfigMap seed silently drops safety gates present in the prior runtime overlay

1 participant