Skip to content

πŸ› fix(config): warn, report and optionally refuse when a seed boot drops a safety gate the previous runtime overlay carried - #16449

Merged
clubanderson merged 3 commits into
v5from
fix-16438
Nov 3, 2026
Merged

clubanderson merged 3 commits into
v5from
fix-16438

Conversation

@clubanderson

Copy link
Copy Markdown
Member

Fixes #16438

What

When a spoke boots from the ConfigMap seed instead of /data/hive.yaml.runtime (overlay moved aside during a recovery, truncated, or never written), safety gates present in the prior overlay vanished with no log line. On the a1 spoke that re-enabled reviews on every unlabelled PR for ~4 hours.

Change

  • Detector (pkg/config/gate_regression.go): DetectGateRegressions(effective, prior) compares review.require_labels, project.issue_filter.require_labels (non-empty β†’ empty), project.paused_repos (paused repo no longer paused, org-qualified/case-folded match) and writeback.per_repo (restrictive mode β†’ looser mode or no override). DetectPriorOverlayGateRegressions reads hive.yaml.runtime, hive.yaml.runtime.bak, hive.yaml.dashboard (skipping the file the effective config was loaded from; empty/malformed ignored; first overlay wins per gate). Legacy hive.yaml.bak is deliberately not consulted β€” it is frozen at the rename and would warn forever about gates removed on purpose.
  • Boot (cmd/hive/main.go): one WARN safety gate dropped: <gate> (was <value>) β€” effective config has <value> per gate; result kept on cfg.GateRegressions.
  • Knob config.refuse_gate_regression (bool, default false): refuse to start with exit code 21 and the gates named.
  • Status: /api/status β†’ config.gate_regressions: [] ({gate, previous, effective, source}).
  • Dashboard: .cmp-banner warning at the top of every Settings tab while the list is non-empty (static-src/js/config-tabs.js, index.html regenerated).
  • Docs: operator-reference notable fields, config-layering, api-reference status row; entrypoint comment at the seed-boot write; changelog fragment.

No change to how the gates are enforced or to which file the entrypoint boots from.

Tests

  • pkg/config/gate_regression_test.go: table test (unchanged, no prior gates, all dropped, weakened/tightened per-repo mode, dropped full override, label list changed but non-empty, pause spelling match, nil), prior-overlay file reader (missing, empty/malformed, source-path skip, first-wins/merge), error message.
  • cmd/hive/boot_config_deps_test.go: warns-and-boots, refuses with exit 21, quiet when clean.

…ops a safety gate the previous runtime overlay carried

A spoke booting from the ConfigMap seed (runtime overlay moved aside,
truncated or never written) silently lost review.require_labels,
project.paused_repos, writeback.per_repo and
project.issue_filter.require_labels. The boot now compares the effective
config against the previous overlays still on the PVC
(hive.yaml.runtime, hive.yaml.runtime.bak, hive.yaml.dashboard), logs one
'WARN safety gate dropped' per gate with the previous value, serves the
list on /api/status as config.gate_regressions, shows a Settings banner,
and with config.refuse_gate_regression: true refuses to start (exit 21).

Fixes #16438

Signed-off-by: Andrew Anderson <andy@clubanderson.com>
@kubestellar-prow kubestellar-prow Bot added the dco-signoff: yes Indicates the PR's author has signed the DCO. label Nov 3, 2026
@kubestellar-prow

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign kellyaa for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@kubestellar-prow kubestellar-prow Bot added the size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. label Nov 3, 2026
@github-actions

github-actions Bot commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

Open PR overlap check

Base branch: v5

Conflicts are checked with every PR merged onto the current tip of v5, so a PR that only needs a rebase is not reported as conflicting with its siblings.

PR Title Relation Files
#16454 πŸ› Restore activity monitor layout and compact chevrons same files src/pkg/dashboard/static/index.html
#16446 πŸ› fix: warn before boot recovery drops safety gates CONFLICT src/pkg/dashboard/static-src/js/config-tabs.js
src/pkg/dashboard/static/index.html
#16398 ✨ feat(github): enforce pin-only exception in merge-request relay same files src/docs/operator-reference/README.md
src/pkg/config/config.go
#16443 πŸ› fix: keep Matrix overlays above terminal rendering same files src/pkg/dashboard/static/index.html

@github-actions github-actions Bot added the conflicts-with-open-pr Open PR has a merge conflict with another open PR label Nov 3, 2026
@hivecommons-hive

Copy link
Copy Markdown
Contributor
f54a5fa

ℹ️ Sentinel notice β€” informational; author is trusted (owner), merge is not blocked

Hive flagged this PR (author @clubanderson, head f54a5fa72023) because it matches behaviors that can override security controls, escalate privileges or damage the codebase. This is a heuristic, not an accusation β€” a maintainer should confirm the change is intended before it merges.

  • sensitive_path β€” changes 1 sensitive path(s) (Touches a sensitive path (OWNERS, workflows, policies, hive config, security docs, …))
    • src/deploy/entrypoint.sh

Hive did not add the sentinel-alert label because this author is trusted. The finding remains recorded for audit and the dashboard Security tab; set sentinel.trusted_authors_block: true to require the blocking label for trusted authors too.

…with agent_sandbox_gate_test

Signed-off-by: Andrew Anderson <andy@clubanderson.com>

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Decision needed: @clubanderson should decide whether #16449 (Go-side boot check) or #16446 (entrypoint hive config diff before the snapshot write) is the implementation of #16438 that lands β€” they add the same config.refuse_gate_regression key with different Go fields, status shapes and prior-file sets, so both cannot merge.

Conclusion: correct detector, but this PR conflicts with #16446 (same issue, same YAML key, incompatible shapes), and the boot warning it records does not survive the first config reload. A maintainer needs to pick one implementation.

Conflict with #16446 (both Fixes #16438)

Read both diffs in full. Same intent, not the same change:

#16449 (this) #16446
config: block Go field Config.ConfigGuard ConfigGuardConfig (config.go:215) Config.Config *ConfigLoadPolicy
/api/status shape config.gate_regressions: [{gate,previous,effective,source}] (server.go:3745) configSafetyWarnings: []string
where the compare runs Go boot, after the entrypoint has already cp'd the seed over hive.yaml.runtime entrypoint, before that cp, via new hive config diff
prior files runtime, runtime.bak, dashboard + legacy /data/hive.yaml.bak

Both carry conflicts-with-open-pr; whichever lands second will not apply. Neither should be judged "the duplicate" β€” they are competing designs.

Finding β€” boot warning lost on first config reload (medium)

  • src/cmd/hive/main.go:668 sets b.cfg.GateRegressions once at boot.
  • The config watcher callback at src/cmd/hive/main.go:3771 does *b.cfg = *newCfg. newCfg comes from LoadWithDashboardOverlay, which never populates GateRegressions, and the carry-over list just above it (HiveID, ReviewerAutoscaleActive at main.go:3735, ACMMLevel, tombstones) does not include it.
  • src/pkg/config/snapshot.go:17-20 likewise copies ReviewerAutoscaleActive and LaunchCmdDegradations but not GateRegressions.

Consequence: any hive.yaml save (a dashboard Settings edit triggers the watcher) empties /api/status config.gate_regressions and the Settings banner, so the operator who is most likely to be in Settings is the one who stops seeing it. The WARN log lines remain. Fix is one line next to main.go:3735 (newCfg.GateRegressions = b.cfg.GateRegressions) plus the same in Snapshot(). #16446 avoids this by carrying the result in an exported env var read at status-build time.

Design note (not a defect)

Excluding legacy /data/hive.yaml.bak (gate_regression.go PriorOverlayPaths) is the right call: the entrypoint leaves that file in place at migration (entrypoint.sh:1189) so it is frozen; #16446 does consult it and would warn on every steady-state boot on a migrated hive whose operator removed a gate on purpose.

Body vs diff

Body matches the diff. One scoping caveat worth stating in the issue: in the seed-boot branch the entrypoint only gets there when hive_runtime_config_read returned empty (entrypoint.sh:991), i.e. hive.yaml.runtime was already missing/empty β€” so, as the entrypoint comment in this PR says, the a1 scenario is caught only if the operator parked the overlay as hive.yaml.runtime.bak. Same limitation in #16446.

CI is red on this head (build-go, golangci-lint, test); not triaged here.

β€” hive: agent=reviewer-2 backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive

Copy link
Copy Markdown
Contributor

Hive status β€” updated in place; edits do not notify. Full history is in the Activity Monitor.

hive/awaiting-ci: pushed a90850c β€” adds the new config block to agent_runtime_golden.json (TestAgentRuntimeConfigGolden) and moves the gate-regression banner's inline margin into a ul.cmp-banner-detail rule (TestStyleRatchet baseline 1403). CI pending β€” sweep will merge when green.


🐝 Hive Agent: scanner | Instance: hosted-available-oke-11-placeholder-r05x | SHA: unknown

β€” hive: agent=scanner backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive

Copy link
Copy Markdown
Contributor

@clubanderson this PR is back in your court (stage/071-needs-author). Its commits are signed off as the hive App rather than as you, so a squash merge would misattribute them and the sign-off attribution check stays red. No hive agent can rewrite a contributor's commits β€” the fix is yours:

  • 1 commit is signed off as the hive App instead of you, so a squash merge would misattribute it β€” set git config user.email to your own address, then run git rebase --exec 'git commit --amend --no-edit --reset-author -s' HEAD~3 && git push --force-with-lease

Once the new head lands, CI and the review swarm re-run automatically and the PR returns to the normal path.

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Decision needed: @clubanderson should decide whether #16449 or #16446 (both Fixes #16438, same config.refuse_gate_regression knob) survives, and whether the surviving detector must run before the entrypoint's cp seed β†’ hive.yaml.runtime (entrypoint.sh:1146), which this PR's Go-side check runs after.

Conclusion: the detector itself reads correctly (DetectGateRegressions tightening/loosening logic in gate_regression.go:135-230, first-overlay-wins dedupe at :88-103, source-path skip via os.SameFile), the boot wiring is the existing bootConfigDeps pattern with its own exit code, and the docs are complete. Two things a maintainer should weigh before merging.

1. This duplicates and conflicts with #16446. Both PRs Fixes #16438, add config.refuse_gate_regression, log WARN safety gate dropped: … for the same four gates and add a Settings banner. They add a config: block to config.Config under different Go field names (ConfigGuard ConfigGuardConfig here, Config *ConfigLoadPolicy there) and expose different /api/status shapes (config.gate_regressions[] vs configSafetyWarnings[]). Only one can land; I have not closed anything.

2. The check runs after the entrypoint has already overwritten the evidence. The scenario in #16438 and in this PR body is "runtime overlay moved aside during a recovery, truncated, or never written". In that case hive_runtime_config_read (src/deploy/entrypoint.sh:750-756) returns nothing, the first-boot branch runs, and cp "$HIVE_CONFIG_PATH" "$HIVE_CONFIG_RUNTIME" (entrypoint.sh:1146 at base) replaces /data/hive.yaml.runtime with the seed before the Go binary starts. PriorOverlayPaths() (gate_regression.go:75-77) then compares against:

  • hive.yaml.runtime β€” now identical to the seed, so no regression;
  • hive.yaml.runtime.bak β€” nothing in the entrypoint or Go tree writes this name (grep of entrypoint.sh finds no runtime.bak); it exists only if an operator chose it by hand, which the new entrypoint comment (:1152-1153) asks for but cannot enforce;
  • hive.yaml.dashboard β€” the only baseline that survives the copy, and only when the dropped gates were last written through a dashboard save.

So in the truncated-overlay case, or when a gate was set via a path that did not refresh the dashboard overlay, this PR logs nothing and refuse_gate_regression: true does not refuse. #16446 places the same comparison in the entrypoint immediately before that cp (entrypoint.sh:1154 in its diff), which is the last moment the prior overlay is still readable β€” though its hook has its own defects (noted there). A combination β€” this detector and docs, invoked before the snapshot write β€” would cover #16438 as filed. If the dashboard-overlay baseline is judged sufficient for the real incident (a1 spoke), the PR body should say so explicitly, because the operator-reference text currently promises detection "typically [on] a boot from the ConfigMap seed after the overlay was moved aside" without that caveat.

Open question, not a defect: does the hub heartbeat write gates into hive.yaml.runtime without a dashboard save? If yes, the overlay baseline misses those too.

β€” hive: agent=reviewer backend=copilot model=claude-fable-5.1 copilot=1.0.88

…e banner margin into CSS

TestAgentRuntimeConfigGolden saw the new config: block as an unexpected key;
TestStyleRatchet counted the banner's inline style= attribute above baseline.

Signed-off-by: Andrew Anderson <andy@clubanderson.com>
@hivecommons-hive

Copy link
Copy Markdown
Contributor

Needs a decision, not a merge:

Head: verdict requires_human @3c33f46, 17/25 checks green, 8 pending, DCO yes, soak 0s of 10m0s.

How to release it:

  • After reading the verdict, a trusted merger can submit a GitHub Approve review for the current head and the lane clears the hold on its next pass; /hive approve replies on a PR are acknowledged but not consumed.

Clear the cause and the hive re-evaluates on its next pass; the stage moves to stage/090-ready-for-human-merge, stage/095-pre-merge-soak or the merge queue on its own. Approval does not bypass CI, hold labels or other merge gates.

@clubanderson
clubanderson merged commit 8049c14 into v5 Nov 3, 2026
91 of 92 checks passed
@kubestellar-prow
kubestellar-prow Bot deleted the fix-16438 branch November 3, 2026 17:27
@github-actions

github-actions Bot commented Nov 3, 2026

Copy link
Copy Markdown
Contributor

Thank you for your contribution! Your PR has been merged.

We'd love to hear how your experience was: share feedback

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

conflicts-with-open-pr Open PR has a merge conflict with another open PR dco-signoff: yes Indicates the PR's author has signed the DCO. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Boot from ConfigMap seed silently drops safety gates present in the prior runtime overlay

1 participant