Repository navigation
π fix(config): warn, report and optionally refuse when a seed boot drops a safety gate the previous runtime overlay carried - #16449
Conversation
β¦ops a safety gate the previous runtime overlay carried A spoke booting from the ConfigMap seed (runtime overlay moved aside, truncated or never written) silently lost review.require_labels, project.paused_repos, writeback.per_repo and project.issue_filter.require_labels. The boot now compares the effective config against the previous overlays still on the PVC (hive.yaml.runtime, hive.yaml.runtime.bak, hive.yaml.dashboard), logs one 'WARN safety gate dropped' per gate with the previous value, serves the list on /api/status as config.gate_regressions, shows a Settings banner, and with config.refuse_gate_regression: true refuses to start (exit 21). Fixes #16438 Signed-off-by: Andrew Anderson <andy@clubanderson.com>
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
Open PR overlap checkBase branch: Conflicts are checked with every PR merged onto the current tip of
|
f54a5fa
βΉοΈ Sentinel notice β informational; author is trusted (owner), merge is not blockedHive flagged this PR (author @clubanderson, head
Hive did not add the |
β¦with agent_sandbox_gate_test Signed-off-by: Andrew Anderson <andy@clubanderson.com>
There was a problem hiding this comment.
Decision needed: @clubanderson should decide whether #16449 (Go-side boot check) or #16446 (entrypoint hive config diff before the snapshot write) is the implementation of #16438 that lands β they add the same config.refuse_gate_regression key with different Go fields, status shapes and prior-file sets, so both cannot merge.
Conclusion: correct detector, but this PR conflicts with #16446 (same issue, same YAML key, incompatible shapes), and the boot warning it records does not survive the first config reload. A maintainer needs to pick one implementation.
Conflict with #16446 (both Fixes #16438)
Read both diffs in full. Same intent, not the same change:
| #16449 (this) | #16446 | |
|---|---|---|
config: block Go field |
Config.ConfigGuard ConfigGuardConfig (config.go:215) |
Config.Config *ConfigLoadPolicy |
/api/status shape |
config.gate_regressions: [{gate,previous,effective,source}] (server.go:3745) |
configSafetyWarnings: []string |
| where the compare runs | Go boot, after the entrypoint has already cp'd the seed over hive.yaml.runtime |
entrypoint, before that cp, via new hive config diff |
| prior files | runtime, runtime.bak, dashboard |
+ legacy /data/hive.yaml.bak |
Both carry conflicts-with-open-pr; whichever lands second will not apply. Neither should be judged "the duplicate" β they are competing designs.
Finding β boot warning lost on first config reload (medium)
src/cmd/hive/main.go:668setsb.cfg.GateRegressionsonce at boot.- The config watcher callback at
src/cmd/hive/main.go:3771does*b.cfg = *newCfg.newCfgcomes fromLoadWithDashboardOverlay, which never populatesGateRegressions, and the carry-over list just above it (HiveID,ReviewerAutoscaleActiveatmain.go:3735,ACMMLevel, tombstones) does not include it. src/pkg/config/snapshot.go:17-20likewise copiesReviewerAutoscaleActiveandLaunchCmdDegradationsbut notGateRegressions.
Consequence: any hive.yaml save (a dashboard Settings edit triggers the watcher) empties /api/status config.gate_regressions and the Settings banner, so the operator who is most likely to be in Settings is the one who stops seeing it. The WARN log lines remain. Fix is one line next to main.go:3735 (newCfg.GateRegressions = b.cfg.GateRegressions) plus the same in Snapshot(). #16446 avoids this by carrying the result in an exported env var read at status-build time.
Design note (not a defect)
Excluding legacy /data/hive.yaml.bak (gate_regression.go PriorOverlayPaths) is the right call: the entrypoint leaves that file in place at migration (entrypoint.sh:1189) so it is frozen; #16446 does consult it and would warn on every steady-state boot on a migrated hive whose operator removed a gate on purpose.
Body vs diff
Body matches the diff. One scoping caveat worth stating in the issue: in the seed-boot branch the entrypoint only gets there when hive_runtime_config_read returned empty (entrypoint.sh:991), i.e. hive.yaml.runtime was already missing/empty β so, as the entrypoint comment in this PR says, the a1 scenario is caught only if the operator parked the overlay as hive.yaml.runtime.bak. Same limitation in #16446.
CI is red on this head (build-go, golangci-lint, test); not triaged here.
β hive: agent=reviewer-2 backend=copilot model=claude-fable-5.1 copilot=1.0.88
|
Hive status β updated in place; edits do not notify. Full history is in the Activity Monitor. hive/awaiting-ci: pushed a90850c β adds the new π Hive Agent: β hive: agent=scanner backend=copilot model=claude-fable-5.1 copilot=1.0.88 |
|
@clubanderson this PR is back in your court (
Once the new head lands, CI and the review swarm re-run automatically and the PR returns to the normal path. |
There was a problem hiding this comment.
Decision needed: @clubanderson should decide whether #16449 or #16446 (both Fixes #16438, same config.refuse_gate_regression knob) survives, and whether the surviving detector must run before the entrypoint's cp seed β hive.yaml.runtime (entrypoint.sh:1146), which this PR's Go-side check runs after.
Conclusion: the detector itself reads correctly (DetectGateRegressions tightening/loosening logic in gate_regression.go:135-230, first-overlay-wins dedupe at :88-103, source-path skip via os.SameFile), the boot wiring is the existing bootConfigDeps pattern with its own exit code, and the docs are complete. Two things a maintainer should weigh before merging.
1. This duplicates and conflicts with #16446. Both PRs Fixes #16438, add config.refuse_gate_regression, log WARN safety gate dropped: β¦ for the same four gates and add a Settings banner. They add a config: block to config.Config under different Go field names (ConfigGuard ConfigGuardConfig here, Config *ConfigLoadPolicy there) and expose different /api/status shapes (config.gate_regressions[] vs configSafetyWarnings[]). Only one can land; I have not closed anything.
2. The check runs after the entrypoint has already overwritten the evidence. The scenario in #16438 and in this PR body is "runtime overlay moved aside during a recovery, truncated, or never written". In that case hive_runtime_config_read (src/deploy/entrypoint.sh:750-756) returns nothing, the first-boot branch runs, and cp "$HIVE_CONFIG_PATH" "$HIVE_CONFIG_RUNTIME" (entrypoint.sh:1146 at base) replaces /data/hive.yaml.runtime with the seed before the Go binary starts. PriorOverlayPaths() (gate_regression.go:75-77) then compares against:
hive.yaml.runtimeβ now identical to the seed, so no regression;hive.yaml.runtime.bakβ nothing in the entrypoint or Go tree writes this name (grep ofentrypoint.shfinds noruntime.bak); it exists only if an operator chose it by hand, which the new entrypoint comment (:1152-1153) asks for but cannot enforce;hive.yaml.dashboardβ the only baseline that survives the copy, and only when the dropped gates were last written through a dashboard save.
So in the truncated-overlay case, or when a gate was set via a path that did not refresh the dashboard overlay, this PR logs nothing and refuse_gate_regression: true does not refuse. #16446 places the same comparison in the entrypoint immediately before that cp (entrypoint.sh:1154 in its diff), which is the last moment the prior overlay is still readable β though its hook has its own defects (noted there). A combination β this detector and docs, invoked before the snapshot write β would cover #16438 as filed. If the dashboard-overlay baseline is judged sufficient for the real incident (a1 spoke), the PR body should say so explicitly, because the operator-reference text currently promises detection "typically [on] a boot from the ConfigMap seed after the overlay was moved aside" without that caveat.
Open question, not a defect: does the hub heartbeat write gates into hive.yaml.runtime without a dashboard save? If yes, the overlay baseline misses those too.
β hive: agent=reviewer backend=copilot model=claude-fable-5.1 copilot=1.0.88
β¦e banner margin into CSS TestAgentRuntimeConfigGolden saw the new config: block as an unexpected key; TestStyleRatchet counted the banner's inline style= attribute above baseline. Signed-off-by: Andrew Anderson <andy@clubanderson.com>
a90850c to
3c33f46
Compare
|
Needs a decision, not a merge:
Head: verdict requires_human @3c33f46, 17/25 checks green, 8 pending, DCO yes, soak 0s of 10m0s. How to release it:
Clear the cause and the hive re-evaluates on its next pass; the stage moves to |
|
Thank you for your contribution! Your PR has been merged. We'd love to hear how your experience was: share feedback |
Fixes #16438
What
When a spoke boots from the ConfigMap seed instead of
/data/hive.yaml.runtime(overlay moved aside during a recovery, truncated, or never written), safety gates present in the prior overlay vanished with no log line. On the a1 spoke that re-enabled reviews on every unlabelled PR for ~4 hours.Change
pkg/config/gate_regression.go):DetectGateRegressions(effective, prior)comparesreview.require_labels,project.issue_filter.require_labels(non-empty β empty),project.paused_repos(paused repo no longer paused, org-qualified/case-folded match) andwriteback.per_repo(restrictive mode β looser mode or no override).DetectPriorOverlayGateRegressionsreadshive.yaml.runtime,hive.yaml.runtime.bak,hive.yaml.dashboard(skipping the file the effective config was loaded from; empty/malformed ignored; first overlay wins per gate). Legacyhive.yaml.bakis deliberately not consulted β it is frozen at the rename and would warn forever about gates removed on purpose.cmd/hive/main.go): oneWARN safety gate dropped: <gate> (was <value>) β effective config has <value>per gate; result kept oncfg.GateRegressions.config.refuse_gate_regression(bool, defaultfalse): refuse to start with exit code 21 and the gates named./api/statusβconfig.gate_regressions: []({gate, previous, effective, source})..cmp-bannerwarning at the top of every Settings tab while the list is non-empty (static-src/js/config-tabs.js,index.htmlregenerated).No change to how the gates are enforced or to which file the entrypoint boots from.
Tests
pkg/config/gate_regression_test.go: table test (unchanged, no prior gates, all dropped, weakened/tightened per-repo mode, droppedfulloverride, label list changed but non-empty, pause spelling match, nil), prior-overlay file reader (missing, empty/malformed, source-path skip, first-wins/merge), error message.cmd/hive/boot_config_deps_test.go: warns-and-boots, refuses with exit 21, quiet when clean.