20260924 - Boot guard, stack-health inventory, watchdog fix - #61
Merged
Merged
Conversation
retina-env-guard runs as ExecStartPre=- in retina-node.service. A power cut during the config-merger's write left ret9573ecda's compose .env entirely NUL on 2026-08-27, and compose then refused to load the project at every boot, including the run of the config-merger that would have repaired it: four weeks without radar. The guard moves a NUL or unparseable .env aside, by the same rule as retina-node's install preflight, and the config-merger regenerates it from user.yml. The "-" means it can never stop the stack starting. mender-inventory-retina-stack reports retina_stack (up, degraded, down, absent), running and restarting counts, blah2's state, whether .env and the compose project load, and retina-gui's mode. Telemetry runs inside the same stack, so a node whose stack is broken went silent instead of unhealthy; mender-updated runs this every 600 s regardless. blah2 stopped on purpose in spectrum or sdrconnect mode is not counted as degraded. On the test nodes it read Josh Test Node up and Josh Test Node 2 degraded (no RSPduo, blah2 restarting), in 116-160 ms. 13 tests run both scripts under dash with a stub docker; the Tests workflow now runs every test directory and checks all three scripts are executable. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
With no answer from /api/map, TIMESTAMP was empty and DIFF_TIMESTAMP=$(($CURR_TIMESTAMP-$TIMESTAMP)) failed with "operand expected" on every check. The restart still happened through the FIRST_CHAR test, so this was noise, but noise in exactly the situation where the watchdog's output gets read. A missing or non-numeric timestamp now reads as stale. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Contributor
Author
Evidence on
|
| # | Test | Node | Result |
|---|---|---|---|
| 1 | OS install | both | committed; /usr/local/sbin/retina-env-guard 0755; ExecStartPre=-/usr/local/sbin/retina-env-guard in retina-node.service; inventory script 0755; watchdog fix present; fork 8abfe66d with its diversion; stack active, 8 containers; guard silent on a clean boot |
| 2 | Health in Mender | both | Josh Test Node retina_stack=up, 8 running, blah2 running, .env/compose ok; Josh Test Node 2 retina_stack=degraded, 1 restarting, blah2 restarting (no RSPduo) |
| 3 | Boot guard end to end | Josh Test Node | .env zeroed (208 NUL), sysrq-b at 13:51:24. 13:51:58 retina-env-guard: set aside .../.env (contains NUL bytes) as .env.corrupt-20260924T135158Z; 13:52:02 config-merger wrote a new .env (0 NUL, identical to the original); 13:52:03 service finished; blah2 running, /api/map 1 s old, tar1090 on the node's location. No hands on the node. |
| 4 | Watchdog fix | Josh Test Node 2 | blah2-api stopped (/api/map unreachable): no operand expected; same decision (crash-looping ... bypassing grace period, Successfully restarted blah2); blah2-api back in 6 s |
| 5 | Fork regression | Josh Test Node | dev2 to v0.4.6.0: .env seeded, stop 13:57:43.3 to start 13:57:54.1 (10.8 s), blah2 0 restarts, map 1 s old |
Every expected result in the PR description met.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #60. Phase 2 items 2b and 5a of ClickUp OTA deployments miss part of the fleet on every release (123zgec4tmx), plus a watchdog fix found during #60's testing. Draft until proven on the
os-v0.17.2-devimage.feaf2cc: boot guard and stack health
retina-env-guard,ExecStartPre=-inretina-node.service. Moves a NUL or unparseablemanifests/.envaside at every boot, same rule as retina-node's install preflight, so the config-merger can regenerate it fromuser.yml. Without it a zeroed.envkeeps the stack down through every reboot: compose cannot load the project, so it cannot run the config-merger that would repair it (ret9573ecda, 2026-08-27 to 2026-09-24). The-means it can never stop the stack starting.mender-inventory-retina-stack, picked up by the existing inventory glob. Reportsretina_stack(up, degraded, down, absent),retina_stack_running,retina_stack_restarting,retina_blah2,retina_env_ok,retina_compose_ok,retina_mode. blah2 stopped on purpose in spectrum or sdrconnect mode is not degraded. Hand-run on the test nodes: Josh Test Nodeup, Josh Test Node 2degraded(no RSPduo), 116-160 ms.eb4ca91: watchdog fix
blah2_rspduo_restart.basherrored withoperand expectedwhenever blah2-api was down, becauseTIMESTAMPwas empty. A missing or non-numeric timestamp now reads as stale. Behaviour is unchanged: theFIRST_CHARtest already triggered the restart.Expected results on
os-v0.17.2-dev.envzeroed then hard reboot: guard sets it aside (journal ofretina-node.service), stack and radar come up by themselves.retina_stack=upfor Josh Test Node anddegradedfor Josh Test Node 2 within 10 minutes of boot.operand expectederror.🤖 Generated with Claude Code