Skip to content

20260924 - Install a fork of the docker-compose Update Module - #60

Merged
Purple10101 merged 2 commits into
mainfrom
20260924-fork-compose-module
Sep 24, 2026
Merged

Purple10101 merged 2 commits into
mainfrom
20260924-fork-compose-module

Conversation

@Purple10101

Copy link
Copy Markdown
Contributor

Phase 2, item 3 of ClickUp OTA deployments miss part of the fleet on every release (123zgec4tmx). Draft until the acceptance evidence is complete; the full record lives in the ticket's Verification section.

Review commit by commit: 4462353 vendors upstream mender-docker-compose 1.0.0-1 byte for byte (sha256 5be40882...); 6bd2637 is the fork, so its diff is exactly our change (every hunk marked FORK:).

What the fork changes

Upstream behaviour Fork
stops the running stack, then extracts and loads images (147-149 s outage) loads first, then stop and start (10.7 s, measured 3 times)
an early failure (corrupt download, full disk) looks like "rollback after commit" and stops and moves the live stack new/ created before anything can fail
rollback after a pre-stop failure runs down from new/, stopping the live stack (shared project name) leaves the live stack running
nothing coordinates with the RSPduo watchdog holds /data/retina-gui/restart.lock for install and rollback
new stack starts without .env, so tar1090 gets 0,0 and adsb.lol off seeds .env from current/
cleanup fails a committed deployment over an image already removed skips it

Install

Two tasks in roles/mender: dpkg-divert moves the packaged file to /usr/share/mender/docker-compose.upstream (outside modules/v3), then the fork is copied in, 0755. Undo is dpkg-divert --rename --remove. The artifact type stays docker-compose.

Tests

tests/update-modules/ runs the real module under dash with a stub docker and jq and real tar payloads; new Tests workflow runs it. 15 pass. Upstream fails 8; the first fork revision fails the 3 that caught its own regressions (watchdog race, destructive early-failure rollback).

Evidence on test nodes so far

  • 3 installs on Josh Test Node 2: 10.7 s stop-to-start each; .env seeded; watchdog run by hand mid-load printed A stack restart is already in flight - skipping this check.
  • Power cut (sysrq-b) during image load, twice: node rebooted into its previous stack, rollback leaving it running, deployment failure and retryable.
  • Power cut right after Stopping previous: retina-node.service skipped, rollback restored previous and started it, deployment failure and retryable.

Still to prove before this leaves draft

Install whose new stack fails to start; node that skipped a release; normal upgrade on Josh Test Node (with RSPduo, radar health); this packaging through an os-v*-dev build.

🤖 Generated with Claude Code

Purple10101 and others added 2 commits September 24, 2026 11:55
Verbatim copy of /usr/share/mender/modules/v3/docker-compose from
mender-docker-compose 1.0.0-1+debian+bookworm, sha256
5be40882cbb1ecb5944ffd4a523c4c667fea5660d9f908b1c05754ced04771fd, identical
on Josh Test Node and Josh Test Node 2. Nothing installs it yet. Committed on
its own so the fork that follows is reviewable as a diff against upstream.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The packaged module stops the running retina-node stack before it loads the
new images (147-149 s of radar outage per install on Josh Test Node, and the
worst window for a power cut), rolls back into the live stack when an install
fails early, and fails a committed deployment over an image already removed.

The fork, every change marked FORK:
- loads the new images before stopping the running stack (10.7 s outage,
  measured three times on Josh Test Node 2);
- creates new/ before anything can fail, so an early failure is never read as
  a rollback after a commit, which would stop and move the live stack;
- on a failure before the stop, rolls back without touching the live stack;
- holds /data/retina-gui/restart.lock, which retina-gui and the RSPduo
  watchdog already use, so the watchdog cannot restart the stack mid-install
  (it did, on Josh Test Node 2, before the lock was added);
- seeds new/manifests/.env from current/, so tar1090 starts with the node's
  own location;
- skips an image that is already gone during cleanup.

owl-os installs it with dpkg-divert, moving the packaged file to
/usr/share/mender/docker-compose.upstream (outside modules/v3, where Mender
would list it as a module), so apt stays consistent and undo is one
dpkg-divert --remove. The artifact type stays docker-compose.

tests/update-modules runs the real module under dash against a stub docker
and jq with real tar payloads, in a new Tests workflow. Upstream fails 8 of
the 15.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@Purple10101

Copy link
Copy Markdown
Contributor Author

Acceptance evidence, 2026-09-24 (full record: ClickUp 123zgec4tmx, Verification)

Test Node Result
Normal upgrade, real radar Josh Test Node 11.0 s stop to start (upstream 147-149 s); blah2 0 restarts; /api/map 0 s old
Watchdog run by hand mid-load Josh Test Node 2 A stack restart is already in flight - skipping this check
Power cut (sysrq-b) during image load, x2 Josh Test Node 2 rebooted into previous stack; rollback leaving it running; deployment failure, retryable
Power cut right after Stopping previous Josh Test Node 2 retina-node.service skipped; rollback restored and started previous; deployment failure, retryable
New stack never healthy (deliberately broken v0.4.7.0-dev3) Josh Test Node 120 s health wait, rollback restored v0.4.6.0, radar back (map 1 s old)
Skipped two releases (v0.4.4.0 to dev2) Josh Test Node 2 success, 10.6 s, old images cleaned up
Packaging via os-v0.17.1-dev both diversion listed, module 8abfe66d 0755, upstream at /usr/share/mender/docker-compose.upstream (5be40882), modules/v3 unchanged, package still hold ok installed, stack back after reboot
retina-node install on the image-installed fork Josh Test Node 10.8 s stop to start, .env seeded, blah2 0 restarts, map 1 s old
Image already removed during cleanup unit test only passes; not reproduced on a node

Known cost, upstream behaviour: a release whose stack never becomes healthy costs about 2 min 45 s of radar before rollback (the module's 120 s health wait).

@Purple10101
Purple10101 marked this pull request as ready for review September 24, 2026 14:01
@Purple10101
Purple10101 merged commit b979ee0 into main Sep 24, 2026
2 checks passed
@Purple10101
Purple10101 deleted the 20260924-fork-compose-module branch September 24, 2026 14:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant