os/release: Make it safe, simple, and easy for users to apply software updates

[ethanjli]

This is a tracking issue for our project to improve the PlanktoScope's software update process in order to reduce the barriers for users to update their PlanktoScope software deployments.

Motivation

Currently, safely updating the software on a PlanktoScope requires flashing a new SD card and swapping it out with the old SD card (as described in #88, the previous Git-based system was too finicky to be usable); in order to make it easy to downgrade the software, the old SD card (with the old OS image) must be kept around indefinitely instead of being reassigned for other purposes. Re-flashing SD cards requires internet bandwidth (to download the new SD card image), time (to flash the SD card), planning (to prevent loss of data), potentially spare SD cards (if the device owner wants to keep old SD cards for easy downgrades), and potentially reconfiguration (to change the software's hardware settings again to match the actual hardware configuration). It would be useful if most updates could be applied without re-flashing the SD card - in other words, if we could deliver updates over the air (or over a sneakernet) and apply them in-place.

This is particularly useful for people with v2.1 PlanktoScope hardware, in which the SD card is extremely difficult to access - it requires partial disassembly of the PlanktoScope.

A non-SD-card-based update method has been requested by Rodrigo Gonçalves on the #6-dev-software channel in the PlanktoScope Slack.

Goals

• Reduce the number of software updates requiring an SD card re-flash (i.e. an irreversible upgrade) to a frequency of once every (KPI) two years (i.e. the same frequency as Raspberry Pi OS major releases).
• Make in-place upgrades (i.e. not requiring a re-flash of the SD card) reversible
• For in-place upgrades:
  • Reduce the total size of files which must be downloaded to less than (KPI) 1 GB on average for major software changes, and less than (KPI) 100 MB on average for minor software changes.
  • Enable upgrading/downgrading the software to newer or older releases with (KPI) one button-press and a confirmation dialog.
  • Enable automatic background downloading of software artifacts for software upgrades.
  • Provide a mechanism to prevent loss of data and settings through a software update.
  • Make upgrades atomic.
• Enable both OTA and local updates
• Avoid the need for us (or our users) to self-host (or pay for a SaaS which hosts) a centralized update server for OTA updates

Steps

(these steps will be converted into tracking issues for tasks)

• frontend: Make a landing page at the PlanktoScope's base http URL #160
• os: Distribute third-party applications as Docker Compose applications in a Github repository #132
• os/pallet: Containerize the Python segmenter and deliver it via Forklift #326
• os: Manage OS scripts and config files using Forklift #349
• os/pallet: Containerize the Node-RED frontend and deliver it via Forklift #190
• os/pallet: Containerize the Python hardware controller and deliver it via Forklift #455
• Implement automatic rollback from failed pallet changes, via a systemd service
  • Also detect failures of systemd services managed by forklift, via a systemd service?
• updates: Enable air-gapped updating via a USB drive forklift-run/forklift#235
• Implement opt-out automatic background downloading of updates:
  • updates: Enable background downloading of pallet updates forklift-run/forklift#254
  • Make changes to PlanktoScope OS
• Implement opt-out automatic updates on the stable/calico release channel:
  • updates: Make it easier to bump the local pallet to the latest release or branch commit forklift-run/forklift#246
  • Make changes to PlanktoScope OS
• Manage /boot/config.txt using Forklift:
  • host/files: Enable managing bootloader/kernel files used before early boot forklift-run/forklift#241
  • Make changes to PlanktoScope OS
• toolchain: Add CI tooling to check that Forklift packages, repositories, and pallets are correctly specified forklift-run/forklift#252
• ui: Provide a basic web browser GUI forklift-run/forklift#251
• composition: Authenticate pallet requirements forklift-run/forklift#243

Unresolved Questions

How can we achieve atomic & easy-to-rollback upgrades of the base OS (i.e. apt packages and stuff) on Raspberry Pi OS? e.g. can we achieve a container image-based approach like bootc? (Note: most of the comment replies I've added on this issue are related to this unresolved question which is out-of-scope for this particular issue)

[ethanjli]

Since the Node-RED dashboard runs host commands, containerizing it will be a bit tricky. https://stackoverflow.com/a/63140387 and https://security.stackexchange.com/a/218379 provide some hints about how we might do it, starting with a privileged container. We could then gradually lock down the frontend.

[ethanjli]

For the base OS, it might be worth investigating Yocto, either to use (though Yocto might be too weird/difficult/complex) or as inspiration for us. Refer to:

• https://medium.com/nerd-for-tech/build-your-own-linux-image-for-the-raspberry-pi-f61adb799652
• https://github.com/agherzan/meta-raspberrypi
• https://lancesimms.com/RaspberryPi/HackingRaspberryPi4WithYocto_Introduction.html
• https://github.com/advancedtelematic/meta-updater/ (it uses OSTree for atomic OTA updates; see also https://news.ycombinator.com/item?id=18092300)

[ethanjli]

It might be worth investigating whether we can deliver PlanktoScope OS-level binaries and scripts via a container image for Distrobox - refer to https://distrobox.it/usage/distrobox-export/ . We would need to evaluate whether we can integrate Distrobox with Forklift.

We might also be able to use chezmoi to manage config files (especially in the home directory) - we would need to evaluate whether we can integrate chezmoi with Forklift.

[ethanjli]

https://www.youtube.com/watch?v=gSePCve0yZI discusses a packaging-based approach to making configuration files easier to upgrade, and some principles generalizable beyond their specific approach.

[ethanjli]

https://www.youtube.com/watch?v=XTy3scX6rF4 discusses the use of filesystem images (which from my understanding are not container images) to manage config files like in /etc. The relevant systemd manpage is at https://www.freedesktop.org/software/systemd/man/latest/systemd-confext.html (note that the required systemd version (248 for most things, 254 for starting/restarting units in extensions) is not in Debian 11 bullseye, whose version of systemd (247) is still too old, and Debian 12 bookworm only has systemd 252)

[ethanjli]

https://github.com/ublue-os/sysext shows an example of how to use systemd-sysext, for anything which we'd need to distribute outside /etc which we can't just distribute using forklift. We could probably distribute shell scripts as part of forklift packages, and run them from the cached packages (by making a forklift subcommand to return the path of files in the package used by a particular app deployment, and using that path to invoke the script). So we probably won't need to use sysext for our own custom stuff. But (and this is what Flatcar does) we could just distribute Docker as a sysext, instead of installing it through apt-get). In fact, we could probably reuse the images Flatcar already provides for docker anddocker compose, maybe?

However, Flatcar's sysext-bakery repo recommends only shipping static binaries, though that may not be necessary if we specify the target OS version correctly - so we'd want to have a build for each OS version.

After rewatching https://www.youtube.com/watch?v=XTy3scX6rF4 , I think it would be great if we could take advantage of systemd-sysext as an image-based workflow replacement for many of the files we're manually copying in our distro setup scripts right now.