[RFC] OS profile architecture: btrfs subvolumes + overlayfs over a read-only base

[VoidChecksum]

The Flipper OS concept says you're not 100% sure how to architect profiles yet, so here's a concrete comparison and a proposal. I've built profile-ish setups on RK3399/RPi boards and most of the pain is in the update story and the bootloader handoff, so I'll focus there.

The concept page pins the constraints: Debian base, apt must keep working inside a profile, instant clone/reset, boot-menu selection before Linux starts (so U-Boot is involved), SD-card friendliness, and a mental model a non-expert can hold.

Options

	apt works inside	clone/reset	disk overhead	base update	mental model	pre-Linux boot pick
(a) btrfs subvol + overlay /etc..rootfs	yes	instant (snapshot)	CoW deltas only	atomic (new RO subvol)	"bootable folders"	easy (rootflags/initramfs)
(b) ostree/composefs	no (image-based; apt-ostree still incubating in StarlingX)	checkout = cheap	low (hardlinks)	excellent, it's the whole point	hardest	ok
(c) A/B raw partitions	yes	no — fixed slots, full copies	worst	good	fine	easy
(d) systemd-sysext/confext	no (extensions are read-only /usr+/etc overlays)	n/a	low	good	leaky	poor
(e) nspawn containers	yes	cheap if on btrfs anyway	low	host rots again	leaky ("am I in the container?")	indirect — host boots first

(b) is the right answer for an appliance, wrong for this device: "users configure the system in the usual way using the package manager" is the stated requirement, and ostree breaks exactly that. (c) doesn't scale past two profiles. (d) is for vendor add-ons, not user-writable state. (e) deserves a real look — apt works, kernel is shared — but this device's whole point is raw hardware ownership (monitor mode, netdev juggling, kernel modules, DT overlays from a profile), and containers fight you on all of it. Worse, the host OS becomes a new thing that rots, which is the original problem recursed.

Proposal: (a), structured as your docs already describe it

One btrfs partition, plus a small FAT boot partition for U-Boot/extlinux/kernels:

@base/<version>      read-only Debian rootfs, shipped via update (btrfs send/receive stream)
@profiles/<id>/upper overlayfs upperdir (+workdir) — ALL profile state
@data                user files, mounted into every profile

A profile is literally "an overlay on top of the base system" — the wording from the concept page, made into a mount: lowerdir=@base/current,upperdir=@profiles/<id>/upper. apt/dpkg work unmodified (Docker has run apt-in-overlayfs at planetary scale for a decade; the old dpkg rename quirks are handled). Then:

• Clone = btrfs subvolume snapshot of the upper — instant, CoW, costs ~nothing.
• Reset built-in profile = delete upper, re-snapshot the shipped pristine upper.
• Rollback = auto-snapshot the upper before apt runs (apt/dpkg hook), expose "undo last change" in FlipCTL.
• Atomic base update = receive @base/<v+1> as a complete subvolume, flip a symlink, reboot. Old base stays until GC → instant rollback, and an interrupted update changes nothing. This answers the "reliable atomic updates" question without ostree.
• SD wear: CoW naturally spreads writes; mount noatime,commit=120. Profiles on removable SD = same subvolume layout on a second btrfs filesystem.

The honest cost: when @base jumps a Debian release, a profile's upper can hold stale conffiles/packages that conflict with the new lower. Every overlay system has this; mitigation is keeping base minimal (kernel + core userspace) so officials profiles carry their packages in the upper, and a flipper-profile check that diffs upper contents against the new base before flipping.

Boot menu / U-Boot

#68 already has FlipCTL rewriting the extlinux.conf default entry — fine from a running system, but the MCU can't write eMMC, so the pre-Linux picker has to hand the selection across the interconnect. U-Boot already talks I2C, and the interconnect has I2C + UART + GPIO between RP2350 and the SoC. Cleanest: boot.scr does i2c read of a "selected profile" register the MCU exposes (fallback: saved env), then appends flipper.profile=<id> flipper.base=<ver> to bootargs. A ~30-line initramfs hook mounts btrfs, assembles the overlay, switch_root. Keeping kernel+initramfs on the FAT partition avoids depending on U-Boot's btrfs reader (it exists, but FAT is boring and proven). One LABEL per profile in extlinux.conf also works and matches the current flow, but sysboot can't be steered to a label non-interactively, so env + boot.scr composes better with the MCU.

MVP path

1. Now: keep systemd targets for user scenarios #68's systemd targets on the single rootfs — it's the right prototyping vehicle.
2. Step 2: rootfs → btrfs, whole-rootfs-as-subvolume, profiles as full-fork snapshots. No overlay, no initramfs logic beyond rootflags=subvol=. Boot/clone/reset all work; ship the FlipCTL UI against this.
3. Step 3: split RO @base + overlay uppers; add the initramfs hook and send/receive base updates. Existing full-fork profiles migrate by diffing against the base they forked from.
4. Step 4: per-profile metadata (name, dates, origin — a JSON file inside the subvolume), apt snapshot hook, update agent.

Step 2 alone already kills the SD-card-shuffling workflow, which is the headline user win.

Open questions I'd want input on: per-profile /var and machine-id (share logs or isolate?), and whether a profile may carry its own DT overlays/cmdline fragment (I'd say yes — metadata file consumed by boot.scr).

[VoidChecksum]

Follow-up comment for #80 (ready to post, 137 words):

I went ahead and built the tool this RFC sketches: https://github.com/VoidChecksum/flipper-profile

flipper-profile manages the exact layout above — @base/<version> read-only subvolumes with an atomically flipped current symlink, @profiles/<id> overlay uppers, shared @data. Create/clone/reset/undo are all snapshots; base-adopt takes a rootfs dir or a btrfs send stream; set-default rewrites the same extlinux.conf FlipCTL touches in #68, one label per profile with flipper.profile=/flipper.base= appended. An initramfs-tools local-bottom script assembles the overlay before switch_root, and an apt Pre-Invoke hook snapshots the upper so a bad transaction is one undo away.

There's a loopback-btrfs e2e script in tests/ exercising the whole lifecycle, including the pinned-base behaviour across a base update (38 checks, green). README lists the honest gaps: no MCU handoff, no GC, full send streams only. Happy to adapt any of it if the layout discussion moves.

[alchark]

Btrfs is the approach we are currently testing internally, but many open questions remain especially around rebasing of user-modified profiles onto updated base profile releases.

@Yury-MonZon please have a look. I believe it might be helpful to briefly describe the approach we are currently testing out and the key open questions that we see. Thus we can invite the broader community to brainstorm it with us.