Skip to content

fix(installer): stop root-a growth from starving the /var partition - #185

Closed
mendezr wants to merge 1 commit into
projectbluefin:mainfrom
mendezr:fix/installer-root-var-partition-sizing
Closed

mendezr wants to merge 1 commit into
projectbluefin:mainfrom
mendezr:fix/installer-root-var-partition-sizing

Conversation

@mendezr

@mendezr mendezr commented Sep 18, 2026

Copy link
Copy Markdown
Contributor

Fixes #161.

The installer-test / Install and boot in QEMU gate fails on main itself: the second boot drops into emergency mode waiting for /dev/disk/by-partlabel/var.

Root cause

Both root-a and var are GrowFileSystem=yes partitions, so systemd-repart splits the leftover disk space between them by default weight. But 20-root-a.conf set SizeMaxBytes=16G — the full 16G test/VM disk. Combined with GrowFileSystem=yes, root-a's elastic growth can consume the space /var needs, so the var partition is never created (or is undersized) and var.mount times out on the second boot.

Fix

  • 20-root-a.conf: bound SizeMaxBytes to a realistic OS-payload size (8G) instead of the full test-disk size, so root-a's growth can no longer starve /var.
  • 30-var.conf: give /var an explicit high Weight=10000 so it always retains a dominant share of the leftover space and its SizeMinBytes=4G, regardless of how far root-a grows. /var remains the tail partition that grows to fill the disk.

Verification

  • Static repart-layout invariants (as asserted by tests/unit/test_repart_layout.py) pass: exactly one esp/root/var; root keeps GrowFileSystem=yes and a SizeMaxBytes; var stays xfs, GrowFileSystem=yes, no SizeMaxBytes, no FactoryReset, with its CopyFiles intact.
  • check-release-version.py and check-k0s-version.py pass.
  • just validate (BuildStream merge-contract check) and the just show-me-the-future QEMU smoke should be exercised in CI on this PR.

— hive: backend=goose model=deepseek/deepseek-v4-flash-0731

🐝 Hive Agent: contributor | SHA: 8bebf54

The installer-test 'Install and boot in QEMU' gate fails on main: the second boot drops to emergency mode waiting for /dev/disk/by-partlabel/var. Both 'GrowFileSystem=yes' partitions (root-a and var) split the leftover disk space by default weight, but 20-root-a.conf allowed root-a to grow up to SizeMaxBytes=16G (the full 16G test disk). Depending on the DDI payload's footprint, root-a's elastic growth could consume the space /var needs, so var is never created (or is undersized) and the var.mount unit times out.

Bound root-a's SizeMaxBytes to a realistic OS-payload size (8G) instead of the full test-disk size, and give /var an explicit high Weight= so it always retains a dominant share of the leftover space and its SizeMinBytes=4G regardless of how far root-a grows. /var remains the tail partition that grows to fill the disk.

Assisted-by: DeepSeek via GitHub Copilot

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: mendezr <mendezr@users.noreply.github.com>
@castrojo

Copy link
Copy Markdown
Contributor

Closing. Both halves of this change turn out to be unsupported — the first by direct runtime evidence, the second by measurement. I want to lay out the evidence properly because the diagnosis here was a reasonable inference from what was visible, and the thing that refutes it only became available from #173's runs.

Half one: the SizeMaxBytes 16G -> 8G cap does not fix the failure it targets.

The stated cause is that root-a's elastic growth consumes the space /var needs, so the var partition is never created or is undersized, and var.mount times out into emergency mode.

PR #173 refutes that directly. Its entire diff is two files — elements/flatcar/flatcar-zfs.bst and tests/unit/test_installer_contract.py. It does not touch 20-root-a.conf or any repart.d file; SizeMaxBytes=16G is completely untouched. Its only functional change appends:

rm -rf "%{install-root}/usr/lib/systemd/system/systemd-udevd.service.d"

and its comment explains the real mechanism: leaving Flatcar's drop-in in place makes systemd-udevd wait on systemd-sysext.service -> local-fs.target -> var.mount -> the udev device unit. That is a circular ordering deadlock, not a disk-space shortage.

With that one drop-in removed and sizing unchanged, the system boots. I confirmed it on a throwaway integration of #173 + #181 (run 35375246792): zero Dependency failed lines and zero emergency mode in the installed-system serial log, against main run 34920225526 which shows Timed out waiting for device /dev/disk/by-partlabel/var, Dependency failed for /var, You are in emergency mode.

There is also a logical tell worth keeping: an undersized partition still mounts. Sizing can never explain a mount timeout. Only a missing or unreadable device can, and that is what the udev deadlock produced.

For the same reasons I closed #165, which proposed this identical 8G cap.

Half two: Weight=10000 is valid, but does the exact harm we are rejecting the cap for — only more so.

Your syntax is correct. repart.d(5) (systemd 261.2) documents Weight= in range 0…10000000, default 1000, splitting free space by relative weight subject to the size constraints. I verified the parser recognises it specifically rather than ignoring it as unknown: feeding Weight=2000000 produces a dedicated range diagnostic, and Weight=10000 parses silently.

But measured against the real configs with systemd-repart --dry-run, here is what it does to root-a:

disk    root-a today    root-a with Weight=10000
20G     9.4G            4G
40G     16G             4G
100G    16G             8.9G

The reason I was rejecting the 8G cap is that halving the A-slot ceiling halves the headroom available to the A/B image payload. Weight=10000 quarters it to the SizeMinBytes=4G floor on a 40G disk. Landing the revert and the weight together would mean refusing a change for halving headroom while accepting one that quarters it.

The determinism argument does not survive either: root-a already carries SizeMaxBytes=16G, so on any disk of roughly 21G or more the layout is already fully deterministic — root-a takes 16G, /var takes the remainder. Weight= only changes behaviour on small disks, and there it changes it in the harmful direction.

I considered keeping Weight= at some smaller value, and decided against it: choosing a weight you never proposed would be me redesigning your change rather than reviewing it.

With both halves removed the PR is empty, which is why this is a close rather than a reduction.

One correction for the record: Fixes #161 is not right. #161 is the /var timeout / emergency-mode issue, and #173 is what fixes it. Worth watching #173 rather than reworking this.

None of this was obvious from main — the symptom really does look like a space problem, and two separate people reached the same conclusion independently. Please do push back if you think the dry-run numbers or the integration-run evidence are wrong; I would rather be corrected than have this stay shut on a mistake of mine.

@castrojo castrojo closed this Sep 18, 2026
@mendezr
mendezr deleted the fix/installer-root-var-partition-sizing branch September 19, 2026 05:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[scanner] installer-test / Install and boot in QEMU fails on main — /var partition times out at boot

2 participants