Skip to content

feat(os): NFS client (nfs-utils + rpcbind, client-only) in the base image - #295

Merged
jeefy merged 4 commits into
projectbluefin:mainfrom
jeefy:feat/nfs-client
Sep 30, 2026
Merged

jeefy merged 4 commits into
projectbluefin:mainfrom
jeefy:feat/nfs-client

Conversation

@jeefy

@jeefy jeefy commented Sep 30, 2026

Copy link
Copy Markdown
Collaborator

Fixes #294

Why

Bluefin Server has the NFS kernel client (nfs/nfs4 in /proc/filesystems) but no userspace helper: nfs-utils is not in os-base.bst and FSDK 26.08 ships no components/nfs-utils.bst (only libtirpc.bst and keyutils.bst). Without /sbin/mount.nfs, util-linux mount -t nfs falls through to the new mount API and the kernel refuses:

mount: /mnt: fsconfig() failed: NFS: mount program didn't pass remote address.

Reproduced on a 26.09.730 kubeadm worker: every kubelet NFS PV mount fails (MountVolume.SetUp failed … exit status 32), so clusters with an NFS provisioner cannot schedule those workloads on Bluefin Server nodes. iSCSI is already covered by open-iscsi in os-base; NFS was the gap. Flatcar ships nfs-utils.

What

Two project-local kind: manual elements built from pinned upstream tarballs (the repo does not register the autotools plugin; same shape as zfs/openzfs.bst and ignition/gptfdisk.bst), added to os-base.bst after open-iscsi:

bluefin-server/nfs-utils.bst (nfs-utils 2.8.7, kernel.org, sha256 from the signed sha256sums.asc) — client only:

  • ships mount.nfs, mount.nfs4, umount.nfs, umount.nfs4, rpc.statd, sm-notify, start-statd, nfsidmap, nfsstat, showmount, rpcdebug, nfsconf, nfsrahead, libnfsidmap + plugins, upstream nfs-client.target, nfs-utils.service, rpc-statd.service, rpc-statd-notify.service, 60-nfs.rules, 99-nfs.rules
  • configure: --disable-sbin-override (FSDK's sbindir is /usr/bin; /usr/sbin -> bin in FSDK and systemd's switch-root creates /sbin -> usr/sbin, so util-linux's /sbin helper search path reaches it), --enable-libmount-mount, --with-rpcgen=internal, --with-systemd=/usr/lib/systemd/system, --with-statdpath=/var/lib/nfs/statd, --with-statduser=rpcuser
  • disabled: --disable-gss --disable-svcgss (no krb5 in the image; sec=krb5* unsupported), --disable-nfsdcld --disable-nfsdcltrack --disable-nfsdctl --disable-nfsv4server (server-side; nfsdctl needs libnl/readline), --disable-junction (libxml2), --disable-ldap, --without-tcp-wrappers
  • nfs-utils always builds the server, so after install the element removes rpc.nfsd, rpc.mountd, exportfs, fsidd, rpc.idmapd, nfs-server.service, nfs-mountd.service, nfs-idmapd.service, fsidd.service, proc-fs-nfsd.mount, var-lib-nfs-rpc_pipefs.mount, rpc_pipefs.target, the three generators, the python tools (mountstats, nfsiostat, rpcctl, nfsdclnts — no python in the image), the internal rpcgen, and everything under /var (the install hook seeds etab/rmtab/statd dirs; /usr is what the erofs carries). It then asserts the client files are present.
  • runtime state: /usr/lib/tmpfiles.d/nfs-utils.conf creates /var/lib/nfs and /var/lib/nfs/statd{,/sm,/sm.bak} (0700 rpcuser); /usr/lib/sysusers.d/nfs-utils.conf adds rpcuser (uid 29, Fedora's) — systemd-sysusers runs at image build so it lands in the factory /etc. rpc.statd/sm-notify drop privileges to the owner of sm/. Drop-ins order both units After=systemd-tmpfiles-setup.service + RequiresMountsFor=/var/lib/nfs/statd (they have DefaultDependencies=no).
  • /etc/request-key.d/id_resolver.conf for the kernel's NFSv4 id-mapping upcall to nfsidmap (keyutils' request-key is already in FSDK).

bluefin-server/rpcbind.bst (rpcbind 1.3.1, sourceforge, matches Fedora rawhide): rpc.statd registers with the local rpcbind and exits when there is none (nfs_svc_create → "failed to create RPC listeners"), and upstream rpc-statd.service has Requires=rpcbind.socket, so shipping statd without rpcbind would leave a broken unit. --enable-warmstarts --with-statedir=/run/rpcbind --with-rpcuser=rpc (uid 32 via sysusers.d), upstream rpcbind.service/rpcbind.socket.

files/os/systemd/system-preset/80-bluefin-nfs.preset (FSDK has no catch-all disable *): enable nfs-client.target, enable rpcbind.socket, disable rpcbind.service. Nothing runs at boot except sm-notify once (exits immediately with no state) and the portmapper socket; mount.nfs starts rpc-statd.service via start-statd only when an NFSv3 mount needs locking, which socket-activates rpcbind. NFSv4 needs neither.

include/aliases.yml gains kernel_pub: https://www.kernel.org/pub/.

Tests

  • tests/unit/test_nfs_client.py: parsed-property contracts (pinned tar sources, autotools sandbox, client-only configure set, server bits removed, tmpfiles/sysusers, nothing under /var, socket-activated rpcbind, preset enables only the two units).
  • just test-unit: 594 pytest + 74 bats pass. just validate passes. python3 .github/scripts/docs-checks.py passes.

Built locally

just bst build bluefin-server/nfs-utils.bst (and rpcbind) succeed in the bst2 container; bst artifact checkout --deps none of nfs-utils gives:

usr/bin: mount.nfs  mount.nfs4 -> mount.nfs  umount.nfs -> mount.nfs  umount.nfs4 -> mount.nfs
         rpc.statd  sm-notify  start-statd  nfsidmap  nfsstat  showmount  rpcdebug  nfsconf
usr/libexec/nfsrahead
usr/lib/systemd/system: nfs-client.target nfs-utils.service rpc-statd.service(+.d) rpc-statd-notify.service(+.d)
usr/lib/{sysusers.d,tmpfiles.d}/nfs-utils.conf   usr/lib/udev/rules.d/{60,99}-nfs.rules
usr/lib/x86_64-linux-gnu/libnfsidmap.so.1 + libnfsidmap/{nsswitch,static,regex}.so
etc/request-key.d/id_resolver.conf

mount.nfs NEEDED: libtirpc.so.3, libmount.so.1; rpc.statd: libcap.so.2, libtirpc.so.3; rpcbind: libtirpc.so.3, libsystemd.so.0. No /var in the artifact. Not boot-tested on hardware yet — CI's image build and dogfood cover the compose; an NFS mount from a booted node is the remaining verification.

Notes / caveats

  • mount.nfs is not setuid (the CAS drops mode bits, like FSDK's mount): fine for kubelet/root; unprivileged user NFS mounts would need the same initial-script treatment as fix(ddi): restore the setuid/setgid bits Flatcar ships on sudo, su, passwd, mount, … #255.
  • rpcbind.socket binds 0.0.0.0:111/[::]:111 at boot (upstream unit, same as Fedora/Flatcar). Remote NFSv3 servers need it to reach rpc.statd for SM_NOTIFY, so it cannot be loopback-only.
  • Diskless nodes lose /var/lib/nfs/statd at reboot, so an NFSv3 server holding locks for a rebooted diskless client is not notified. Inherent to stateless clients; NFSv4 (lease-based) is unaffected.
  • Docs: docs/skills/kubeadm-sysext.md gains an "NFS" section next to "iSCSI".

jeefy and others added 4 commits September 29, 2026 19:25
rpc.statd (NFSv3 locking) registers its listener with the local rpcbind and
exits when there is none, and NFS servers look statd up through port 111 to
send SM_NOTIFY after a reboot. FSDK 26.08 ships libtirpc but no rpcbind, so
build 1.3.1 from the upstream tarball: warm starts in /run/rpcbind (created
by rpcbind itself), running as a fixed-uid "rpc" sysusers.d account, with
upstream's rpcbind.service/rpcbind.socket for socket activation.

Refs projectbluefin#294

Assisted-by: Claude Fable 5.1 via GitHub Copilot

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Without /sbin/mount.nfs, util-linux mount(8) hands "-t nfs" to the new
mount API and the kernel refuses it ("fsconfig() failed: NFS: mount program
didn't pass remote address"), so every kubelet NFS PV fails with exit
status 32. FSDK 26.08 has libtirpc and keyutils but no nfs-utils element.

Build nfs-utils 2.8.7 from the upstream tarball, client only: mount.nfs,
mount.nfs4, umount.nfs, rpc.statd, sm-notify, start-statd, nfsidmap,
nfsstat, showmount, rpcdebug, nfsconf, nfsrahead, with upstream's
nfs-client.target, rpc-statd*.service and udev rules. Configured with
--disable-sbin-override so the helpers honour sbindir (FSDK's /usr/sbin is
a symlink to bin, reached from mount's /sbin search path via the
/sbin -> usr/sbin link switch-root creates), --enable-libmount-mount, and
GSS, nfsdcld/nfsdcltrack/nfsdctl, the NFSv4 server, junctions, LDAP and
tcp-wrappers off. The always-built server pieces (rpc.nfsd, rpc.mountd,
exportfs, fsidd, rpc.idmapd, their units, rpc_pipefs, the generators), the
python tools and the internal rpcgen are dropped after install.

/usr is read-only and /var starts empty, so /var/lib/nfs/statd comes from
tmpfiles.d, owned by a fixed-uid rpcuser account (sysusers.d) that
rpc.statd and sm-notify drop to; drop-ins order both units after
systemd-tmpfiles-setup. NFSv4 id mapping goes through the request-key
upcall (id_resolver.conf in the factory /etc). 80-bluefin-nfs.preset
enables nfs-client.target and rpcbind.socket and keeps rpcbind.service off:
mount.nfs starts rpc-statd.service on demand for NFSv3 locking.

Fixes projectbluefin#294

Assisted-by: Claude Fable 5.1 via GitHub Copilot

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Parsed-property assertions on nfs-utils.bst and rpcbind.bst: pinned
upstream tarballs, FSDK autotools sandbox, the client-only configure set
(--disable-sbin-override, libmount, no GSS/server bits), statd state from
tmpfiles.d and sysusers.d with nothing shipped under /var, socket-activated
rpcbind, and a preset that enables only nfs-client.target and
rpcbind.socket.

Refs projectbluefin#294

Assisted-by: Claude Fable 5.1 via GitHub Copilot

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
What the base image ships for NFS volumes, why (the fsconfig() failure
without mount.nfs), what is left out (server, GSS) and the diskless caveat
for NFSv3 lock notification.

Refs projectbluefin#294

Assisted-by: Claude Fable 5.1 via GitHub Copilot

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

HUMAN DECISION NEEDED — @jeefy: merge without an end-to-end mount -t nfs on a booted node (PR body: "Not boot-tested on hardware yet"), or require that check first? This reviewer cannot boot the image.

Reviewed at fccf8c8.

security

  • files/os/systemd/system-preset/80-bluefin-nfs.preset:7 enables rpcbind.socket at boot on every node. Upstream rpcbind.socket (rpcbind 1.3.1 systemd/rpcbind.socket.in) listens TCP+UDP 0.0.0.0:111 and [::]:111, and the image ships no firewall. The boot-time listener buys nothing: rpc-statd.service (nfs-utils 2.8.7 systemd/rpc-statd.service) carries Requires=rpcbind.socket, so start-statd → systemctl start rpc-statd.service already pulls the socket in on the first NFSv3 locking mount, and rpc-statd-notify.service/nfs-client.target do not need rpcbind. Servers only send SM_NOTIFY to a client whose statd is running, which implies the socket is already up. Suggest dropping enable rpcbind.socket (optionally disable rpcbind.socket) so port 111 is open only while NFSv3 locking is in use; update tests/unit/test_nfs_client.py:104,115 accordingly.

correctness

  • The change's sole purpose — /sbin/mount.nfs found by util-linux and a working mount -t nfs — is unverified at runtime; tests/unit/test_nfs_client.py only checks element YAML. I verified the path chain (FSDK 26.08 elements/bootstrap/symlinks: sbin -> usr/bin, usr/sbin -> bin; include/install-dirs.yml:5 sbindir: %{bindir}) and that rpc-statd.service's hardcoded /usr/sbin/rpc.statd resolves, and that systemd-sysusers.service runs before systemd-tmpfiles-setup.service so rpcuser in usr/lib/tmpfiles.d/nfs-utils.conf (elements/bluefin-server/nfs-utils.bst:129-134) resolves on an empty /etc. What remains unverifiable here is the boot: one NFSv3 (locking → statd/rpcbind) and one NFSv4 mount on a dogfood node.

No findings from: intent-alignment, style, docs-currency.

Confidence: 3/5 (needs attention) — 2 medium findings

— hive: agent=reviewer backend=copilot model=claude-fable-5.1 copilot=1.0.88

@jeefy
jeefy merged commit a0359d9 into projectbluefin:main Sep 30, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Bluefin Server has no mount.nfs: Kubernetes NFS volumes cannot mount

1 participant