fatal: could not create work tree dir: No space left on device

[biergaizi]

Hello. I'm the project contributor of openEMS, a free and open source 3D electromagnetic field solver. As the de-facto maintainer of the build system, I rely on cross-platform-actions to ensure that the project is POSIX-compatible. However, in recent builds, the FreeBSD workflow has stopped working because the git repository has grown sufficiently large, that the VM image is too small for the repo.

Clone 6ddc54692a9f337de09e5264a6c4d2e98ee33cdd from https://github.com/thliebig/openEMS-Project.git
fatal: could not create work tree dir 'openEMS-Project': No space left on device

The git repo is cloned to ~/ of the default user. See the link for the full log.

I know that it's possible to build a custom VM image, but doing so would increase the burden of project maintenance even further, since the project doesn't have any full-time maintainer. It's already a struggle to maintain the existing build matrix to the lack of developers.

Is there an easy way to mount an additional image onto the VM? Thanks.

[jacob-carlborg]

I apologize for seeing this so late. Is the space running out inside the virtual machine or on the runner? The size of the VM images are quite small, because the hard drives on the runners are very small. They only provide 16 GB of hard drive, if they haven't changed that lately. You need space both inside the VM and on the runner. The VM image needs to fit on the runner and you might need some additional space to copy files to and from the VM to the runner.

[biergaizi]

I believe the VM itself is full.

Inside the VM, I have (after removing some packages to work around the disk space limitation):

Filesystem       Size    Used   Avail Capacity  Mounted on
/dev/vtbd0s1a     11G    7.9G    1.8G    81%    /
devfs            1.0K      0B    1.0K     0%    /dev
/dev/vtbd1s1      38M    1.0K     38M     0%    /mnt/resources

I was able to fix my CI/CD pipeline by installing only a subset of Qt 5 on FreeBSD:

  # use separate packages, not the meta-package "qt5" to save disk space
  sudo pkg install -y bash cmake git boost-libs tinyxml vtk9 hdf5 cgal octave \
                      qt5-core qt5-gui qt5-opengl qt5-xml \
                      python3 \
                      py311-setuptools \
                      py311-cython3 py311-numpy py311-h5py py311-matplotlib
                      # prefix must match FreeBSD's default python
                      # version. FreeBSD 14.2 uses Python 3.11 so
                      # use "py311-"

As you can see, I've just installed 20 packages, Qt 5, GNU Octave, and VTK. So I believe this is a fairly reasonable and minimalist setup. Unfortunately, after installing these, ~8 GiB of disk space has already evaporated.

Number of packages to be installed: 305
Number of packages to be upgraded: 1

The process will require 5 GiB more space.
949 MiB to be downloaded.

Installing the full set of Qt 5 would take 10 GiB+, leaving no room for git repos.

The size of the VM images are quite small, because the hard drives on the runners are very small. They only provide 16 GB of hard drive, if they haven't changed that lately. You need space both inside the VM and on the runner.

Yes, this makes sense. Unfortunately, if GitHub Actions are truly limited to only 16 GiB disk, the VM overhead means the remaining in-VM space even smaller, making cross-platform-actions very difficult to use in any practical project with significant dependencies such as GUI. Sigh...

[jacob-carlborg]

Do you know how much space you need? You could try and build a custom VM image. Then you can increase its size, but you're still limited to the size of the hard drive of the runner.

https://github.com/cross-platform-actions/action?tab=readme-ov-file#custom-vm-image-image_url

[biergaizi]

The original post read:

I know that it's possible to build a custom VM image, but doing so would increase the burden of project maintenance even further, since the project doesn't have any full-time maintainer. It's already a struggle to maintain the existing build matrix to the lack of developers.

Is there an easy way to mount an additional image onto the VM?

[jacob-carlborg]

Right, sorry.

[jacob-carlborg]

To be clear, there's no way to mount an additional image.

[biergaizi]

The best possible solution is to create an option that allows one to manually attach additional VM disk images in the host file system, or alternatively, allowing users to somehow access host files from a guest VM (via 9pfs?). I believe this is a reasonable feature request, as it has many use cases beyond addressing the VM disk space problem.

[jacob-carlborg]

The best possible solution is to create an option that allows one to manually attach additional VM disk images in the host file system

You're still limited to the size of the runner disk.

allowing users to somehow access host files from a guest VM (via 9pfs?)

That would be an alternative that could work. Unfortunately this requires support in the guest operating system and none of the supported operating systems support 9pfs. It looks like the upcoming FreeBSD 15 might support it [1]. Another option would be file sharing over NFS, but I tried that when I initially created Cross Platform Action and ran into some issues. That might have been an issue only with macOS, since, back then, that was the only supported runner platform.

Another option would be to increase the VM image size and use the qcow2 format instead of raw. With qcow2 the VM image will only occupy space on the host when the guest uses that disk space. The downside is that qcow2 has worse performance compared to raw. There's still the limitation of the runner disk size.

it has many use cases beyond addressing the VM disk space problem

Do you have examples of other use cases?

[1] https://forums.freebsd.org/threads/how-to-properly-mount-a-directory-via-9p-plan9-in-the-bhyve-of-a-freebsd-guest.96364/

[biergaizi]

The best possible solution is to create an option that allows one to manually attach additional VM disk images in the host file system

You're still limited to the size of the runner disk.

The point is that users will no longer be limited to the hardcoded disk image size chosen by the cross-platform-actions/action risk-averse maintainers (who must choose slightly smaller sizes to avoid overfilling the host). Instead, end-users can check whether any free space exists in their CI/CD pipeline. if it does exist, additional file-backed disk image can be created in the host, and by specifying the file (using an optional argument in cross-platform-actions/action) as a secondary disk for the VM as the repo and build directories, the existing images without the need to recreate the whole system from scratch.

I believe this would be an useful feature.

Another option would be to increase the VM image size and use the qcow2 format instead of raw. With qcow2 the VM image will only occupy space on the host when the guest uses that disk space.

Right. If optional settings such as use-qcow2 or grow-qcow2 are available, it would also solve the main problem.

Do you have examples of other use cases [for 9pfs or NFS]?

For one example, this would allow one to unify the host-based checkout process using GitHub's actions/checkout@v4, the checked files can be read directly by the VM from the host (if the file permissions are right), and without recopying if there are BLOBs in the repos.

But if option 1 and 2 are already available, the third option is diminished in its importance.

[jacob-carlborg]

For reference, I found this action [1] which can be used to cleanup and get back som free space on the runner.

[1] https://github.com/jlumbroso/free-disk-space

[biergaizi]

Interesting.