GC can delete images that are currently mounted

[giuseppe]

There is currently no tracking of which images are mounted. This means garbage collection (or manual deletion) can remove an image from the repository while it is still actively mounted.

Should we enforce a convention where each image is mounted at a deterministic path so we can check if it is mounted (still won't work with containers)? Just document this behavior?

[cgwalters]

One idea: have a suggested mapping from "composefs repo" to "state in /run". For the system repo it'd be /run/composefs/refs/mounted/<image> and when we mount an image we drop a symlink in there to the EROFS and treat it as a GC root and also block deletion of mounted images.

Alternatively, refs/mounted could live in the repo itself, however it wouldn't work with a readonly storage; though we could special case that anyways since there's no GC.

So yeah this is probably just something like refs/mounted in the repo where the erofs is hardlinked?

[giuseppe]

what happens if there are multiple users mounting the same image, at what point it is safe to remove the symlink?

Should it be /run/composefs/refs/mounted/<image>/<user>?

Is it something we want to expose to users or we add an unmount command to undo the mount?

[cgwalters]

Yes, unprivileged users mounting from a system repository store is the hardest and most interesting case. This also of course strongly relates to composefs/composefs#409 - if we did that, then we could offer a native EROFS mount plus GC root handling.

Actually an overall interesting sub-question here is whether the composefs repository mode should be 0700 or 0755. I guess it's "configurable" of course.

Should it be /run/composefs/refs/mounted/<image>/<user>?

But what writes to that? A systemd service (e.g. speaking varlink)? Other alternatives feel hacky like creating a writable dir per logged-in user

[giuseppe]

But what writes to that? A systemd service (e.g. speaking varlink)? Other alternatives feel hacky like creating a writable dir per logged-in user

sorry I wasn't clear. With user I didn't mean another system user but another process, e.g. podman vs bootc.

If both wants to mount the same image, at what point will it be safe to unlink the symlink? I don't think one symlink is enough, we need one for caller

[cgwalters]

Ah then I'd s/user/component/ or so in the terminology ("process" is already taken).

Hmm...for privileged callers that can write to the repo, why wouldn't we just use a flock() on the EROFS file?

[giuseppe]

Hmm...for privileged callers that can write to the repo, why wouldn't we just use a flock() on the EROFS file?

who will hold the fd open for native mounts?

[alexlarsson]

So, in flatpak we solve this issue for deploy directores using locks. It works like this:

A deploy dir used for an app or runtime in a newly started app has a .ref file, like for example ~/.local/share/flatpak/app/org.gnome.gedit/x86_64/stable/819d067f59de3da21100ef9a752e0f87770bb44c975bc27430307fb66e41ec14/files/.ref. Every time we start a new flatpak app that bind-mounts that directory we take a non-exclusive lock on this file: fcntl (lock_fd, F_SETLKW, &non_exclusive_lock). This lock will be inherited by the process as we fork and exec the app itself. So, until pid 1 dies there will be a non-exclusive lock on it.

To remove a deploy directory we first atomically move it to a "deleted apps" direcotry, and then we try to take an exclusive lock on it (fcntl (lock_fd, F_SETLK, &l)). If this succeeds we delete the directory, otherwise we leave it be, and it will be GC:ed at a later time.

I think we can do something similar. We could have every user of an composefs erofs image take a non-exclusive lock on the image object, and then consider image files where we can't take an exclusive lock to be GC roots, even if no other root points at it. Then we could have the fuse mount grab the lock, as well as other users of mounts. So, you could remove the ref if you wanted, but the image itself and its referenced external object will not be GC:ed until the final user dies.

Of course, this only works for apps that takes a lock like this, but it would be in the interest of apps using composefs to do this.

[alexlarsson]

Not sure who would own the lock for the bootc root composefs mount though?

[alexlarsson]

Hmm...for privileged callers that can write to the repo, why wouldn't we just use a flock() on the EROFS file?
who will hold the fd open for native mounts?

Generally there would be some user processs who can do this. In flatpak, we have bwrap leak an fd into the app with the lock. In podman we could own the fd in conmon.

[giuseppe]

I am currently playing with some new kernel APIs right now, let's see how far I go with this attempt

[alexlarsson]

Also, note that flatpak uses F_SETLK even for system owned apps, so there is no need for the locker to have write permissions to the file. Not sure if this works with flock() too.

[giuseppe]

I've worked on a way to avoid a daemon to hold the mount fd open: https://github.com/giuseppe/composefs-rs/tree/reuse-mounts-and-prevent-gc

but it needs some kernel changes. I'll propose them, let's see how it goes. I've not managed to achieve anything similar with what is currently available: https://github.com/giuseppe/linux/tree/open-mount-by-handle

[giuseppe]

and a version that uses ioctls to get the lower dirs from the overlay mount:

• https://github.com/giuseppe/linux/tree/ovl-ioctls

• https://github.com/giuseppe/composefs-rs/tree/reuse-mounts-and-prevent-gc-overlay

@hsiangkao @amir73il what do you think of the kernel patches? Any comments/suggestions?

[cgwalters]

and a version that uses ioctls to get the lower dirs from the overlay mount:

Yes...we definitely need this, xref e.g. systemd/systemd#35017 (comment)

[hsiangkao]

and a version that uses ioctls to get the lower dirs from the overlay mount:

• https://github.com/giuseppe/linux/tree/ovl-ioctls
• https://github.com/giuseppe/composefs-rs/tree/reuse-mounts-and-prevent-gc-overlay

@hsiangkao @amir73il what do you think of the kernel patches? Any comments/suggestions?

I think on my side it's fine.

but I guess it could be a generic ioctl uapi for filesystems to implement (get source fd), at least reserve an vfs ioctl number? since the community might not like introducing a random ioctl for a specific filesystem (e.g. since it's more like a generic one), but I think we might need Christian's (@brauner) suggestion on this too...