Potential pulling / syncing optimization

[travier]

composefs-rs used to fsync each file written when pulling which was costly.

This got changed to a global syncfs call in c3677f1.

We could potentially do better by following a logic similar to what's done in ostree:

• When pulling, link all the files to be written into a tmp folder
• Then fsync the entire folder once ready to move them into the composefs repo
• Then complete the repo import
• We would have a tmp directory per pull operation to enable parallel pulls

Some notes from Pragyan:

Adding some stats related to fsync:
For 37857 objects, we call fsync 4 times (I’m not sure at what point we sync everything)

[cgwalters]

This is a hugely complex topic but just to repeat something I said in the other PR, I think a key thing to try here is to let writeback do its thing ideally by default - IOW if someone fetches an image (app or OS update) and then their laptop dies right after - I think most of the time it's OK if that image isn't there!

We could have a concept of "committed" vs "uncommitted" refs/content. My hope here is at the kernel side we're just doing a lot of fsyncs on inodes which are not dirty, and then no journal traffic is required at all.

[alexlarsson]

Yes, I think its fine if the image isn't there. What I think is important to avoid is the case wherewe download an image, and in the middle of the pull operation the laptop dies. Then, on next boot, the image (splitstream, ref, etc) isn't there (which is fine), but there are some external files referenced by that splitstream in the objects dir that may be partial or corrupted. This is bad, because when you next time pull the same image, it will see these objects and not re-download them again, breaking the re-pull.

So, my main worry is about introducing valid names (i.e. matching the expected digest), in the objects directory before they have been sync:ed. This is why I think we should download the new files to a separate folder (or with some filename prefix) and not move them into the "final" place before the file content has been fs-synced. "Losing files" is "ok", but corrupt objects is less ok.

[alexlarsson]

For example, I think we currently repo.sync() (ie. syncfs) after each layer splitstream has been written. What if we created all the external object in that file with a prefix, or in a separate dir, then in sync() we syncfs(); move_files_in_place(); syncfs().