Binary cache system proposal

[EarldridgeJazzedPineda]

It's finally time for me to describe my proposal for the binary cache system:
A binary cache consists of tarballs compressed using XZ. Each tarball corresponds to a package in /ix/store (not including realms). Here is what a binary cache would look like:

0fV5uqtPXTsrkX7JskmFa5-bin-sched-builddir-scripts.tar.xz
0LN9hQ6OogcSMvCXN1P040-bin-ip-utils.tar.xz
0v1daEETXZaarNMrrRF0L1-bin-mingetty-runit-scripts.tar.xz
0X0lHjAfccfdd3Vs4y4Se0-bin-glib-codegen-old.tar.xz
0Xke54dzOWTyPt8chp2n27-bin-nasm-old.tar.xz
2qHcAltBuYlW3pk0Kz1cW2-bin-acpi-d-runit-scripts.tar.xz
2QRbV1Q33yCGSyErTPKk50-aux-magic.tar.xz
3h5antwOejZN3tGDR7EKo1-bin-sched-scripts.tar.xz
4p066NgWSQz04zD9DjoT06-bin-pkg-config.tar.xz
4Tr9PIePRhd4BphvUaY097-bin-bash-lite.tar.xz
...

Binary caches will be used to provide pre-built packages in stal/IX so that users don't have to build large packages (such as LLVM) from source code. For context, LLVM accounts for around 56% of the time needed to bootstrap stal/IX. It takes up 47% of space in a rootfs tarball, with the Clang executable alone taking up 10%. The packages that define a binary cache for x86 (64-bit) are set/ci/unwrap/linux/x86_64, set/ci/tier/1 and set/ci/tier/2 (which, according to Anton, costs 800 CPU-hours).

There will be two binary caches, one for standalone IX installations (IX_EXEC_KIND=local) and one for stal/IX (IX_EXEC_KIND=system). The following variables are set:

• IX_ROOT=/ix
• If IX_EXEC_KIND=local, the IX repository is at /home/ix/ix and IX is invoked using /usr/bin/python3
• No flags except those required to install stal/IX i.e. --failsafe and those used for 3D acceleration (shown below)
• --opengl=soft or --opengl=angle --vulkan=swift/shader flags for 3D acceleration (some packages require this to be set)
• No bin/ix/tmpfs installed

Additional information on reproducible builds: https://reproducible-builds.org/docs/

The binary cache workflow goes as follows:

1. Set up an IX installation, either with IX_EXEC_KIND=system or with IX_EXEC_KIND=local.
2. Build the set/ci/unwrap/linux/x86_64, set/ci/tier/1 and set/ci/tier/2 package sets (the name of the realm doesn't matter).
3. Take all directories in /ix/store that aren't rlm, lnk, or url packages and compress them using XZ into tarballs.
4. Upload the tarballs to the binary cache.
5. When the IX repository gets a new commit, rebuild the package sets and take note of the package hashes between the old and new commits. Beware of commits that can trigger a full rebuild.
6. Add package hashes that are in the new commit but not in the old one to the binary cache.
7. If needed, delete package hashes that are in the old commit but not in the new one from the binary cache.
8. Repeat from step 5.

Binary caches are distributed using the GitHub container registry or Hugging Face, or if not feasible, using a self-hosted mirror. Here's how IX will use a binary cache:

1. IX will try to fetch a tarball of the package (and its runtime dependencies obtained recursively, if any) that has a matching hash from binary caches. This process is independent from the build graph.
2. If a tarball was found, then it will extract the contents of that tarball to the appropriate location in /ix/store .
3. If no tarball was found after going through all binary cache mirrors, then it will fallback to building the package from source code. Its dependencies are recursively fetched from binary caches using steps 1 to 3.

Anton said that if the cache was easy to make, it would have been made a long time ago. But it's better to do it now than later.

[EarldridgeJazzedPineda]

@pg83 I've finalized my proposal and it should be good to go. All that is left to do is to set up an actual binary cache and add the necessary code to IX.