ci(e2e): keep every e2e leg inside the runner's disk

.github/workflows/e2e.yml

@@ -417,6 +417,31 @@ jobs:
         with:
           version: "26.10.5"
 
+      # Docker 29 on the ubuntu-26.04 image keeps images in containerd's
+      # store, which holds each layer twice: the compressed blob in its
+      # content store and the unpacked snapshot. DeepVariant's, GATK's and
+      # VNtyper's images took 24.8 GB of disk there and 17.6 GB in the
+      # classic overlay2 store, on 26.04 and on 24.04 alike (probe run
+      # 41957796291 on the geiresx fork). With every case image on it, a
+      # 72 GB runner (48 GB free) ran out in the pre-pull (run 41938436657,
+      # shards 2 and 5). Runs before any image is restored or pulled; the
+      # images already in the containerd store are pruned first, as overlay2
+      # would not see them.
+      - name: Docker image store
+        run: |
+          if docker info -f '{{json .DriverStatus}}' | grep -q io.containerd.snapshotter; then
+            docker image prune -a -f > /dev/null
+            f=/etc/docker/daemon.json
+            { [ -s "$f" ] && sudo cat "$f" || echo '{}'; } \
+              | jq '.features["containerd-snapshotter"] = false' | sudo tee "${f}.new" > /dev/null
+            sudo mv "${f}.new" "$f"
+            sudo systemctl restart docker
+          fi
+          driver=$(docker info -f '{{.Driver}}')
+          echo "Docker $(docker version -f '{{.Server.Version}}'), storage driver ${driver}" | tee -a "$GITHUB_STEP_SUMMARY"
+          [ "$driver" = overlay2 ] || { echo "::error::Docker's storage driver is ${driver}, not overlay2."; exit 1; }
+          df -h /
+
       # Images are cached in four entries, one per group of
       # `scripts/ci/e2e-run.sh --image-groups`, each keyed on its own image
       # list: a version bump misses one group's key, and that group restores
@@ -432,7 +457,11 @@ jobs:
           for g in 0 1 2 3; do
             # Only refs that can be saved: a digest bump never changes an entry.
             h=$(awk -v g="$g" '$1 == g && $2 !~ /@/ {print $2}' "${RUNNER_TEMP}/image-groups.txt" | sort | sha256sum | cut -c1-16)
-            echo "g${g}=e2e-images-g${g}-${h}" | tee -a "$GITHUB_OUTPUT"
+            # "ovl": saved from the overlay2 store (step "Docker image store").
+            # 7 of the 46 archives saved from containerd's store did not load
+            # into overlay2 (run 41960912641), and an exact key hit never
+            # saves again, so the old entries are left behind.
+            echo "g${g}=e2e-images-ovl-g${g}-${h}" | tee -a "$GITHUB_OUTPUT"
           done
 
       - name: Restore cached images, group 0
@@ -441,31 +470,31 @@ jobs:
         with:
           path: ${{ runner.temp }}/images/g0
           key: ${{ steps.keys.outputs.g0 }}
-          restore-keys: e2e-images-g0-
+          restore-keys: e2e-images-ovl-g0-
 
       - name: Restore cached images, group 1
         id: images1
         uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
         with:
           path: ${{ runner.temp }}/images/g1
           key: ${{ steps.keys.outputs.g1 }}
-          restore-keys: e2e-images-g1-
+          restore-keys: e2e-images-ovl-g1-
 
       - name: Restore cached images, group 2
         id: images2
         uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
         with:
           path: ${{ runner.temp }}/images/g2
           key: ${{ steps.keys.outputs.g2 }}
-          restore-keys: e2e-images-g2-
+          restore-keys: e2e-images-ovl-g2-
 
       - name: Restore cached images, group 3
         id: images3
         uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
         with:
           path: ${{ runner.temp }}/images/g3
           key: ${{ steps.keys.outputs.g3 }}
-          restore-keys: e2e-images-g3-
+          restore-keys: e2e-images-ovl-g3-
 
       # One image per file; the four groups load side by side. An image that
       # does not load is pulled by e2e-run.sh like any missing image.
@@ -483,6 +512,8 @@ jobs:
                 [ -f "$f" ] || continue
                 if zstd -dc "$f" | docker load -q > /dev/null; then echo "${f##*/}" >> "$loaded"
                 else echo "::warning::${f##*/} did not load; e2e-run.sh pulls it"; fi
+                # Loaded, the archive only takes disk.
+                rm -f "$f"
               done
             ) &
           done
@@ -588,6 +619,8 @@ jobs:
           E2E_CASES: ${{ steps.cases-list.outputs.list }}
           E2E_JOB: ${{ matrix.job == 'base' && 'base' || format('shard {0}', matrix.job) }}
           E2E_TRACE: ${{ needs.e2e-plan.outputs.trace }}
+          # The free disk a leg needs before its first case (e2e-run.sh).
+          E2E_MIN_FREE_GB: 10
           E2E_BASE_OUT: ${{ matrix.job == 'base' && format('{0}/base.tar.zst', runner.temp) || '' }}
           E2E_BASE_IN: ${{ matrix.job != 'base' && format('{0}/e2e-base/base.tar.zst', runner.temp) || '' }}
           # Keep the downloads to the end of the job only when a save step uses them.

docs/lessons-learned.md

@@ -610,6 +610,11 @@ Most bioinformatics containers run as non-root users. If writing to bind-mounted
 - **Fix:** `tests/e2e/mei-1.sh` removes its whole sample directory and the image by its id when it ends, and logs the free disk before and after. With the image gone every case passed. The image is pinned by digest, so the e2e image cache never saves it anyway.
 - **Image sizes step:** the job's "Image sizes" step then stopped with `numfmt: invalid number: ''`. For an image that is gone, `docker image inspect -f` prints an empty line before the `|| echo 0` fallback, so the step's own "not here" branch never ran. The fix in `.github/workflows/e2e.yml` reads the size into a variable first (`size=$(docker image inspect ...) || size=0`) and prints `${size:-0}`.
 
+### Docker's containerd image store on ubuntu-26.04 doubles the e2e images on disk
+- **Failed:** shards 2 and 5 of main's run 41938436657 stopped in the pre-pull with "no space left on device" under `/var/lib/containerd`, while pulling MEGAnE's image. The same commit passed on other legs.
+- **Why:** `ubuntu-latest` hands out runners of two sizes, 145 GB and 72 GB (48 GB free after the clean-up step), and on the ubuntu-26.04 image Docker 29 keeps images in containerd's store, which holds each layer as a compressed blob and as an unpacked copy: DeepVariant's, GATK's and VNtyper's images took 24.8 GB there against 17.6 GB on overlay2. In that store `docker image inspect -f '{{.Size}}'` gives the compressed size (7.2 GB for the same three), so the image table understated what the disk held. Every leg also pre-pulled all 56 case images, the five largest included, whatever its cases ran.
+- **Fix:** each e2e leg turns `containerd-snapshotter` off in `/etc/docker/daemon.json` and restarts Docker before it restores an image, so images go to overlay2; the five largest images are pulled at a case's first `docker run` of them (`tests/e2e/bin/docker`) and removed when the case ends; the image cache keys moved to `e2e-images-ovl-`, because 7 of the 46 archives saved from containerd's store did not load into overlay2; and `e2e-run.sh` fails a leg with less than `E2E_MIN_FREE_GB` free before its first case, so a short disk shows as one clear error instead of a failure in a pull or a case.
+
 ## Downloads (2026-11)
 
 ### One bad transfer of a large file failed the whole setup

docs/testing.md

@@ -92,7 +92,9 @@ Two downloads are cached the same way, because they come from EBI's 1000 Genomes
 
 How long it took in one job, before the split: a passing e2e job took about 105 to 205 minutes with 84 cases, and 230 minutes of cases with 130 (run 41390434697, 2026-11-02). The runner decides much of it, not the cache: the same CPU model took 106 minutes in one run and 166 in another. In October 2026, with 84 cases, the same code took 106 and 166 minutes on an AMD EPYC 9V45 (runs 39197502832 and 39075176392, the second of which also saved the cache), 112 minutes on an Intel Xeon 6973P (run 39088086725), 156 and 191 minutes on an AMD EPYC 7763 (runs 39075764278 and 39075201465) and 205 minutes on an AMD EPYC 9V74 (run 39197512594, which saved three groups). The image cache saves about 7 minutes of that: pulling every image took 9 minutes, restoring and loading the four groups took 2.5 (run 39075201465 restored 39 images in 118 seconds), and saving them all took 12. Each leg has a limit of 240 minutes (355 for a traced shard leg); the base leg waits up to 75 minutes for a fixture a pull request bumps. Each leg's first steps print the runner's CPU model and CPU count to the log and the job summary, so a later timing can name the hardware.
 
-Before the first case of a full run, and of every leg, `e2e-run.sh` pulls every image the cases use that is not on the machine yet: the `NAME_IMAGE` lines of `versions.env` and of the case files, less a list of images no case runs. A pull that fails with a registry 5xx is tried again, up to four tries in all, 20, 40 and 80 seconds apart. Any other error, such as a missing tag or a rate limit, fails the job at once, and the tools' own `docker run` is never retried. A Docker Hub 500 on one image's manifest is what failed main's run 37980248545 the first time. `scripts/ci/e2e-run.sh --self-test-pull` checks the retry against a fake docker, and `e2e-plan` runs it. A run given a pattern skips the pre-pull, because most images reach a case through the steps it runs and cannot be picked per case; its cases pull their own images, with one try each.
+Before the first case of a full run, and of every leg, `e2e-run.sh` pulls every image the cases use that is not on the machine yet: the `NAME_IMAGE` lines of `versions.env` and of the case files, less a list of images no case runs. A pull that fails with a registry 5xx is tried again, up to four tries in all, 20, 40 and 80 seconds apart. Any other error, such as a missing tag or a rate limit, fails the job at once, and the tools' own `docker run` is never retried. A Docker Hub 500 on one image's manifest is what failed main's run 37980248545 the first time. `scripts/ci/e2e-run.sh --self-test-pull` checks the retry against a fake docker, and `e2e-plan` runs it. A run given a pattern skips the pre-pull, because most images reach a case through the steps it runs and cannot be picked per case; its cases pull their own images, with one try each. The five largest images (DeepVariant, PharmCAT, GATK, MEGAnE and VNtyper, 1.9 to 6.0 GB each to download, none of them in the image cache) are left out of the pre-pull: the docker shim `tests/e2e/bin/docker` pulls one, with the same retry, when a case first runs it, and removes it when that case ends, so a leg holds at most one of them at a time. The image cache keys start with `e2e-images-ovl-`: archives saved from containerd's store did not all load into overlay2.
+
+Each leg switches Docker to the classic overlay2 image store before it restores or pulls an image. Docker 29 on the ubuntu-26.04 runner image uses containerd's store, which keeps every layer twice (the compressed blob and the unpacked copy): DeepVariant's, GATK's and VNtyper's images took 24.8 GB there and 17.6 GB on overlay2, and the legs that landed on a 72 GB runner (48 GB free) ran out of disk in the pre-pull (run 41938436657). Before its first case `e2e-run.sh` fails the leg when less than `E2E_MIN_FREE_GB` is free under the work area (10 GB in `e2e.yml`), and after the cases it prints the least free space it saw while they ran.
 
 What it covers: