ci: stop the nextflow stub run and e2e data fills from failing on disk and slow mirrors, and give every e2e case its timing row

.github/actions/free-disk/action.yml

@@ -17,23 +17,39 @@ runs:
         df -h / /mnt 2> /dev/null || df -h /
         before=$(df --output=avail -k / | tail -1)
         # The deletions run side by side: the Android SDK alone is about
-        # 100,000 files. A path that is not there is skipped.
-        for d in /usr/local/lib/android /usr/share/dotnet /opt/ghc /usr/local/.ghcup \
+        # 100,000 files. A path that is not there is skipped. The cleanup is
+        # best effort, but a part that fails is named in the log and the
+        # summary instead of showing only as a smaller number freed.
+        # FREE_DISK_PATHS replaces the list (tests/test_e2e_data_cache.sh).
+        paths=${FREE_DISK_PATHS:-/usr/local/lib/android /usr/share/dotnet /opt/ghc /usr/local/.ghcup \
                  /usr/local/share/boost /usr/share/swift /usr/local/share/powershell \
                  /usr/local/share/chromium /opt/microsoft /opt/google /opt/az \
                  /opt/hostedtoolcache/CodeQL /opt/hostedtoolcache/go /opt/hostedtoolcache/PyPy \
                  /opt/hostedtoolcache/Python \
-                 /opt/hostedtoolcache/Ruby /opt/hostedtoolcache/node; do
-          [ -e "$d" ] && sudo rm -rf "$d" &
+                 /opt/hostedtoolcache/Ruby /opt/hostedtoolcache/node}
+        pids=() what=()
+        for d in $paths; do
+          [ -e "$d" ] || continue
+          sudo rm -rf "$d" & pids+=($!); what+=("rm -rf ${d}")
         done
         # The images the runner image ships (none of them is ours).
         if command -v docker > /dev/null 2>&1; then
-          docker image prune -a -f > /dev/null 2>&1 &
+          docker image prune -a -f > /dev/null 2>&1 & pids+=($!); what+=("docker image prune -a")
         fi
-        wait
+        failed=()
+        for i in "${!pids[@]}"; do
+          rc=0
+          wait "${pids[$i]}" || rc=$?
+          [ "$rc" -eq 0 ] || failed+=("${what[$i]} (exit ${rc})")
+        done
         after=$(df --output=avail -k / | tail -1)
         echo "Free space after:"
         df -h / /mnt 2> /dev/null || df -h /
         msg="Free disk: $(( (after - before) / 1048576 )) GB freed on /, $(( after / 1048576 )) GB free now, in ${SECONDS} s"
+        if [ "${#failed[@]}" -gt 0 ]; then
+          msg="${msg}; cleanup failed (best effort, the job goes on): $(printf '%s; ' "${failed[@]}")"
+          msg=${msg%; }
+          echo "::warning::${msg}"
+        fi
         echo "$msg"
         echo "$msg" >> "$GITHUB_STEP_SUMMARY"

.github/workflows/e2e.yml

@@ -698,9 +698,10 @@ jobs:
       # prs-scores-and-run-options-2, 268 MB), from EBI's FTP server, keyed
       # on the sha256 tests/e2e/lib.sh pins. The cases take it from
       # E2E_DATA_CACHE/hapnest, check it and then run setup.sh with EBI
-      # unreachable; on a miss the first one downloads it and puts it there
-      # (setup_ancestry_panel in lib.sh). Saved like the entries above: on
-      # main, or a run by hand with save_images, on a miss.
+      # unreachable; on a miss the first one takes it from E2E_DATA_MIRROR
+      # (else EBI, through setup.sh) and puts it there (setup_ancestry_panel
+      # in lib.sh). Saved like the entries above: on main, or a run by hand
+      # with save_images, on a miss.
       - name: Data cache, HAPNEST panel
         id: hapnest
         env:
@@ -730,8 +731,12 @@ jobs:
           E2E_TRACE: ${{ needs.e2e-plan.outputs.trace }}
           # The free disk a leg needs before its first case (e2e-run.sh).
           E2E_MIN_FREE_GB: 10
-          # Copies of the pinned e2e data, tried before EBI (data_fetch in e2e-run.sh).
+          # Copies of the pinned e2e data, tried before EBI (data_fetch in
+          # e2e-run.sh, setup_ancestry_panel in tests/e2e/lib.sh).
           E2E_DATA_MIRROR: https://github.com/GeiserX/OwnGenome/releases/download/e2e-data-1
+          # The most seconds one data cache fill may take; a fill still
+          # running then fails its case instead of the leg's time limit.
+          E2E_DATA_BUDGET: 2400
           E2E_BASE_OUT: ${{ matrix.job == 'base' && format('{0}/base.tar.zst', runner.temp) || '' }}
           E2E_BASE_IN: ${{ matrix.job != 'base' && format('{0}/e2e-base/base.tar.zst', runner.temp) || '' }}
           # Keep the downloads to the end of the job only when a save step uses them.

.github/workflows/nextflow.yml

@@ -122,6 +122,30 @@ jobs:
           python3 -m json.tool nextflow_schema.json > /dev/null
           python3 -m json.tool assets/schema_input.json > /dev/null
 
+      # The stub run below pulls every image of the default tools: the pinned
+      # leg ran out of disk in it twice on 2026-11-05 (#300, job 127077180393;
+      # #290, run 41966270772). The free-disk step above runs first. On that
+      # leg, the free space under / and under Docker's root goes to the log
+      # and the summary, and
+      # below MIN_FREE_GB the job stops here, naming the cause, instead of
+      # failing in a task or in Nextflow's onComplete handler.
+      - name: Free disk before the stub runs
+        if: steps.scope.outputs.run == 'true' && matrix.line == 'pinned'
+        env:
+          MIN_FREE_GB: 20
+          LINE: ${{ matrix.line }}
+        run: |
+          root=$(docker info --format '{{.DockerRootDir}}' 2> /dev/null || true)
+          root=${root:-/var/lib/docker}
+          sudo df -h / "$root" "$RUNNER_TEMP" || df -h /
+          free=$(sudo df --output=avail -k "$root" | tail -1)
+          msg="Free disk before the stub runs (nextflow ${LINE}): $((free / 1048576)) GB under the Docker root ${root}, at least ${MIN_FREE_GB} GB wanted"
+          echo "$msg" | tee -a "$GITHUB_STEP_SUMMARY"
+          if [ "$free" -lt $((MIN_FREE_GB * 1048576)) ]; then
+            echo "::error::${msg}: the runner has too little disk for the stub runs' images; re-run the job"
+            exit 1
+          fi
+
       # The default tools, each stub run inside its own image.
       - name: Stub run of the test profile with Docker
         if: steps.scope.outputs.run == 'true' && matrix.line == 'pinned'
@@ -664,6 +688,7 @@ jobs:
       - name: Show what Nextflow was waiting for
         if: failure() && steps.scope.outputs.run == 'true'
         run: |
+          echo "Free disk when the job failed:"; df -h / "$RUNNER_TEMP" || true
           log=.nextflow.log
           [ -f "$log" ] || { echo "No .nextflow.log: no Nextflow run had started."; exit 0; }
           # A report, not a check: no command here may stop the step, and a

assets/steps.tsv

@@ -72,7 +72,7 @@ step	label	tool	recorded	processes	sections	upstream	hours	hours_from	hours_low
 47	Viral DNA	viral	yes	VIRAL_READS,VIRAL_REPORT	viral	02				
 48	Exomiser (HPO terms)	exomiser	yes	EXOMISER_PREPARE,EXOMISER,EXOMISER_REPORT	exomiser	03				
 49	CHIP (clonal blood cells)	chip	yes	CHIP_PREPARE,CHIP_MUTECT2,CHIP_ANNOTATE,CHIP_REPORT	chip	02				
-50	Mobile element insertions	mei	yes	MEI_DEPTH,MEI,MEI_REPORT	mei	02				
-51	Mosaic alterations (MoChA)	mocha	yes	MOCHA_PREP,MOCHA_PHASE,MOCHA_CALL,MOCHA_REPORT	mosaic	03				
-52	MUC1 VNTR (VNtyper)	vntyper	yes	VNTYPER	vntyper	02				
-53	Hard loci (Locityper)	locityper	yes	LOCITYPER	locityper	02				
+50	Mobile element insertions	mei	yes	MEI_DEPTH,MEI,MEI_REPORT	mei	02				MEGANE_IMAGE
+51	Mosaic alterations (MoChA)	mocha	yes	MOCHA_PREP,MOCHA_PHASE,MOCHA_CALL,MOCHA_REPORT	mosaic	03				MOCHA_PANEL_VERSION,MOCHA_IMAGE
+52	MUC1 VNTR (VNtyper)	vntyper	yes	VNTYPER	vntyper	02				VNTYPER_IMAGE
+53	Hard loci (Locityper)	locityper	yes	LOCITYPER	locityper	02				LOCITYPER_PANEL_VERSION,LOCITYPER_IMAGE

bin/owngenome_recheck.py

@@ -79,6 +79,13 @@
 )
 
 
+# The opt-in steps' pins: the setup.sh option that installs their data and image (the main setup leaves
+# them out). VNtyper's reference data is inside its image, which the run pulls, so it needs no option.
+OPT_IN_SETUP = {"MEGANE_IMAGE": "--megane-kmers", "MOCHA_PANEL_VERSION": "--mocha-data", "MOCHA_IMAGE": "--mocha-data",
+                "LOCITYPER_PANEL_VERSION": "--locityper-data", "LOCITYPER_IMAGE": "--locityper-data",
+                "VNTYPER_IMAGE": None}
+
+
 class Failed(Exception):
     """A lookup with no usable answer."""
 
@@ -387,6 +394,11 @@ def classify(item, release_tag, release_pins, release_error, clinvar=False):
 def action(item, sample):
     c, key = item["class"], item["item"]
     if c == "rerun-ready":
+        if item["kind"] == "pin" and key in OPT_IN_SETUP:
+            opt = OPT_IN_SETUP[key]
+            how = (f"run ./scripts/setup.sh {opt} $GENOME_DIR (the main setup leaves this step's data out)"
+                   if opt else "the run pulls the image")
+            return f"this checkout pins {item['installed']}: {how}, then rerun: bin/owngenome_recheck.py --sample {sample} --rerun"
         if item["kind"] == "pin":
             return (f"this checkout pins {item['installed']}: run ./scripts/setup.sh (it installs the data), "
                     f"then rerun: bin/owngenome_recheck.py --sample {sample} --rerun")
@@ -437,6 +449,18 @@ def add(item, fn, *args, clinvar=False):
         if key == "hla_database" and not this_hla:
             item["error"] = "could not read this checkout's HLA_DB_RELEASE (scripts/lib/common.sh)"
         add(item, answers.pinned, key, lookup, pattern)
+    # The other versions.env pins a step reads (steps 50 to 53): no freshness source exists for them, so
+    # nothing is looked up; a pin here that differs from the run's makes its step rerun-ready.
+    known = {k for k, *_ in PINNED}
+    for key in dict.fromkeys(r[:-5] if r.endswith("@self") else r for s in steps for r in reads_of(s)):
+        if key in known or key not in this_pins:
+            continue
+        ran, inst = pin_value(key, version.get(key)), pin_value(key, this_pins.get(key))
+        item = {"item": key, "label": f"{key} (no upstream lookup)", "kind": "pin", "ran_with": ran,
+                "installed": inst, "class": "rerun-ready" if ran and inst and ran != inst else "no-newer-release"}
+        item["steps"] = affected_steps(steps, [key])
+        item["action"] = action(item, sample)
+        items.append(item)
     for (sec, key), value in sorted(man.items()):
         if sec != "data" or not key.startswith("pgs:"):
             continue

docs/53-locityper.md

@@ -87,7 +87,7 @@ Both reports carry a section "Locityper Second Opinion at Hard Loci" in the medi
 
 ## How it was checked
 
-The image test (`tests/smoke/locityper.sh`, the `LOCITYPER_IMAGE` row of `tests/smoke/commands.tsv`) runs the database build and the step on the fixture's GIAB HG002 reads at HLA-DRB1, HLA-DQB1 and CYP2D6. The panel holds HG002's own two assembled haplotypes, which are the truth. Both called haplotypes at each locus must match them at QV 20 or more (one difference per 100 bases or fewer), aligned with `locityper align`.
+The image test (`tests/smoke/locityper.sh`, the `LOCITYPER_IMAGE` row of `tests/smoke/commands.tsv`) runs the database build and the step on the fixture's GIAB HG002 reads at HLA-DRB1, HLA-DQB1 and CYP2D6. The panel holds HG002's own two assembled haplotypes, which are the truth. Both called haplotypes at each locus must match them at QV 20 or more (one difference per 100 bases or fewer), aligned with `locityper align`. That QV measures how close each called haplotype is to HG002's true one; it is not the `quality` column, Locityper's own genotype quality.
 
 The red control takes out of the panel HG002's haplotypes and every other haplotype within QV 20 of them, rebuilds the database and genotypes again; the same check must then fail at every locus. Leaving HG002 out with `--leave-out` is not enough: other people in the panel carry near copies of HG002's haplotypes, and the call stays close.
 

docs/lessons-learned.md

@@ -638,3 +638,11 @@ Most bioinformatics containers run as non-root users. If writing to bind-mounted
 - **Failed:** in November 2026, fork runs and a cache-seeding run with no Actions cache for the two data entries had shards 2 and 5 cancelled at the 240-minute leg limit, still downloading. An Actions cache belongs to one repository, so a fork starts with none, and an evicted entry leaves main in the same place.
 - **Why:** both EBI hosts for 1000 Genomes (`ftp.1000genomes.ebi.ac.uk` and `ftp.ebi.ac.uk/1000g`) swung between a few KB/s and about 3 MB/s from one request to the next in the same hour. No other copy of the pinned v5c sites file exists: AWS's `1000genomes` bucket holds only v5b, and NCBI's 1000 Genomes path answers 404.
 - **Fix:** the pinned files are kept as assets of the GitHub release `e2e-data-1`, and `data_fetch` in `scripts/ci/e2e-run.sh` tries `E2E_DATA_MIRROR` once before the source. The pin checks both copies, so the mirror changes only where the bytes come from. A file missing from the release falls back to EBI.
+
+### A data fill with no time cap could still eat a leg's limit
+- **Failed:** with the mirror in place, a data fill that falls back to EBI had no bound on its total time. `fetch` cuts only a stall (under 1 KB/s for two minutes), so a transfer that moves slowly runs on, and two shard legs reached their 240-minute limit on the 1000 Genomes sites VCF (fork run 41957608395). The PGS Catalog score files and the HAPNEST panel had no mirror copy at all.
+- **Fix:** `data_fill` in `scripts/ci/e2e-run.sh` runs each fill within `E2E_DATA_BUDGET` seconds (2400 in `e2e.yml`) and stops it after that, with a message; its case then fails without running. The four score files and the HAPNEST panel are in `e2e-data-1` too, pinned by md5 (`PGS_MD5`) and sha256 (`HAPNEST_SHA256`), and tried there before EBI.
+
+### free-disk hid a cleanup that failed
+- **Failed:** `.github/actions/free-disk` ran its deletions and the image prune in the background and never read their exit codes, so a failed part showed only as fewer GB freed.
+- **Fix:** each part is waited for; a failure is a warning and a line in the job summary, and the step still passes. `tests/test_e2e_data_cache.sh` plants a failing `rm` and a failing prune.