Snakemake jobs killed by SLURM (timeout/OOM/cancel) are never detected — pipeline hangs indefinitely #141

Summary

When a Snakemake job submitted to SLURM is killed by the scheduler (walltime, OOM, node failure, scancel), the ASPEN master driver job never notices and hangs indefinitely instead of retrying or failing. --keep-going and --restart-times never kick in because Snakemake is never told the job died.

How this was found

While validating PR #140 on a real dataset (/data/Boufraqech_group/analysis/.temp/aspen_run_for_Ying_test2), the run stalled at 160/213 steps (75%) for over a day. The master aspen SLURM job was still RUNNING, but no child jobs were running or queued.

Root cause of the stall: the align rule for one replicate (U2OS_Control_03) hit the resources/cluster.json walltime limit (24:00:00) and was killed by SLURM mid-way through its last step (samtools index), confirmed via sacct:

29596375     align.rep+    TIMEOUT      0:0 2026-09-11T17:51:22 2026-09-12T17:51:30 1-00:00:08
29596375.ba+      batch  CANCELLED     0:15 ...

This was not a bug introduced by PR #140 — the new dedup/nondedup and spike-in fail-fast code both ran correctly. It's a pre-existing gap in how ASPEN talks to SLURM.

Root cause

aspen/bin/aspen generates submit_script.sbatch using Snakemake's generic cluster integration:

--cluster "sbatch --cpus-per-task {cluster.threads} -p {cluster.partition} -t {cluster.time} --mem {cluster.mem} ..." \
-j 500 --rerun-incomplete --restart-times 2 --keep-going \

With no --cluster-status script, Snakemake's GenericClusterExecutor only knows a job is "done" by checking whether its expected output files eventually appear. If SLURM kills the job before it writes those files, Snakemake has no signal that anything went wrong — it just waits forever. --restart-times/--keep-going are effectively dead code in this failure mode.

Recommendation

Add a --cluster-status script that queries sacct for the actual SLURM job state, and wire it (plus --cluster-cancel scancel for clean teardown) into the submit_script.sbatch template in aspen/bin/aspen.

Example resources/slurm_status.sh:

#!/usr/bin/env bash
# Usage: slurm_status.sh <jobid>
# Prints "running", "success", or "failed" for Snakemake's --cluster-status.

jobid="$1"
state=$(sacct -j "$jobid" --format=State --noheader --parsable2 2>/dev/null | head -n1 | tr -d ' ')

if [[ -z "$state" ]]; then
  echo "running"   # sacct DB write can lag briefly; don't false-positive as failed
  exit 0
fi

case "$state" in
  COMPLETED) echo "success" ;;
  PENDING|RUNNING|COMPLETING|CONFIGURING|SUSPENDED|RESIZING) echo "running" ;;
  FAILED|TIMEOUT|CANCELLED*|NODE_FAIL|OUT_OF_MEMORY|BOOT_FAIL|DEADLINE|PREEMPTED) echo "failed" ;;
  *) echo "running" ;;
esac

Wire-up in submit_script.sbatch:

--cluster "$CLUSTER_SBATCH_CMD" \
--cluster-status resources/slurm_status.sh \
--cluster-cancel scancel \

With this in place, a walltime/OOM/node-failure kill is correctly reported as failed, so --restart-times retries it automatically (helpful for transient node issues) and, if retries are exhausted, Snakemake exits with a clear error instead of hanging silently for days.

Additional context

Separately, the align rule's default walltime (24:00:00 in resources/cluster.json) was insufficient for at least one real-world hs1_chrR replicate. Worth revisiting this default (or documenting how to bump it per-run) independently of the --cluster-status fix.

_{⚡ Generated using AI ⚡}