fix(aspen): attempt-aware resource scaling and dry-run handling

CHANGELOG.md

@@ -1,5 +1,6 @@
 ## ASPEN development version
 
+- Add attempt-aware resource scaling for the main Snakemake rules, document how baseline `cluster.json` resources interact with retry-based `resources.gres` scaling, and fix dry-run failure handling so failed dry-runs now exit non-zero instead of reporting success. (#152, @kopardev)
 - Detect SLURM-killed Snakemake child jobs in classic cluster mode by adding a `--cluster-status` hook and `scancel` integration, so timed out, OOM-killed, cancelled, or otherwise failed child jobs now retry/fail cleanly instead of leaving the master workflow hanging indefinitely. (#148, @kopardev)
 - Clarify motif enrichment outputs and interpretation in the documentation: fix the reported location of HOMER/AME result folders, explain how ASPEN reuses HOMER-generated `target.fa`/`background.fa` in AME, document the AME parallelization strategy, and add simpler guidance for interpreting `knownResults.txt` and `ame_results.txt`. (#146, @kopardev)
 

aspen

@@ -326,7 +326,7 @@ function set_cluster_arg(){
   CLUSTER_SBATCH_CMD="sbatch --cpus-per-task {cluster.threads} -p {cluster.partition} -t {cluster.time} --mem {cluster.mem} --job-name {cluster.name} --output {cluster.output} --error {cluster.error}"
   clustername=$(scontrol show config|grep ClusterName|awk '{print $NF}')
   if [[ "$clustername" == "biowulf" ]];then
-    CLUSTER_SBATCH_CMD="${CLUSTER_SBATCH_CMD} --gres {cluster.gres}"
+    CLUSTER_SBATCH_CMD="${CLUSTER_SBATCH_CMD} --gres {resources.gres}"
   fi
 }
 
@@ -397,7 +397,14 @@ function dryrun() {
       mv ${WORKDIR}/dryrun_git_commit.txt ${WORKDIR}/logs/dryrun_git_commit.${modtime}.txt
     fi
   fi
-  run "--dry-run" | tee ${WORKDIR}/dryrun.log
+  set -o pipefail
+  run "--dry-run" 2>&1 | tee ${WORKDIR}/dryrun.log
+  dryrun_rc=${PIPESTATUS[0]}
+  set +o pipefail
+  if [ "${dryrun_rc}" -ne 0 ];then
+    log_error "Dry-run failed with exit code ${dryrun_rc}. See ${WORKDIR}/dryrun.log for details."
+    exit "${dryrun_rc}"
+  fi
   echo "Git Commit/Tag: $GIT_COMMIT_TAG" > ${WORKDIR}/dryrun_git_commit.txt
   log_ok "Dry-run completed."
   log_info "Full dry-run output captured in ${WORKDIR}/dryrun.log"
@@ -694,7 +701,7 @@ _START_TS=\$(date -u +"%Y-%m-%dT%H:%M:%SZ")
 _progress_monitor &
 _monitor_pid=\$!
 
-snakemake -s $SNAKEFILE --directory $WORKDIR $SINGULARITY_STR --printshellcmds --latency-wait 120 --configfile $CONFIGFILE --cluster-config $CLUSTERFILE --cluster "$CLUSTER_SBATCH_CMD" --cluster-status "$CLUSTERSTATUSCMD" --cluster-cancel scancel -j 500 --rerun-incomplete ${RERUNTRIGGERS} --restart-times ${retries} --keep-going --stats ${WORKDIR}/snakemake.stats 2>&1 | tee ${WORKDIR}/snakemake.log
+snakemake -s $SNAKEFILE --directory $WORKDIR $SINGULARITY_STR --printshellcmds --latency-wait 120 --configfile $CONFIGFILE --cluster-config $CLUSTERFILE --default-resources "gres='lscratch:256'" --cluster "$CLUSTER_SBATCH_CMD" --cluster-status "$CLUSTERSTATUSCMD" --cluster-cancel scancel -j 500 --rerun-incomplete ${RERUNTRIGGERS} --restart-times ${retries} --keep-going --stats ${WORKDIR}/snakemake.stats 2>&1 | tee ${WORKDIR}/snakemake.log
 snakemake_rc="\${PIPESTATUS[0]}"
 kill "\$_monitor_pid" 2>/dev/null || true
 wait "\$_monitor_pid" 2>/dev/null || true
@@ -770,6 +777,7 @@ snakemake $1 -s $SNAKEFILE \
 --latency-wait 120 \
 --configfile $CONFIGFILE \
 --cluster-config $CLUSTERFILE \
+--default-resources "gres='lscratch:256'" \
 --cluster "$CLUSTER_SBATCH_CMD" \
 -j 500 \
 --rerun-incomplete \

docs/deployment.md

@@ -262,6 +262,8 @@ aspen -m=run -w=<path_to_output_folder>
 
 This command submits a master job to the Slurm workload manager, which orchestrates the entire analysis workflow, managing job submissions and monitoring progress.
 
+ASPEN submits the generated Snakemake job with rule-level resources exposed to Slurm. The heavy rules that may need extra scratch space now define an attempt-aware `resources.gres` value, so the first submission uses the baseline `cluster.json` request and later retries can scale the requested `lscratch` allocation automatically if a rule is retried by Snakemake. This keeps the default configuration in `cluster.json` while still allowing individual rules to become more conservative on repeated failures.
+
 - 🛠️ **Optional Argument**:
 
 `--singcache` or `-c`: Specify a Singularity cache directory. The default is `/data/${USER}/.singularity` if available; otherwise, it defaults to `${WORKDIR}/snakemake/.singularity`.