feat: implement attempt-based resource scaling (RAM/walltime doubling on retry) #127

Summary

The pipeline currently sets --restart-times 2 in the aspen CLI, so Snakemake retries failed jobs up to twice. However, resources are read from the static cluster.json and do not scale with attempt number — each retry runs with identical RAM and walltime, so jobs that fail due to resource exhaustion will fail again.

Desired behavior

On the first retry (attempt 2), automatically double walltime (and optionally RAM). On the second retry (attempt 3), double again. This is directly supported by Snakemake via attempt-aware resource lambdas:

# in a rule:
resources:
    mem_mb = lambda wildcards, attempt: attempt * 48_000,
    runtime = lambda wildcards, attempt: attempt * 960   # 16 h × attempt

What needs to change

1. Add resources: blocks to all resource-intensive rules (trim, align, remove_BL, atac_tss, etc.) using attempt-based lambdas.
2. Update the SLURM cluster-submission command in aspen to read from {resources.mem_mb} and {resources.runtime} instead of cluster.json.
3. cluster.json can then be retired or kept only for defaults/fallbacks.

Why this matters

• Fixes walltime failures (issues Reduce RAM & increase walltime for trimming step #112, Increase walltime for below steps #114) more robustly — a job that barely misses the 16 h limit will succeed on its second attempt at 32 h rather than failing twice and requiring manual intervention.
• Reduces wasted cluster allocations: the first attempt stays lean; resources escalate only when actually needed.

Related

• Reduce RAM & increase walltime for trimming step #112 (trim RAM/walltime)
• Increase walltime for below steps #114 (align / atac_tss walltime)

_{⚡ Generated using AI ⚡}