onsuccess/onerror telemetry hook can falsely mark a fully successful run as failed #144

Summary

A fully successful ASPEN run (213/213 Snakemake jobs completed) was reported as failed via pipeline.failed / pipeline.status.json (reason: snakemake_failed), even though every pipeline rule succeeded.

Root cause

• workflow/Snakefile's onsuccess/onerror hooks run an on_complete shell block that calls jobby --tsv snakemake.log and spooker, falling back to export PATH="$PATH:{ccbr_tools_path}" only if ! command -v jobby.
• SLURM jobs submitted via the generated submit_script.sbatch inherit the submitting shell's full environment (no module purge, no --export=NONE). If a different/older jobby (predating the --tsv flag) is already resolvable on PATH at submission time, the command -v jobby check succeeds, the pinned ccbr_tools_path fallback never triggers, and the stale jobby runs and crashes on --tsv.
• Even when the fallback does trigger, it only appends to PATH, not PYTHONPATH. ccbr_tools v0.7's jobby isn't installed to site-packages — it requires PYTHONPATH to include the ccbr_tools package root — so a PATH-only fallback still fails (ModuleNotFoundError) in that case.
• Because this block runs inside a Snakemake onsuccess/onerror hook, any nonzero exit inside it is fatal to the whole snakemake process, even though every pipeline rule had already finished. submit_script.sbatch then records pipeline.failed with reason=snakemake_failed, which is misleading.
• pipeline.{running,completed,failed,canceled} marker files are currently empty touch-files, so pipeline.failed gave no indication of what actually happened — the real error (jobby: error: unrecognized arguments: --tsv) was only visible by digging into snakemake.log.

Evidence

Observed in a real run under /data/Boufraqech_group/analysis/.temp/aspen_run_for_Ying_test3: snakemake.log shows Finished job 0. (all 213/213 jobs done); pipeline.status.json shows tasks_done: 213, tasks_total: 213, reason: "snakemake_failed", exit_code: 1; traceback points to jobby: error: unrecognized arguments: --tsv inside Snakefile's __onsuccess.

Fix (tracked on issue_118 / PR #119)

1. Make the onsuccess/onerror telemetry block environment-independent: explicitly module load ccbrpipeliner and prioritize the pinned ccbr_tools_path on both PATH and PYTHONPATH rather than trusting whatever jobby is already on PATH.
2. Make the telemetry block non-fatal (|| true/warn-and-continue per command) so a jobby/spooker failure never masks actual pipeline success/failure.
3. Add module purge to the generated submit_script.sbatch so SLURM jobs don't inherit a stale module/PATH environment from the submitting shell.
4. Write a human-readable summary into pipeline.<state> itself (not just the JSON sidecar) so failures are diagnosable at a glance.

_{⚡ Generated using AI ⚡}