Skip to content

Repository files navigation

📉 slurmpast

Why your Slurm jobs failed, and what to ask for next time.

CI PyPI PyPI downloads per month

The slurmpast dashboard: finished jobs grouped into workloads, filtered to failures, then one job's report showing it used all its time at 0% CPU, meaning it hung rather than ran out of time, then per-node failure rates.

✨ Install

pip install slurmpast

🧰 Use

slurmpast              # dashboard, last 7 days
slurmpast --sizing     # what to request next time
slurmpast 51170455     # one job, every field Slurm recorded
slurmpast --demo       # no Slurm? try a synthetic cluster
slurmpast -u alice     # someone else's history
slurmpast --all-users  # every account on the cluster
sp                     # short alias

enter opens · q goes back · / searches · f filters · s sorts · ? lists every key

🎯 What to request next time

argonne35-pretrain   test · 101 runs
  --time            raise to 10:00:00   (from 08:05:00)
      longest of 90 completed runs took 07:58:23.
  --mem             already about right
  #SBATCH --time=10:00:00

That job had seven minutes to spare. With fewer than three usable runs, it says not enough evidence instead of guessing.

More screens Overview: finished jobs grouped into workloads, ranked by resource use, each with its failure count and last run. One workload: every run of it, with a repeat-failure warning and sizing advice. Patterns across runs: repeated failures at an unchanged time limit, and a memory request being hand-tuned. One finished job: time, CPU and memory against their limits, then every field Slurm recorded. Failure rate per node, with a ready-to-paste --exclude.

📌 Good to know

🚫 A node is blamed only when it is really worse. Tested one at a time, twenty equally reliable nodes produce a false culprit more than half the time.

🩹 sacct often misreports memory and limits. These are handled, and anything unreadable shows as n/a, never a fake 0.

🔁 Before, during, after: slurmate writes the job, slurmwatch watches it, and this reviews it.

📦 --json emits everything: 103 values per job, plus 40 for every step. All 85 fields, and every sacct trap.

The demo is recorded against slurmpast --demo. Regenerate it with pip install -e ".[assets]" && python tools/demo_gif.py.


Slurm 20.11.8 · Python 3.10-3.13 · Textual 0.86-8.2 · MIT

Releases

Packages

Contributors

Languages