Skip to content

Slurm and SSH execution for scans (Scans part of #420) - #456

Open
chrisbrahms wants to merge 23 commits into
LupoLab:masterfrom
chrisbrahms:slurm-only
Open

chrisbrahms wants to merge 23 commits into
LupoLab:masterfrom
chrisbrahms:slurm-only

Conversation

@chrisbrahms

Copy link
Copy Markdown
Collaborator

This pull request was generated by Claude (Claude Fable 5.1, running in Claude Code) at the request of @chrisbrahms. The commits on the branch are by @jtravs and @chrisbrahms; Claude wrote this description from the diff, checked that the branch merges cleanly onto master, and ran test/test_scans.jl.

This is the Scans/Slurm part of #420, split out so that it can be reviewed and merged on its own. The RK45 step_on changes of #420 are not included (commit 8db4b98 is a cherry-pick of ff4cb1c with the RK45.jl, Luna.jl and test_rk45.jl parts removed). 21 commits, 7 files, +1409/−44.

What it adds

Scans.SlurmExec gains the keywords memory, project, nthreads, procs, workdir, arraymode, time, partition, cpus and instances:

  • memory="48G", time="12:00:00", partition= are passed to sbatch; the Julia heap-size hint is derived from the memory (80 %, divided among worker processes).
  • project selects the Julia project for the remote script (active project by default, explicit path, or none).
  • workdir controls where the generated script, queue file and job output go (default <scriptdir>/<scanname>_slurm/, via --chdir).
  • arraymode=:queue (default) lets the array tasks share the scan dynamically through the queue file; arraymode=:batch gives one task per scan point.
  • procs adds Distributed worker processes per array task; cpus sets --cpus-per-task and FFTW threads; nthreads sets Julia threads.
  • instances=n runs one fresh Julia process per scan point in a respawn loop, which bounds memory growth over long scans; the loop stops on a .done or .nowork marker.

Scans.QueueExec gains maxpoints (points per process before exit) and writes completion markers (.done, .nowork) so that respawn loops terminate and a completed scan is not re-run; stranded queue entries are handled.

Scans.SSHExec gains remotehostname= for hosts whose login name differs from gethostname(), and can wrap a SlurmExec.

runscan loads the root package of the scan closure on distributed workers, so scans defined inside a user package work with procs > 0.

Luna.set_fftw_threads / Utils.FFTWthreads: an explicit fftw_threads setting is honoured even with a single Julia thread, in which case libfftw3 uses its own pthreads pool. This is the documented way to get threaded FFTs on Julia ≥ 1.12, where the Julia-task FFTW callback can segfault.

Docs: a new "Execution on Slurm" chapter in docs/src/scans.md (basic usage, memory management, thread pinning, many simulations per array task with a worked example, instances, project environment, working directory, array mode, a full example, combining with SSHExec).

Tests: test/test_scans.jl (+486 lines: SlurmExec construction, heap-size-hint parsing, workdir resolution, script generation for both array modes and for instances, queue markers, maxpoints, stranded entries) and a manual smoke test test/manual/slurm_ssh_test.jl for a real cluster.

Status

  • Merges cleanly onto master at 08a53b3 (git merge-tree).
  • test/test_scans.jl run on this branch in a scratch environment (Julia 1.13, macOS arm64): all 23 testsets pass, about 350 assertions, 0 failures. The two multi-process queue testsets take about 10 minutes each on this machine (worker start-up). SlurmExec and SSHExec were not exercised against a real cluster here; test/manual/slurm_ssh_test.jl exists for that.
  • Relation to Slurm #420: supersedes the Scans part of it; Slurm #420 can be reduced to the RK45 changes once this is merged.

🤖 Generated with Claude Code

https://claude.ai/code/session_018SqaCtmY34nxoDQc6wy8xX

jtravs and others added 21 commits October 21, 2026 08:56
…ints limit [Scans part only]

Cherry-picked from ff4cb1c without the RK45 step_on changes
(src/RK45.jl, src/Luna.jl, test/test_rk45.jl).
- clear_queue_state also removes the .nowork marker so queue tests leave no litter
- instances job-script example in docs matches the generated script
  (NOWORKFILE check and final settle pass)
- test SSHExec `files` keyword alongside `remotehostname`

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019CR9LuhpTBTeuzLE29AKXp
@chrisbrahms chrisbrahms mentioned this pull request Nov 5, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants