Repository navigation
Slurm and SSH execution for scans (Scans part of #420) - #456
Open
chrisbrahms wants to merge 23 commits into
Open
chrisbrahms wants to merge 23 commits into
chrisbrahms wants to merge 23 commits into
Conversation
…threading support
…ints limit [Scans part only] Cherry-picked from ff4cb1c without the RK45 step_on changes (src/RK45.jl, src/Luna.jl, test/test_rk45.jl).
… instances finish
…g and prevent infinite respawning
- clear_queue_state also removes the .nowork marker so queue tests leave no litter - instances job-script example in docs matches the generated script (NOWORKFILE check and final settle pass) - test SSHExec `files` keyword alongside `remotehostname` Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019CR9LuhpTBTeuzLE29AKXp
Closed
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This is the Scans/Slurm part of #420, split out so that it can be reviewed and merged on its own. The
RK45step_onchanges of #420 are not included (commit 8db4b98 is a cherry-pick of ff4cb1c with theRK45.jl,Luna.jlandtest_rk45.jlparts removed). 21 commits, 7 files, +1409/−44.What it adds
Scans.SlurmExecgains the keywordsmemory,project,nthreads,procs,workdir,arraymode,time,partition,cpusandinstances:memory="48G",time="12:00:00",partition=are passed tosbatch; the Julia heap-size hint is derived from the memory (80 %, divided among worker processes).projectselects the Julia project for the remote script (active project by default, explicit path, or none).workdircontrols where the generated script, queue file and job output go (default<scriptdir>/<scanname>_slurm/, via--chdir).arraymode=:queue(default) lets the array tasks share the scan dynamically through the queue file;arraymode=:batchgives one task per scan point.procsaddsDistributedworker processes per array task;cpussets--cpus-per-taskand FFTW threads;nthreadssets Julia threads.instances=nruns one fresh Julia process per scan point in a respawn loop, which bounds memory growth over long scans; the loop stops on a.doneor.noworkmarker.Scans.QueueExecgainsmaxpoints(points per process before exit) and writes completion markers (.done,.nowork) so that respawn loops terminate and a completed scan is not re-run; stranded queue entries are handled.Scans.SSHExecgainsremotehostname=for hosts whose login name differs fromgethostname(), and can wrap aSlurmExec.runscanloads the root package of the scan closure on distributed workers, so scans defined inside a user package work withprocs > 0.Luna.set_fftw_threads/Utils.FFTWthreads: an explicitfftw_threadssetting is honoured even with a single Julia thread, in which case libfftw3 uses its own pthreads pool. This is the documented way to get threaded FFTs on Julia ≥ 1.12, where the Julia-task FFTW callback can segfault.Docs: a new "Execution on Slurm" chapter in
docs/src/scans.md(basic usage, memory management, thread pinning, many simulations per array task with a worked example,instances, project environment, working directory, array mode, a full example, combining withSSHExec).Tests:
test/test_scans.jl(+486 lines:SlurmExecconstruction, heap-size-hint parsing, workdir resolution, script generation for both array modes and forinstances, queue markers,maxpoints, stranded entries) and a manual smoke testtest/manual/slurm_ssh_test.jlfor a real cluster.Status
masterat 08a53b3 (git merge-tree).test/test_scans.jlrun on this branch in a scratch environment (Julia 1.13, macOS arm64): all 23 testsets pass, about 350 assertions, 0 failures. The two multi-process queue testsets take about 10 minutes each on this machine (worker start-up).SlurmExecandSSHExecwere not exercised against a real cluster here;test/manual/slurm_ssh_test.jlexists for that.RK45changes once this is merged.🤖 Generated with Claude Code
https://claude.ai/code/session_018SqaCtmY34nxoDQc6wy8xX