Skip to content

solx 1.0.2: fix job jump from inside another allocation#42

Merged
Shu-Wan merged 1 commit into
mainfrom
fix/jump-nested-cpu-bind
Jul 17, 2026
Merged

solx 1.0.2: fix job jump from inside another allocation#42
Shu-Wan merged 1 commit into
mainfrom
fix/jump-nested-cpu-bind

Conversation

@Shu-Wan

@Shu-Wan Shu-Wan commented Jul 16, 2026

Copy link
Copy Markdown
Owner

Problem

Running solx job jump <OTHER_JOBID> from inside an existing allocation fails. The reported symptom:

srun: error: CPU binding outside of job step allocation, allocated CPUs are: 0x…F.
srun: error: Task launch for StepId=….13 failed on node scc006: Unable to satisfy cpu bind request

job jump execs srun --pty to open a shell on the target node. When that srun runs inside another allocation, it inherits the enclosing job's SLURM_* step-shaping environment, which is wrong for the target job. Four independent leaks (all reproduced on Sol's srun 25.11.6):

Leaked env var (from the enclosing job) Symptom on the target node
SLURM_CPU_BIND Unable to satisfy cpu bind request (a cpuset from a different job)
SLURM_MEM_PER_NODE / SLURM_MEM_PER_CPU Memory required by task is not availablehard step-creation failure (e.g. jumping from a --mem=64G GPU job)
SLURM_NTASKS launches N tasks (jumping from an -n 16 job starts 16; only task zero gets the pty)
SLURM_JOB_NUM_NODES warning: can't run 1 processes on 2 nodes

Fix

Pin the pty step so it derives its shape from the target job, not the enclosing one:

srun --jobid=N --overlap --nodes=1 --ntasks=1 --cpu-bind=none --mem-bind=none --mem=0 --pty $shell

srun's precedence is command line > input env var, so each flag deterministically overrides its leak — each override verified on real Sol. The target job's cgroup still confines the shell to its cpuset, and jumping from a login node is unaffected (nothing to leak from).

job jump is the only command that launches a job step, so it's the only one exposed — job start (new allocation via salloc), stop, list, time, and keep (direct filesystem touch) are not.

Residual, left as-is (no clean flag override): SLURM_CPUS_PER_TASK (-c has no zero/none sentinel) and GRES vars (--gres=none would strip the shell's GPU cgroup access).

Also in this PR

  • DRY: the solx job start attach hint is now derived from the same srun_pty_argv builder, so the printed command can't drift from what jump actually runs.
  • Test: new integration test asserts the argv is env-independent under leaked SLURM_* vars and that the nesting notice fires.
  • Release 1.0.2 (bug fix → patch): bumped CLI + skill version, refreshed Cargo.lock, moved CHANGELOG [Unreleased][1.0.2], bumped docs/coverage.md.

Verification

  • cargo test: 104 unit + 40 integration pass · cargo clippy -D warnings clean · cargo fmt --check clean · release build clean.
  • Behavioral: each of the four leaks reproduced on real Sol, and the full production argv neutralizes all four in one clean launch.

⚠️ Before tagging v1.0.2, the maintainer still needs to run the L1/L2 eval suite and walk the L3 manual checklist on real Sol (per DEVELOPMENT.md) — those can't be driven from here. CHANGELOG date is 2026-07-16; adjust if tagging later.

🤖 Generated with Claude Code

A `solx job jump` run from within another allocation execs a nested
`srun` step that inherits the enclosing job's SLURM_* step-shaping
environment, which breaks on the target node: leaked SLURM_CPU_BIND
fails cpu binding ("Unable to satisfy cpu bind request"), SLURM_MEM_PER_*
fails step creation ("Memory required by task is not available"),
SLURM_NTASKS launches N tasks/shells, and SLURM_JOB_NUM_NODES warns
about spreading the shell across nodes.

Pin the pty step with `--nodes=1 --ntasks=1 --cpu-bind=none
--mem-bind=none --mem=0` — srun's command line overrides the inherited
env vars (each override verified on Sol's srun 25.11.6), the target
job's cgroup still confines the shell to its cpuset, and jumping from a
login node is unaffected. `job jump` is the only command that launches a
job step, so it is the only one exposed.

Also: derive the `job start` attach hint from the same argv builder so
it can't drift, and add a nested-jump integration test asserting the
env-independent argv plus the nesting notice.

Release: bump CLI + skill to 1.0.2, refresh Cargo.lock, move CHANGELOG
[Unreleased] -> [1.0.2], bump coverage.md.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Copilot AI review requested due to automatic review settings July 16, 2026 23:44

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR fixes solx job jump when invoked from inside another SLURM allocation by ensuring the srun --pty step is shaped by explicit CLI flags rather than inherited SLURM_* environment from the enclosing job. It also aligns the job start “attach” hint with the actual jump argv builder, adds an integration test to prevent regressions, and bumps release metadata to 1.0.2.

Changes:

  • Pin srun --pty with --nodes=1 --ntasks=1 --cpu-bind=none --mem-bind=none --mem=0 to neutralize leaked step-shaping SLURM_* vars for nested jumps.
  • Reuse the same srun_pty_argv builder for the job start attach hint to prevent drift.
  • Add CLI integration coverage for env-independence + nesting notice; bump versions/docs/changelog for 1.0.2.

Reviewed changes

Copilot reviewed 8 out of 9 changed files in this pull request and generated no comments.

Show a summary per file
File Description
solx/tests/cli.rs Adds a shared expected srun string and an integration test covering nested-jump env leaks + nesting stderr notice.
solx/src/slurm.rs Updates srun_pty_argv to include explicit anti-leak flags and documents the rationale; updates unit test accordingly.
solx/src/jobs.rs Derives the printed attach hint from srun_pty_argv via shell_join so it matches jump behavior.
solx/Cargo.toml Bumps crate version to 1.0.2.
solx/Cargo.lock Updates lockfile’s solx package version to 1.0.2.
skills/sol-skill/SKILL.md Bumps skill version to 1.0.2.
docs/solx.md Documents the pinned srun flags and the nested-jump behavior/implications.
docs/coverage.md Updates documented version to v1.0.2.
CHANGELOG.md Adds 1.0.2 entry and updates compare/tag links.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@Shu-Wan
Shu-Wan merged commit 43949c0 into main Jul 17, 2026
3 checks passed
@Shu-Wan
Shu-Wan deleted the fix/jump-nested-cpu-bind branch July 17, 2026 00:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants