[cclerget]

What changed

The multi-host launcher staged the WALDO binary at /tmp/waldo-launch/<sha256> on every secondary host and never checked who owned that path. Another local user could create it first, replace the binary, and have it executed with the launching user's identity.

The staging commands were also chained with ;. As a result, a SHA-256 mismatch never stopped the mv that publishes the binary, and the same was true for resume checkpoints.

This PR:

• Per-user staging directory. The binary is staged under /tmp/waldo-launch-<uid>/<sha256>, where <uid> is the remote user's. The resolved directory is printed back, validated, and recorded per host. Worker invocations and launcher scratch use it.
• Ownership check (waldo_private_dir, a shell function). The target directory and every ancestor must be owned by the remote user or root. User-owned directories must not be writable by others; group write is allowed only for a user private group, the umask 002 default on RHEL-family systems. Root-owned directories that others can write to, such as /tmp, must be sticky. Staging fails and names the offending path otherwise.
• Fail-closed uploads. Uploads go to a unique mktemp file, so concurrent launches no longer share one temporary name. The script runs under set -e and exits before the rename when the digest does not match.
• Resume checkpoints. Checkpoints staged on secondaries get the same private-directory check and fail-closed digest check. The "already staged" shortcut only trusts an existing file inside a verified private directory.
• Docs. docs/MULTI-NODE-TRAINING.md documents the new location and requirements.

Fixes #18

Verification

• New unit tests run the real staging scripts locally through an ssh shim:
  • staging into a private per-user directory with 0700 modes, and restaging
  • refusing a launch directory writable by other users
  • leaving no binary behind on a digest mismatch
  • failing closed on a checkpoint digest mismatch, and staging successfully with the right digest
• Checked by hand: a user-private-group 775 directory is accepted and a 777 directory is rejected.
• ./testing/unit.sh, ./testing/vet.sh, ./testing/docs.sh
• Not run: testing/e2e/model-torchtitan-multinode.sh (needs 2+ CUDA GPUs and TorchTitan).

Contracts

• Secondary staging moves from /tmp/waldo-launch/ to /tmp/waldo-launch-<uid>/. Staging now fails if a staging or checkpoint directory could be modified by another user.
• Secondary hosts need a POSIX shell with GNU coreutils/findutils (realpath -e, stat -c, find -perm /mode, mktemp).
• No format changes.

DCO

• Every commit includes a Signed-off-by trailer (git commit --signoff).

[gmkurtzer]

Thanks for addressing the staging race and fail-open digest checks. The security goal is correct, but this implementation needs to be rebased and redesigned around the current multi-host staging pathway before it can merge.

The branch is 137 commits behind main and currently conflicts in all three changed files. More importantly, current main no longer stages launch state under /tmp/waldo-launch-*. It uses a session-scoped directory beneath the configured cache scratch root:

remoteRoot := filepath.Join(cache.Scratch(), "multinode", digest, cluster.RendezvousID)

Please preserve that contract and apply the private-directory validation and fail-closed upload within session.remoteRoot. The updated implementation also needs to retain the current cache arguments, delayed worker startup, worker PID/termination handling, and cleanupStaging behavior. The documentation should continue to describe configured scratch storage rather than restoring /tmp/waldo-launch-<uid>.

The new staging tests also fail on macOS because their local SSH shim executes the remote script using the host's BSD tools:

realpath: illegal option -- e
usage: realpath [-q] [path ...]

Five hostfile tests fail at realpath -e; stat -c and GNU find -perm have the same portability issue. It is fine for actual secondary hosts to require Linux/GNU tooling, but the test boundary must not assume those tools are installed on the local test runner. Please provide controlled command shims, run this portion through a Linux-specific harness, or otherwise keep the macOS unit lane passing while retaining meaningful coverage.

Finally, the current private-group exception does not establish the stated security property. Comparing id -gn with id -un does not prove that no other account belongs to the group, and a named ACL may also expose write access through the group permission mask. Either reject all group/other write access for user-owned staging ancestors, or perform a check that actually accounts for group membership and ACL-based access.

What I would expect after the rebase is roughly:

1. Validate the existing configured session.remoteRoot and its ancestors.
2. Upload the binary and checkpoints to unique temporary files within that validated root.
3. Verify each digest before an atomic rename.
4. Preserve the current session cleanup and worker/cache contracts.
5. Exercise successful staging, unsafe-directory rejection, digest mismatch, and cleanup on both Linux and macOS CI.

Local validation performed here:

• ./testing/vet.sh: passed
• ./testing/docs.sh: passed
• ./testing/unit.sh: the five new staging tests failed on macOS due to realpath -e
• The existing local host-memory inspection failures remain unrelated baseline failures

Please rebase and update the implementation along those lines, and I will re-review it.

[gmkurtzer]

Re-reviewed commit 76f9930 against current main. The requested changes are addressed: staging stays under the configured session remoteRoot, uploads use unique temporary files with fail-closed digest verification and atomic publication, group/other-writable user-owned paths are rejected, root-owned shared ancestors require the sticky bit, resolved paths and ACL exposure are checked, and cleanup/cache/worker contracts remain intact. The portable staging harness now passes on macOS and CI is green on both Ubuntu and macOS; DCO passes.\n\nLocal validation:\n- focused hostfile tests: passed\n- vet: passed\n- docs: passed\n- Python worker tests: passed\n- full Go suite: changed packages passed; only the known host-memory inspection failures reproduced in internal/host and internal/cli