[ElmoPA]

Image forecast quality cannot be attributed to joint proprio/action prediction
from the historical image-only comparison because learning rate, context and
forecast horizon differed. Add a matched discrete2×2 experiment: image only,
image+proprio,image+action and allthree, each fresh for100k optimizer updates
with identical initial image weights and paired RGB data,masks and noise.

Excluded streams have no expert parameters, attention tokens, loss or sampler
updates. Full-bank construction and noising preserve image RNG pairing. The
default three-stream model remains unchanged. This is a single-seed screen of
whole-stream effects, including conditioning and supervision; it does not
isolate loss-only effects or match compute across differing token/parameter counts.

The maintained joint service binds exact source,data,runtime,resources and smoke
evidence. Each arm must pass actual optimization, scheduled validation, strict
full-state reload and hidden-label-NaN sampling; a four-arm gate compares initial
image hashes and the first three actual RGB batch/noise/mask hashes on every
rank. Full runs retain every20k checkpoint and validation event, with five
immutable epoch+step milestones and last.ckpt symlink per100k run.

The requested ICE/Phoenix placement uses a common four-rank,batch8/rank recipe
and requires new matching GPU smokes. The reusable PACE container boundary
proves current allocation ownership from native job/cgroup,kernel permissions
and UUID/BDF/physical minor identity. It handles Slurm/NVML namespace differences,
restricts CDI to physical owned devices and requires actual four-rank BF16/NCCL
admission. Smoke-to-full evidence binds the positive allocation proof and actual
training GPU UUIDs. Lambda's accepted admission remains unchanged. Source and
training jobs already running retain their recorded pins.

Validation:134 focused checks pass across model/control, scheduled-validation,
strict-checkpoint, portable-container, GPU-ownership, allocation-binding and
runtime-archive fixtures, including explicitly selected public W&B/logging namespaces and certificate-only CA bundles. Ruff and diff checks pass. Exact allocated four-rank
smokes and paired-controls PASS remain required; local tests do not certify
training launch. Fresh-only service is not an automatic-requeue implementation;
ICE full admission also requires exact recovery and a separate CPU checkpoint
mirror under the authorized retention policy.

Protocol: docs/JOINT_MODALITY_ABLATION.md. PACE admission contract:
docs/PACE_JOINT_ADMISSION.md. No full100k jobs have been submitted yet.

Portable payload assembly now verifies every file against the pinned component manifests and checks symlink closure without extracting on login nodes. The runtime base plus explicit public-library supplement, source and complete data produced a locally verified 16.45 GB payload; destination GPU certification and four-rank optimizer/validation smokes remain pending. Original production jobs and scientific configs are unchanged.

[ElmoPA]

Warning

This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

• Add paired modality ablations for joint image forecast quality #265 : 2 dependent PRs (#276 , #320 ) 👈 (View in Graphite)
• Train joint image, proprio and action MoT with smaller vector experts #264
• Add modular pixel video training and resilient log mirroring #262
• Optimize MoT attention and enable Flash4 BF16 defaults #261
• Add continuous-time MoT world model and offline data pipeline #260
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.