[ElmoPA]

Train pixel video, measured proprio and recorded commands jointly from scratch with three MoT experts. Image and proprio share their observed/query masks and corruption strength at each timestamp; commands draw independent corruption strengths. Each stream has its own projections, attention/FFN parameters and AdaLN-Zero conditioning, with joint attention across streams. The image expert retains width 768 and 88,178,688 parameters. Proprio and action use width 256, each about 13.94M parameters, with Q/K/V projections into the common 768-wide attention space; all input/output and conditioning parameters are included. Total/trainable parameters are 116,087,360 (discrete) and 116,062,736 (continuous), and each vector expert is below 20M.

• Add separate graph nodes for noising, denoising, three plain clean-target MSE losses and sampling. All experts train together; no frozen-image stage or checkpoint initialization.
• Add matched 5e-6 peak LR with 3,000 optimizer-update linear warmup, then a constant rate, global-batch-32 recipes: discrete uses 2 GPUs × 16 examples, 16 stride-four anchors and four-state/four-command chunks; continuous uses 4 GPUs × 8 examples, sampling 2–64 timestamps across full episodes.
• Preserve the gripper-only 450/50 split, two 160×160 camera views, 16×16 RGB patches, spatial/temporal RoPE, camera conditioning, FA4 and BF16 mixed precision with FP32 weights/AdamW/MSE. Production runs target 300k updates and retain every 20k checkpoint, with validation at the same cadence.
• Document reproduction commands and pinned attention-runtime versions in docs/JOINT_PIXEL_STATE_ACTION.md. Add a worst-frame capacity check and a real optimization/100-anchor scheduled-validation smoke with strict checkpoint reload, distributed-state checks and finite generated samples.

Warmup lives beside the optimizer in model configuration, outside graph nodes. Update one uses 5e-6 / 3000; update 3,000 reaches 5e-6. Actual LR is logged, and both optimizer and scheduler state are verified on checkpoint reload. Add a standard-library cache-admission tool that binds a complete 1,502-artifact producer receipt and native successful completion, validates the manifest and every file size before container start, and preserves DataModule content/provenance checks.

Validation: 292 model/data/attention/scheduler regression checks and nine cache-admission checks passed; one CUDA-dependent test was skipped in the current regression suite. A tiny shared Lightning CPU training/validation/checkpoint integration verified the effective warmup LR and stored scheduler state. Lint passed. Real CUDA optimization plus scheduled validation has not yet passed; production remains gated on the exact two-/four-rank smoke, strict checkpoint reload and sampling, zero exits, memory fit and a measured training-time report.

The discrete/continuous pair intentionally differs in GPU count, vector chunk widths and frame exposure, so wall-time and full tensor-initialization comparisons require those differences to be reported. This PR is stacked on codex/pixel-video-training (PR #262).

[ElmoPA]

Warning

This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

• Add paired modality ablations for joint image forecast quality #265 : 2 dependent PRs (#276 , #320 )
• Train joint image, proprio and action MoT with smaller vector experts #264 👈 (View in Graphite)
• Add modular pixel video training and resilient log mirroring #262
• Optimize MoT attention and enable Flash4 BF16 defaults #261
• Add continuous-time MoT world model and offline data pipeline #260
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.