[ElmoPA]

MoT experts can now opt into per-head Q/K RMS normalization before RoPE, non-affine RMSNorm on AdaLN paths, and a parameter-budget-matched SwiGLU feed-forward block. The flags apply to the image, separate robot/human proprioception, action and camera experts; disabled defaults preserve existing checkpoint keys and behavior.

Validation: 25 focused CPU tests, 544 exact legacy state/RNG/output/gradient tensor comparisons, Python 3.10 grammar checks, and an independent review. The large enabled configuration constructs 314,227,000 total/trainable parameters. CUDA/FlashAttention compilation and the four-rank optimization plus scheduled-validation smoke remain deployment gates.

[ElmoPA]

Warning

This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

• Bind QK full-v5 recovery ops to published service #326
• Add exact QK full-v5 recovery history admission #325
• Add verified ImageNet smoke and full training admission #323
• Add native ImageNet data and generation path for the joint MoT model #322
• Add optional QK normalization, RMSNorm and SwiGLU to MoT experts #321 👈 (View in Graphite)
• Add Elmo continuous objectives and verified interruption recovery #320
• Add paired modality ablations for joint image forecast quality #265 : 1 other dependent PR (#276 )
• Train joint image, proprio and action MoT with smaller vector experts #264
• Add modular pixel video training and resilient log mirroring #262
• Optimize MoT attention and enable Flash4 BF16 defaults #261
• Add continuous-time MoT world model and offline data pipeline #260
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.