[ElmoPA]

This adds the pixel-video training setup used to compare a fixed discrete window with continuous sampling across an episode. Stage one predicts raw RGB from scratch using only the two LIBERO camera views, so generated frames can be inspected directly. It also retains the earlier minimal DINO experiments and provides separate, deferred stage-two configs for measured-state and control-command readout.

Training and execution

• Stage-one graph separates corruption, denoising, and pixel MSE. The image expert uses frame-wise AdaLN-Zero noise conditioning, camera embeddings, and spatial/temporal RoPE; no proprioception, actions, or learned phase embedding enter stage one.
• Both recipes use 160×160 RGB, 16×16 patches, width 768, eight blocks, six heads, AdamW at 5e-5, and 300,000 optimizer steps. Discrete selects 16 timestamps at stride four; continuous samples 2–64 timestamps across the full episode. Prepared data records a balanced 450/50 episode split.
• Stage-one validation and checkpoints run every 20,000 optimizer steps, producing 15 scheduled checkpoints for a fresh 300k run. Every milestone is retained; last.ckpt links to the latest save instead of duplicating its payload. Deferred stage-two readout retains its separate 1,000-step budget and 250-step cadence.
• Execution supports isolated standard batches and packed variable lengths, regional compilation, FlashAttention profiles, mixed BF16 computation with FP32 parameters/loss, explicit four-rank sampling, and deterministic validation/resume behavior. The selected full runs use eight examples per GPU and global batch 32.
• Stage two has configurable image providers, strict initialization from stage one, and an initially frozen image expert. Its diagnostic budget remains explicitly 1,000 steps; no stage-two run is launched by these changes.
• A reusable log-mirror helper keeps transient periodic copy failures from terminating training, records mirror health, and requires a successful bounded final flush.

Validation and provenance

• Training and execution suite (36 test modules): 540 passed, 5 GPU-gated tests skipped, zero failures. This includes optimizer/validation/checkpoint integration, packed and batched isolation, distributed RNG/resume behavior, graph composition, data preparation, and mirror failure/recovery.
• Restored legacy HPT checkpoint fixture: 2 parity tests passed, using hash-verified dependency wheels pinned in uv.lock in an isolated test overlay. The shared environment was unchanged.
• All 385/385 shipped YAML configs resolve. Offline contexts supply synthetic launch inputs only inside the audit; production manifests and checkpoint bindings remain mandatory. The 17 DINO/pixel/world-model component audits construct real meta models and optimizers without opening data or weights. Config and prepared-data regression tests: 206 passed, 1 skipped (including 25 config-context and 27 component/preflight tests).
• Pinned Ruff 0.8.6 lint and formatting on touched Python files, shell syntax checks, and git diff --check passed. Tiny integration fixtures explicitly resize the axial RoPE dimensions and patch grid along with their test-only model width.
• The cadence follow-up passes 67 focused config/profile tests. Resolved comparison confirms only both cadence fields and the linked-last setting change in stage one; deferred readout resolves identically to its previous configuration.
• Separately, the frozen production source strictly reloaded the continuous step-80,000 checkpoint on H100 and generated three held-out examples at [1,2,4,8,16,32]/20 seconds from both-camera t0 observations. A counterfactual check confirmed withheld future pixels do not affect generation. This checks the recorded running source, not a fresh launch of this publication commit.

This is a child of #261, preserving the original MoT and attention review ancestry. The final training tree was assembled from local snapshot 252275a903b9f444a297dda303fbb6a1ca9ddf93, while retaining the upstream legacy HPT fixture. Active runs and checkpoints keep their recorded source 3785f94392b9d900e21d9948fbc5bf94a6d975b8 and explicit run overrides; publication does not relabel or modify their frozen source. CPU tests do not certify CUDA performance or stage-two rollout quality.

Existing CI limitations

The initial GitHub run exposed missing offline audit contexts in this stack; the follow-up commits address those and formatting in this PR. Global static checks also report 23 lint diagnostics in nine files and formatting differences in other files inherited unchanged from #261; those unrelated files remain untouched. The PR Wiki Tracker fails because its destination repository and credential settings are empty. These repository-level failures are separate from the passing focused tests above; this PR does not claim green global CI.

[ElmoPA]

Warning

This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

• Add paired modality ablations for joint image forecast quality #265 : 2 dependent PRs (#276 , #320 )
• Train joint image, proprio and action MoT with smaller vector experts #264
• Add modular pixel video training and resilient log mirroring #262 👈 (View in Graphite)
• Optimize MoT attention and enable Flash4 BF16 defaults #261
• Add continuous-time MoT world model and offline data pipeline #260
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.