Repository navigation
Conversation
This was referenced Oct 22, 2026
Collaborator
Author
|
Warning This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
This stack of pull requests is managed by Graphite. Learn more about stacking. |
ElmoPA
changed the title
ElmoPA
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fresh image/proprio runs need a strict temporal-pair admission contract, and the failed image-only attempt needs explicit crash recovery because it has no scheduled allocation-boundary completion. This change preserves the original temporal gate and adds recovery bound to the failed request, native terminal evidence, the highest healthy production checkpoint and an actual four-rank restore diagnostic.
Recovery restores the same W&B identity, full model/context, FP32 AdamW state, sampler and every rank's RNG. It verifies the first two updates on all ranks, rejects another live writer, replayed boundary validation or checkpoint events, and changed science. Retained attempts contribute only their owned checkpoint/validation intervals, preserving all historical artifacts and the 300k total budget with 20k cadence. The crash path currently admits only the exact constant-1e-4 discrete image-only identity; image/proprio first launches retain their separate smoke gates.
The operational service is based on published
22858e8; scientific source remains frozen at535d4fc. No model, data or Hydra scientific configuration changes. Existing Graphite parentcodex/joint-modality-ablation-20261019-v1remains unchanged.Validation: 101 new focused CPU tests plus 49 temporal-admission regressions pass locally. The publication tests use synthetic path/hash bindings and CPU tensor fixtures; they do not certify the production checkpoint or CUDA/DDP restoration. Publication does not deploy a service or submit training. Storage/node health, allocated checkpoint inspection and the matching four-rank optimizer/100-anchor validation/full-state/hidden-label-NaN sample diagnostic remain required before recovery; the two image/proprio runs still require their exact new smokes before launch.