Skip to content

Add Elmo continuous objectives and verified interruption recovery - #320

Draft
ElmoPA wants to merge 10 commits into
codex/joint-modality-ablation-20261019-v1from
codex/elmo-large-continuous-20261026-v1
Draft

ElmoPA wants to merge 10 commits into
codex/joint-modality-ablation-20261019-v1from
codex/elmo-large-continuous-20261026-v1

Conversation

@ElmoPA

@ElmoPA ElmoPA commented Oct 26, 2026 •

Copy link
Copy Markdown
Collaborator

Adds opt-in continuous Elmo objectives for the larger-model comparison: time-aware strongest-anchor fused-SNR-style weighting, and randomized subtask-boundary emphasis with independent frame noise and plain MSE. Original sampling and loss remain the baseline.

Imports the retained multimodal source byte-for-byte from its 1,030-file manifest onto the existing joint-modality parent; the unavailable original Git object is documented. Operational admission binds container source and requires actual corpus calibration, matching optimization/validation smoke, conditional Adam state, and per-rank RNG evidence.

Production interruptions previously had no valid admission route through the scheduled-boundary dispatcher. The new maintained recovery modules require genuine native failure evidence, inspection of the highest retained production checkpoint, an actual four-rank S-to-S+2-to-S+6 restoration diagnostic, and a separate completed finalizer before a same-identity resume to the original total budget. They preserve committed checkpoint/validation history, reject nonfinite observations and unknown claims, and never initialize production from diagnostic checkpoints.

Validation:

  • 150 focused recovery CPU tests pass against the final integrated export; all 38 exported service/test/fixture files match the tested tree. Python 3.10 grammar checks pass.
  • The earlier combined admission, loss and sampling suite passed 142 tests plus 72 subtests.
  • Each larger-model variant completed its matching four-H100 smoke with 50 optimizer steps, 100 scheduled held-out anchors, full-state/RNG checks, finite 20-step samples and capacity proof. Smoke parents 412778, 412791 and 412793 completed with host exit 0 and native COMPLETED 0:0.

This remains a draft implementation PR. Recovery allocation wrappers, deployment/readback, actual production checkpoint inspection and actual GPU restoration are separate unfinished gates; CPU tests and fresh-start smokes do not certify crash recovery. Running experiments retain their original pinned source and service bindings.

Reproduce the 1,030-file deployed source snapshot on the real published 255fdaf parent and record its archive and manifest hashes. The original recorded 153d Git object and ancestry are unavailable; this import makes no ancestry claim and preserves all historical run identities.

Retain every parent path outside the runtime export. The 42 scientific path updates are byte-identical to the deployed snapshot. Validation: exact hashes/modes, unchanged outside-export files, 79 focused CPU checks, and whitespace/patch checks.
Preserve the ops5acc service content as the operational base. Derive PIXEL_SOURCE_ROOT from the immutable request inside the container so --export=NONE does not remove a required source binding.
@ElmoPA ElmoPA changed the title Import verified deployed Elmo multimodal source snapshot Add Elmo continuous fused-loss and boundary-sampling experiments Oct 26, 2026
@ElmoPA ElmoPA changed the title Add Elmo continuous fused-loss and boundary-sampling experiments Add Elmo continuous objectives and verified interruption recovery Oct 26, 2026
Bind the native controller, execution claims, phase observer and runtime guards to the published observer-proof service. Preserve scientific sources and use exact parent-bound allocation and terminal proof before recovery admission.

Validation: 71 portable CPU cases passed in the intended repository layout; committed artifact hashes and executable modes verified. Actual recovery allocation and GPU restoration remain unperformed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant