Gate image-proprio launches and restore failed image training

docs/operations/joint-crash-recovery.md

@@ -0,0 +1,11 @@
+# Strict operational crash recovery for the constant-LR image run
+
+The original image-only discrete attempt412056 was a300k full allocation, not a scheduled allocation boundary. Its failed workers cannot be admitted through a fabricated scheduled completion. The explicit crash path binds its native terminal record and admitted request, immutable failure logs, and the highest healthy full-state checkpoint inspection. It rejects another live writer and requires a fresh immutable output/log namespace and same W&B identity with resume=must.
+
+The supported scientific identity is exactly `mot-shared-libero-image-discrete-lr1e-4-constant-s42-300k-20261021-v1`, scientific source `535d4fc26c4c85cf57bbd2a776cc79cfec085c4e`, four ranks/global32, constant AdamW1e-4 without warmup or scheduler. The budget remains300000 total updates, checkpoint and validation every20000. Scientific files are unchanged. The unlaunched image/proprio pair retains its separate first-launch admission; future crash recovery for that pair needs explicit generalization and actual matching proof.
+
+`crash_inspect.py` runs only on allocated compute, strictly reloads full wrapper/context/FP32 weights/AdamW/sampler/all-rank RNG and reconciles the highest checkpoint across hash-bound retained attempts. `crash_smoke.py` restores the inspected production checkpoint on four actual CUDA ranks into a distinct diagnostic namespace, executes two optimizer updates and100 scheduled validation anchors, reloads the diagnostic full state and performs a finite20-step hidden-label-NaN sample. A separate finalizer requires hostexit0 and nativeCOMPLETED0:0. Diagnostic artifacts never initialize production.
+
+`CrashRestoreProof` observes actual model/optimizer/sampler/RNG restoration and the first two updates on every rank. It rejects replayed boundary validation/saves and off-cadence saves. Production resume preserves all historical checkpoints. `crash_history.py` aggregates only each attempt's owned checkpoint/validation interval, preserving failed unsaved CSV history while rejecting duplicates. `verify_full.py` requires all15 immutable milestones, finite scheduled validation, final full-state reload/sample and actual restoration evidence. No extra production save is introduced.
+
+The101 new CPU tests and49 existing temporal-admission regressions verify the local implementation only. They do not certify production checkpoint health, destination storage, CUDA/DDP behavior or actual Lambda restoration. Before recovery, require current persistent storage read/write/fsync and space proof, parent terminal/resource-release evidence, immutable service deployment/readback, actual allocated inspection and the exact four-rank diagnostic. Do not submit while Lambda storage or node health is unverified. Never revive intentionally retired identities or change the scientific contract.

docs/operations/joint-service-import-20261021/README.md

@@ -0,0 +1,7 @@
+# Verified operational service import
+
+This import records the exact 23-file deployed operational service named `9a2982a8a611b9d377f7719dbd710161b39e4322` in the Lambda task `joint-modality-ablations-20261020/services/`. Its `FILES.sha256` bytes have SHA256 `1714c1b459dcef0dbe372651f0087e59391231f39f729896b7d75df5ce08d4ae`; every file was verified against that manifest before import.
+
+The name is a deployment identifier. Its Git ancestry was not established, and this commit does not assert that it descends from that identifier as a Git commit. The actual Git parent is the published modality-ablation branch at `255fdaf5137f7043b35df7ef48a3c1b8adc70815`. This creates a reviewable source home for verified operational bytes; it does not alter existing deployed releases or retrofit their provenance.
+
+Only orchestration/verification scripts and their declared dependency closure are imported. No `egomimic` model, data or configuration file changes. Existing scientific checkpoints and new image/proprio requests retain the separately verified frozen source `535d4fc26c4c85cf57bbd2a776cc79cfec085c4e` and its own archive/manifest bindings.

docs/operations/joint-service-import-20261021/deployed-9a-FILES.sha256

@@ -0,0 +1,23 @@
+af44a2641d79754615746fe9a6c708b04b81ea807318cf9700a0ee2465bc7796  scripts/clusters/pace/__init__.py
+bae908608d15f55ff8c0eab6f145baa8d6317112245d44b82b170831739f8b6f  scripts/clusters/pace/gpu_admission.py
+57861cbe471dfb352cc3a8818ec47dae76b062bfa5fe4eeb549d629b2e83e962  scripts/clusters/pace/gpu_ownership.py
+13035d263f431a77e9d90bdf1b1e596787b19d913cbde767b1a2ca0c627bf23e  scripts/clusters/pace/pace_podman.py
+86ae32e41945db33a6d97a4a783efceb019bf817714508f4759db322fdbd75c0  scripts/ice/ice_checkpoint_mirror.py
+64ed05603c3785266cd2e5b268dff71da1daac717f70971ee05b3de0579b70a7  scripts/train/joint/allocation.py
+5ada8f1b544727719e2264c20edbb45e33d29f711455eb725bc7c9c94ee39b08  scripts/train/joint/allocation_boundary.py
+2243cf846feb316d299e243db0a6a7ac378e5fcbffb8cddb934aca7d281c9ff9  scripts/train/joint/candidate_gate.py
+df47a1cc2d73e59f68d2819aa01012e631dc6706f091ae2fea1655c51738214b  scripts/train/joint/capture_native.sbatch
+26b8c6c8f4db2753fb1f4996c603acda0cf8fb14797e7e17f37b1a8f2140ed47  scripts/train/joint/check_wandb.py
+1953c4ebc6c25cc1180ed550671e648cc913f1f1ccde266731ae473491b372d2  scripts/train/joint/continuation.py
+645dade981af90af9ca93851ce21a5f37e6dc54290b481c4b1db2046910c3794  scripts/train/joint/contract.py
+a4feb1eb7db08a40d039942c96a3ab21da7a7db160abd3c0770090193e658c48  scripts/train/joint/lambda_full.sbatch
+67523a129e6aa0b8af436ca97961f50ef3b1fb571c9bf23734bb85e5485e808e  scripts/train/joint/lambda_inside.sh
+4d5695db4e703a2ac1ea29418d34261ffe06c040fc367e5af960788715944d19  scripts/train/joint/lambda_runtime.py
+50d3f31641ebaf494542d63cf43380486ca9ed50bc315c846cee354a2e91ce32  scripts/train/joint/lambda_stage_source.sh
+884299fc0dc876147c32cda032d304b0abde261586148b7ebe18f01706cedac7  scripts/train/joint/lr_policy.py
+3ca92ad5392c536cd9f47c715c6c6b2d3e8ceea0e69ecaf9c04eb427d72dfcd0  scripts/train/joint/recovery_check.py
+caffa078e53eb022a265b0a79be973f37070568e787f64330a9bdcb7ecd061b9  scripts/train/joint/retention.py
+792515e221bbf387c33ba136bdec08b48c7ad563c5fe783e97d869e9ee5d1235  scripts/train/joint/run.py
+88a8e20ef93a958ede13203e1867d64734b05d2aea8cb08ffdf0176f562ce68b  scripts/train/joint/service_bootstrap.py
+e43cfe8483949a6c1c0c7bebf8ff52e589ee1e442564992b57e8f545b3d691e1  scripts/train/joint/verify_factorial.py
+deb15f0a4a48b1f2843c2dfc2fcc87f431da30ca4e9bd61ff75dd953ccdbbade  scripts/train/joint/verify_full.py

docs/operations/joint-service-import-20261021/provenance.json

@@ -0,0 +1,12 @@
+{
+  "deployment_identifier_kind": "Recorded deployed service directory name; Git commit ancestry unverified",
+  "git_parent": "255fdaf5137f7043b35df7ef48a3c1b8adc70815",
+  "import_description": "Exact 23-file exported closure: 22 files match the earlier 8e service and contract.py differs; all manifest hashes were verified.",
+  "operational_manifest_sha256": "1714c1b459dcef0dbe372651f0087e59391231f39f729896b7d75df5ce08d4ae",
+  "operational_revision": "9a2982a8a611b9d377f7719dbd710161b39e4322",
+  "schema": "joint-operational-source-import-v1",
+  "scientific_archive_sha256": "f23b041bb2e8f250e406cbc24ef7be06b463221e8a004125d175f006661d8325",
+  "scientific_files_modified": false,
+  "scientific_revision": "535d4fc26c4c85cf57bbd2a776cc79cfec085c4e",
+  "status": "VERIFIED_OPERATIONAL_IMPORT_NOT_DEPLOYED"
+}

docs/operations/joint-temporal-pair-admission.md

@@ -0,0 +1,13 @@
+# Native image/proprio temporal-pair admission
+
+Fresh image/proprio runs with discrete and continuous temporal representations do not satisfy the existing modality-factorial or optimizer-parent comparison contracts. `verify_temporal_pair.py` adds an explicit contract for this pair without fabricating either parent proof.
+
+The v1 profile pins scientific source `535d4fc`, seed 42, four ranks with eight examples each, global batch 32, constant AdamW LR `1e-4`, and 300k updates. Checkpointing and validation occur every 20k updates, retaining all 15 immutable checkpoints. An exact override allowlist prevents unrequested scientific changes, including changes shared by both arms. Discrete/continuous pose dimensions 36/9 and token limits 3,216/12,864 differ; every other composed scientific value must match after checked output and identity substitutions.
+
+Both actual GPU smoke proofs are independently revalidated, including 50 optimizer updates, 100 scheduled validation anchors, every rank's LR and precision checks, strict reload, and sampling proofs. All eight ranks must share the same initial image weights. Training-draw hashes, whole-model initial weights and data-context hashes are retained separately; they are not claimed to be paired across temporal representations.
+
+Two immutable full-request prototypes omit `temporal_comparison`. The verifier's receipt binds their original files and canonical complete request hashes. Final requests add only that receipt binding. Admission revalidates both prototypes, both smoke proof sets, both configurations and the complete receipt before requiring exact request membership. The driver requires exactly one comparison contract.
+
+This profile supports fresh single-allocation requests with identical runtime and data-cache bindings. It rejects continuations and allocation-boundary requests. The temporal binding is included in continuation stable fields for future extensions, but recovery and different node/cache bindings still require separately reviewed admission support.
+
+The 49 focused tests use configurations composed from frozen source `535d4fc` and synthetic smoke evidence fixtures. They do not certify GPU execution, runtime readiness, deployment or production training. Before deployment, bind a published service export; complete allocated runtime, data and model preflight; obtain two new exact four-rank smoke passes at LR `1e-4`; report measured timings; and refresh ownership, queues, claims, storage, W&B identities and output-collision checks. The scientific archive remains independent and unchanged.

scripts/clusters/pace/gpu_admission.py

@@ -99,6 +99,8 @@ def worker(control):
     import torch.distributed as distributed
 
     rank = int(os.environ["LOCAL_RANK"])
+    world = len(control["expected_uuids"])
+    require(world in (2, 4), "Need two or four assigned GPUs")
     require(
         torch.__version__ == control["expected_torch"],
         "Scientific Torch version changed",
@@ -114,7 +116,7 @@ def worker(control):
         "Frozen disabled-TF32 policy was not applied",
     )
     require(
-        torch.cuda.device_count() == 4,
+        torch.cuda.device_count() == world,
         "Container does not expose exactly four CUDA GPUs",
     )
     require(torch.cuda.is_available(), "CUDA unavailable")
@@ -148,7 +150,7 @@ def worker(control):
     distributed.init_process_group("nccl", timeout=timedelta(seconds=120))
     value = torch.tensor([float(rank + 1)], device="cuda")
     distributed.all_reduce(value)
-    require(value.item() == 10.0, "Four-rank NCCL reduction failed")
+    require(value.item() == world*(world+1)/2, "Four-rank NCCL reduction failed")
     distributed.barrier()
     torch.cuda.synchronize()
     distributed.destroy_process_group()
@@ -157,35 +159,36 @@ def worker(control):
         {
             "status": "PASS",
             "rank": rank,
-            "world_size": 4,
+            "world_size": world,
             "uuid": actual_uuid,
             "name": properties.name,
             "compute_capability": [properties.major, properties.minor],
             "torch": torch.__version__,
             "torch_file": torch.__file__,
             "bf16": True,
-            "nccl_all_reduce": 10.0,
+            "nccl_all_reduce": world*(world+1)/2,
         },
     )
 
 
 def controller(control):
     base = validate_base(control)
-    require(len(control["expected_uuids"]) == 4, "Four assigned UUIDs required")
+    world = len(control["expected_uuids"])
+    require(world in (2, 4), "Need two or four assigned UUIDs")
     with socket.socket() as probe:
         probe.bind(("127.0.0.1", 0))
         port = probe.getsockname()[1]
     children = []
     started = time.monotonic()
     try:
-        for rank in range(4):
+        for rank in range(world):
             environment = dict(os.environ)
             environment.update(
                 {
                     "RANK": str(rank),
                     "LOCAL_RANK": str(rank),
-                    "WORLD_SIZE": "4",
-                    "LOCAL_WORLD_SIZE": "4",
+                    "WORLD_SIZE": str(world),
+                    "LOCAL_WORLD_SIZE": str(world),
                     "MASTER_ADDR": "127.0.0.1",
                     "MASTER_PORT": str(port),
                     "TORCH_NCCL_ASYNC_ERROR_HANDLING": "1",
@@ -222,7 +225,7 @@ def controller(control):
         json.loads(
             (Path(control["receipt_directory"]) / f"gpu-rank-{rank}.json").read_text()
         )
-        for rank in range(4)
+        for rank in range(world)
     ]
     require(
         [receipt["uuid"].lower() for receipt in receipts]

scripts/clusters/pace/gpu_ownership.py

@@ -20,7 +20,8 @@ def require(condition, message):
         raise ValueError(message)
 
 
-def indices(value):
+def indices(value, world=4):
+    require(world in (2, 4), "Need two or four owned GPUs")
     result = []
     for token in value.split(","):
         token = token.strip()
@@ -35,7 +36,7 @@ def indices(value):
             )
             result.append(token)
     require(
-        len(result) == len(set(result)) == 4,
+        len(result) == len(set(result)) == world,
         "Need exactly four distinct Slurm assignments",
     )
     require(
@@ -51,6 +52,8 @@ def bdf(value):
 
 
 def verify(evidence, expected):
+    world = expected.get("world_size", 4)
+    require(world in (2, 4), "Need two or four owned GPUs")
     cluster_lines = re.findall(
         r"^\s*ClusterName\s*=\s*(\S+)\s*$", evidence["scheduler_config"], re.M
     )
@@ -83,12 +86,12 @@ def verify(evidence, expected):
         fields.get("Restarts") == "0" and env.get("SLURM_RESTART_COUNT", "0") == "0",
         "Fresh allocation required",
     )
-    allocations = indices(env.get("SLURM_JOB_GPUS", ""))
+    allocations = indices(env.get("SLURM_JOB_GPUS", ""), world)
     native = re.findall(
-        r"GRES=gpu:(?:h100|h200):4\(IDX:([0-9,-]+)\)", evidence["native"]
+        rf"GRES=gpu(?::h100|:h200)?:{world}\(IDX:([0-9,-]+)\)", evidence["native"]
     )
     require(len(native) == 1, "Need one native four-Hopper GRES allocation")
-    native_ids = indices(native[0])
+    native_ids = indices(native[0], world)
     require(all(x.isdigit() for x in native_ids), "Native IDX malformed")
     if allocations[0].isdigit():
         require(set(native_ids) == set(allocations), "Slurm env and native IDX differ")
@@ -134,8 +137,8 @@ def verify(evidence, expected):
         len(inventory)
         == len({r["uuid"] for r in inventory})
         == len({r["index"] for r in inventory})
-        == 4,
-        "Require four bijective filtered NVML GPUs",
+        == world,
+        "Require the exact bijective filtered NVML GPUs",
     )
     physical = {}
     for path, text in evidence["proc"].items():
@@ -151,7 +154,7 @@ def verify(evidence, expected):
             "Ambiguous /proc GPU identity",
         )
         physical[uuid] = dict(minor=minor, bdf=bus)
-    require(len(physical) >= 4, "Incomplete physical /proc inventory")
+    require(len(physical) >= world, "Incomplete physical /proc inventory")
     devices = evidence["devices"]
     require(
         len(devices) == len({r["minor"] for r in devices}) == len(physical),
@@ -166,7 +169,7 @@ def verify(evidence, expected):
         "Invalid GPU character device",
     )
     allowed = {r["minor"] for r in devices if r["read_open_permitted"]}
-    require(len(allowed) == 4, "Kernel must grant exactly four physical GPU minors")
+    require(len(allowed) == world, "Kernel must grant exactly four physical GPU minors")
     uuids = {r["uuid"] for r in inventory}
     require(
         {u for u, p in physical.items() if p["minor"] in allowed} == uuids,
@@ -180,7 +183,7 @@ def verify(evidence, expected):
     xml = ET.fromstring(evidence["xml"])
     xml_nodes = xml.findall("gpu")
     require(
-        len(xml_nodes) == 4 and {n.findtext("uuid") for n in xml_nodes} == uuids,
+        len(xml_nodes) == world and {n.findtext("uuid") for n in xml_nodes} == uuids,
         "XML/NVML UUID inventories differ",
     )
     for n in xml_nodes:

scripts/clusters/pace/pace_podman.py

@@ -23,7 +23,10 @@
 import xml.etree.ElementTree as ET
 from pathlib import Path, PurePosixPath
 
-import gpu_ownership
+try:
+    from . import gpu_ownership
+except ImportError:
+    import gpu_ownership
 
 SCHEMA = "pace-podman-boundary-v2"
 GIB = 1024**3
@@ -774,6 +777,8 @@ def main():
     parser = argparse.ArgumentParser(description=__doc__)
     parser.add_argument("--request", required=True)
     parser.add_argument("--request-sha256", required=True)
+    parser.add_argument("--check-request-only", action="store_true",
+                        help="Verify request/helper hashes and namespace before GPU submission; no runtime or GPU claim")
     arguments = parser.parse_args()
     request_path = verified_file(
         {"path": arguments.request, "sha256": arguments.request_sha256}
@@ -797,6 +802,10 @@ def main():
         "Boundary helper is outside the reviewed directory",
     )
     require(bool(NAME.fullmatch(request["namespace"])), "Invalid job namespace")
+    if arguments.check_request_only:
+        print(json.dumps(dict(status="BOUND_REQUEST_STATIC_PASS",namespace=request["namespace"],
+                              request_sha256=arguments.request_sha256,bound_helpers=len(bound_helpers))))
+        return 0
     job_id = os.environ.get("SLURM_JOB_ID", "")
     require(
         job_id.isdigit() and os.environ.get("SLURMD_NODENAME"),
@@ -984,7 +993,7 @@ def main():
             "attestation": {
                 "job_id": job_id,
                 "node": os.environ["SLURMD_NODENAME"],
-                "world_size": 4,
+                "world_size": len(ownership["gpus"]),
                 "ownership": ownership,
                 "cluster": ownership["cluster"],
                 "payload_workspace": str(local / "payload" / "workspace"),