ElmoPA · Oct 27, 2026
diff --git a/docs/qk-full-v5-recovery.md b/docs/qk-full-v5-recovery.md
@@ -0,0 +1,114 @@
+# QK full-v5 future recovery: historical census amendment
+
+This local, unpublished amendment preserves sealed candidate-v2 and frozen
+scientific source A (`82f65498e25f41b87c954717722dcf7360e5bde8`). It extends
+the maintained recovery service/ops for original production parent **412831**
+and observer **412832**, exact full-v5 request `8760409c…`, source manifest
+`b5e8e898…`, and **314,227,000** parameters. It creates no request, native
+certificate, current RUNNING admission, failure, checkpoint or restoration proof.
+QK publication/service manifest/count pins remain `None` and execution fails
+closed until actual publication and explicit binding.
+
+## Four historical claims have typed dispositions
+
+| Parent / observer | Retained prelauncher evidence | Parent terminal |
+| --- | --- | --- |
+| 412823 / 412824 | Exact Python hashlib guard failure before launcher | FAILED 1:0 |
+| 412825 / 412826 | Exact completed-smoke reader timeout before launcher | **UNKNOWN** |
+| 412827 / 412828 | Authenticated parser guard failure before launcher | FAILED 1:0 |
+| 412829 / 412830 | Never-released held cancellation; no GPU allocation | CANCELLED 0:0 |
+
+The old observer outcomes are preserved independently. In particular, the
+failed 412826 observer is not substituted for its parent's unknown terminal.
+The v4 observer 412830's failed oversized-PREPARED authentication is retained;
+the reviewed held/cancel and raw-query chain authenticates that special history.
+Scheduler StartTime=EndTime for the held cancellation remains bookkeeping.
+
+The type `REVIEWED_PRELAUNCH_NO_TRAINING_ENTRY` carries separate native-state
+and terminal-proof fields, denies phase completion/checkpoint recovery authority,
+and never emits a fabricated native record. It only discharges these exact
+four old claims from a fresh quiescence census. Current 412831 scientific
+admission, supported failure, genuine terminal/observer completion, positive
+highest production checkpoint, all-rank restoration and finalizer remain required.
+
+## Maintained reader and fresh census
+
+`qk_prelaunch_history_readers/` ports the four already reviewed historical
+reader bodies into a fixed service-owned package. Six exact original SHA/role
+pairs select explicit maintained parsers; old Python bytes are hash-checked
+as evidence data and are never compiled, dynamically imported or executed.
+Four native parsers retain exact original bytes; pure early verification keeps
+its original AST. `historical-reader-provenance.json` records that comparison.
+
+The fixed retained closure is full-v5 `proofs/prelaunch-held-v1/EVIDENCE.json`,
+SHA `15ee3a5b…`: 279 data files plus the manifest, bounded individually at
+2 MiB and collectively at 12 MiB. These are explicit historical-evidence bounds;
+ordinary code-manifest and metadata limits are not globally relaxed. The known
+1,363,584-byte v4 PREPARED and 2,044,783-byte transport leaf are retained.
+
+Each admission reopens the exact current full-v5 request/FILES and historical
+chain, then all 69 pinned old live files plus nine v1 acceptance files not covered
+by the old hash capture. Those additional acceptance fields are semantically
+checked against exact argv, accepted IDs, outcomes, dependency and release.
+No historical hashes are invented. Exact old claim and request-marker inventories,
+unused old maintained claim/status/attempt namespaces, all eight old IDs/aliases
+in complete fresh owner queue/steps, and the original v5 accepted claim/marker
+are checked. Unknown or partial entries fail. One transient verified census is
+reused by four typed checks inside that admission; serialized JSON, stale or
+mutated snapshots cannot become admission. Returned receipts contain compact
+hashes, not recursively embedded PREPARED bodies or generated shell payloads.
+
+The census is direct stdlib filesystem/native work on the existing allocated
+controller/producer. It generates no SSH command or remote literal and adds
+no GPU/CPU allocation. Gateway callers explicitly defer old node-local CSV
+absence. An authenticated gpu21 host records the physical CSV parent device,
+inode and owner in the existing host-native admission record. Container gates
+reopen that same-allocation record and require the same directory identity;
+dispatch also verifies the existing container-native host hash. A masked or
+relocated `/ephemeral` view fails. Reviewed runtime/controller entrypoints set
+the host/container boundary after native authentication, not from request JSON.
+
+Only old v1..v4 CSV/status/attempt namespaces must remain absent. The current
+full-v5 output, CSV, status, checkpoints and scientific history are preserved
+and validated by the original production recovery gates. The fresh-launch
+zero-output census is deliberately not reused for current scientific data.
+
+## Consumer and publication contract
+
+Service `interruption_admission.quiescence` and ops `host_guard.quiescence`
+both require the fixed `qk_prelaunch_history` binding for this run and dispatch
+the exact old claims before ordinary terminal routes. Every other original or
+new recovery claim retains actual observer/native proof requirements.
+`completed_producer` is unchanged and cannot accept a prelaunch disposition.
+The pre-Torch service import allowlist and service export/ownership checks cover
+the fixed nested reader package. Both service and ops changes must be published
+and bound together; publishing sealed v2 alone does not close this adapter gate.
+
+Future quiescence scopes must include full-v1..v4 `/claims`, original full-v5
+`/claims`, all real new execution/scientific claim roots, exact scientific
+names/prefixes, and `qk_prelaunch_history.expected_binding()`. The latter pins
+the fixed full-v5 parent request, FILES and historical EVIDENCE paths/hashes.
+No template/request is materialized here. Root must verify the actual published
+service/ops closure and bind commit, manifest and count; local archive hashes
+are not publication pins. `publication-path-map.json` and before maps describe
+the complete delta and portable test/fixture destinations.
+
+## Validation and remaining gates
+
+There are 84 unique local cases: 50 retained QK observer/558-parameter/private
+state tests and 34 new history/census/mount/import regressions. The new stdlib
+suite runs on actual Python 3.10; the complete suite also passes in the intended
+repository layout under `tests/joint_qk_recovery`, from `/tmp`. Fixtures combine
+retained historical bytes with clearly synthetic fresh filesystem/native
+observations. The actual historical closure is replayed separately. Neither
+kind proves present native health, checkpoint content or GPU continuation.
+
+All original science and positive production checkpoint/FP32 model/conditional
+Adam/scheduler/sampler/all-rank RNG checks remain. Recovery still requires actual
+two-process S→S+2→S+6 restoration, scheduled validation/hidden-label samples,
+host/native/observer completion, separate finalization, same W&B `resume=must`,
+300,000 TOTAL updates, and all fifteen immutable 20k milestones. Diagnostic
+checkpoints never initialize production. Publication, deployment, future request,
+failure, checkpoint and actual restore are UNBOUND. No remote query, publication,
+allocation, deployment, submission, cancellation, canonical change or ImageNet
+transfer action was performed.
diff --git a/scripts/train/joint/interruption_admission.py b/scripts/train/joint/interruption_admission.py
@@ -30,10 +30,12 @@ def fingerprint(request):
 
 
 def compatible(parent, child):
+    from scripts.train.joint.qk_recovery_profile import supported_parameters
+    expected_parameters = supported_parameters(parent)
     need(parent["schema"] == child["schema"] == "joint-training-v2"
          and parent["phase"] == child["phase"] == "full"
          and parent["kind"] == child["kind"] == "native-subtask-world-model-v1"
-         and child["parameters"] == 314187064 and child["world_size"] == 4,
+         and child["parameters"] == expected_parameters and child["world_size"] == 4,
          "Exact production native Elmo314M identity required")
     need({k: v for k, v in parent.items() if k not in CHANGING_ATTEMPT_FIELDS}
          == {k: v for k, v in child.items() if k not in CHANGING_ATTEMPT_FIELDS},
@@ -114,6 +116,8 @@ def quiescence(spec, run_id, *, current_job=None, current_request_sha=None):
         if owned:
             need(job == current_job and state == "RUNNING", "Other pending/active scientific writer: " + job)
     from scripts.train.joint.phase_observer_proof import read_json, text, physical, check_owned_steps
+    from scripts.train.joint.qk_prelaunch_history import census as prelaunch_history_census
+    prelaunch = prelaunch_history_census(spec, run_id, current_job=current_job, require_node=True)
     seen, evidence = set(), []
     for item in spec["claim_roots"]:
         root = physical(item["path"])
@@ -131,13 +135,15 @@ def quiescence(spec, run_id, *, current_job=None, current_request_sha=None):
                 evidence.append(dict(path=str(claim), job_id=job, current=True))
                 continue
             from scripts.train.joint.phase_observer_proof import verify_claim_terminal
-            terminal_proof = verify_claim_terminal(claim, item["terminal_directory"], run_id)
+            terminal_proof = verify_claim_terminal(claim, item["terminal_directory"], run_id,
+                                                    prelaunch_census=prelaunch)
             need(terminal_proof["job_id"] == job, "Terminal observer belongs to another accepted claim")
             check_owned_steps(steps, job)
             evidence.append(dict(path=str(claim), **terminal_proof))
     if current_job is not None:
         need(current_job in seen, "Current allocation is not registered in the complete claim census")
-    return dict(queue=queue, owned_steps=steps, claims=evidence, checked_at=datetime.now(timezone.utc).isoformat())
+    return dict(queue=queue, owned_steps=steps, claims=evidence, prelaunch_history=prelaunch,
+                checked_at=datetime.now(timezone.utc).isoformat())
 
 
 def validate_failure(bundle, parent_binding):
@@ -260,7 +266,7 @@ def finalize(request_path):
 
     start = selected["global_step"]
     rows = read_phase_proofs(diagnostic_q["output"], start, parent["target_steps"], 4, diagnostic["job_id"])
-    flags = validate_phase_proofs(rows, selected, diagnostic.get("allocation_attestation"))
+    flags = validate_phase_proofs(rows, selected, diagnostic.get("allocation_attestation"), parent)
     for phase in rows:
         for row in phase:
             need(row["optimizer_update_lrs"] == [update_lr(parent["lr_policy"], i)

diff --git a/scripts/train/joint/native_recovery.py b/scripts/train/joint/native_recovery.py
@@ -190,7 +190,7 @@ def sample_both_embodiments(module, network, output):
     return rows
 
 
-def summarize_native_phases(phases, *, start_step=50, production=False):
+def summarize_native_phases(phases, *, start_step=50, production=False, require_both_embodiments=True):
     """Validate all-rank conditional evidence for the actual2+4 diagnostic."""
     need(type(start_step) is int and ((production is False and start_step == 50)
          or (production is True and start_step > 0 and start_step % 20000 == 0)),
@@ -251,9 +251,9 @@ def summarize_native_phases(phases, *, start_step=50, production=False):
     for a, b in zip(phases[0], phases[1]):
         need(a["conditional_optimizer_record"]["parameter_update_counts"] == b["initial_parameter_usage"],
              "Second process did not restore conditional optimizer counts")
-    need(all(sum(phase[0]["embodiment_optimizer_batches"][emb] for phase in phases) > 0
-             for emb in expected_streams), "Actual2+4 recovery did not exercise both embodiments")
+    both = all(sum(phase[0]["embodiment_optimizer_batches"][emb] for phase in phases) > 0 for emb in expected_streams)
+    need(not require_both_embodiments or both, "Actual2+4 recovery did not exercise both embodiments")
     return dict(native_multimodal=True, private_bank_optimizer_steps_verified=True,
-                actual_all_rank_rng_restore_observed=True, both_embodiments_optimized=True,
+                actual_all_rank_rng_restore_observed=True, both_embodiments_optimized=both,
                 both_embodiments_hidden_nan_20step_sample=True,
                 diagnostic_scope=("production-checkpoint-only" if production else "scheduled-smoke-checkpoint-only"))
diff --git a/scripts/train/joint/original_observer.py b/scripts/train/joint/original_observer.py
@@ -11,6 +11,8 @@
 
 from scripts.train.joint.contract import bound, need
 
+from scripts.train.joint.qk_recovery_profile import RUN_ID as QK_RUN_ID, PROFILE as QK_PROFILE
+
 ORIGINAL_OBSERVERS = {'mot-elmo-pickplace-continuous-large314m-boundary-robot-human-camera-subtask-s42-300k-20261026-v1': {'capture_script_sha256': 'be8ca57b2e5c09503ec312c9d84c91451d932d4d9a168c484f22439ccf521b57',
                                                                                                       'files_sha256': 'fbce43e2cf1ad9c090563bc7e5492c654f00db7e84890ef83d039e66cc9aea5f',
                                                                                                       'has_smoke_identity': True,
@@ -47,6 +49,8 @@
                                                                                                    'request_sha256': '936e9acd30fa517f6f38311c0bcfd2b88544da8d476bd41125c4008de2845073',
                                                                                                    'root': '/workspace/users/paphiwetsa3/tasks/elmo-large-continuous-20261026-v1/baseline-full-preparation-v1',
                                                                                                    'submitter_sha256': '94c077facd7836fb55708199dab988fb5902db5348ce449daa861a888d95901f'}}
+ORIGINAL_OBSERVERS[QK_RUN_ID] = QK_PROFILE
+
 
 PURGED_STDERR = "slurm_load_jobs error: Invalid job id specified"
 
@@ -55,6 +59,7 @@ def read_binding(binding, json_value=True):
     path = Path(binding["path"])
     need(path.is_absolute() and path.is_file() and not path.is_symlink()
          and path.resolve(strict=True) == path, "Observer proof must be a physical regular file")
+    need(path.stat().st_size <= 4 * 1024 * 1024, "Observer metadata exceeds4MiB pre-read bound")
     return bound(binding, json_value)
 
 
@@ -79,6 +84,9 @@ def validate_original(q, reader=read_binding):
     from scripts.train.joint.native_interruption import fields, native_identity, TERMINAL
 
     parent = reader(q["parent_request"])
+    if parent.get("run_id") == QK_RUN_ID:
+        from scripts.train.joint.qk_original_observer import validate
+        return validate(q, reader)
     profile = ORIGINAL_OBSERVERS.get(parent.get("run_id"))
     need(profile is not None, "No reviewed original observer for this identity")
     root = Path(profile["root"]); pid, oid = profile["parent_id"], profile["observer_id"]

diff --git a/scripts/train/joint/phase_observer_proof.py b/scripts/train/joint/phase_observer_proof.py
@@ -12,6 +12,7 @@
 
 from scripts.train.joint.native_interruption import fields, native_identity, NATIVE_KEYS, TERMINAL
 from scripts.train.joint.original_observer import ORIGINAL_OBSERVERS, strict_step_rows
+from scripts.train.joint.qk_recovery_profile import profile_run_id, supported_parameters, is_qk
 
 MODES = {
     'capture': ('native-elmo-interruption-capture-v1','2','1G','00:10:00',''),
@@ -157,19 +158,19 @@ def execution(ex):
     need(set(ex)=={'path','sha256','mode','ops_manifest_sha256'} and ex['mode'] in MODES,'Wrong exact phase execution binding')
     path=bound({'path':ex['path'],'sha256':ex['sha256']});spec=read_json(path);root=physical(spec['root']);mode=ex['mode']
     need(spec['schema']=='native-elmo-execution-spec-v1' and spec['mode']==mode and path.parent==root/'execution'
-         and spec['arm'] in {'plain','fused','boundary'},'Execution schema/root/arm differs')
+         and spec['arm'] in {'plain','fused','boundary','qk'},'Execution schema/root/arm differs')
     ops={'path':str(root/'FILES.sha256'),'sha256':ex['ops_manifest_sha256']}
     need(spec['ops_manifest']==ops,'Execution/observer operations manifest differs')
     names=manifest(root,ops)
     need({'execution_spec.py','allocated.sbatch','host_guard.py','phase_observer.py','observe-phase.sbatch'}<=names
          and all(Path(n).parts[0] not in {'execution','execution-claims','requests','scientific-claims','producer-claims','observer-specs','observer-claims','status','logs'} for n in names),
          'Missing ops role closure or circular dynamic manifest')
-    parent=read_json(bound(spec['parent_request']));run_id='mot-elmo-pickplace-continuous-large314m-'+spec['arm']+'-robot-human-camera-subtask-s42-300k-20261026-v1'
+    parent=read_json(bound(spec['parent_request']));run_id=profile_run_id(spec['arm'])
     profile=ORIGINAL_OBSERVERS.get(run_id)
     need(profile is not None and spec['parent_request']=={'path':profile['request_path'],'sha256':profile['request_sha256']}
          and str(spec['original_parent_job_id'])==profile['parent_id'] and parent['run_id']==run_id
          and parent['schema']=='joint-training-v2' and parent['kind']=='native-subtask-world-model-v1' and parent['phase']=='full'
-         and parent['target_steps']==300000 and parent['parameters']==314187064 and parent['world_size']==4,'Exact original scientific parent differs')
+         and parent['target_steps']==300000 and parent['parameters']==supported_parameters(parent) and parent['world_size']==4,'Exact original scientific parent differs')
     service(spec['operational_service'])
     template_path=bound(spec['request_template']);need(template_path.parent==root/'templates','Template escaped immutable namespace')
     template=read_json(template_path);schema,cpus,memory,limit,gres=MODES[mode]
@@ -383,9 +384,14 @@ def verify_observer(observer_binding,*,producer_request=None,producer_native=Non
     return ctx
 
 
-def verify_claim_terminal(claim,terminal_directory,run_id):
-    """Exact old-original route or complete new spec-bound phase observer."""
+def verify_claim_terminal(claim,terminal_directory,run_id,*,prelaunch_census=None):
+    """Typed reviewed prelauncher history, original, or new phase observer."""
     claim=physical(claim);terminal_directory=physical(terminal_directory)
+    from scripts.train.joint.qk_prelaunch_history import claim_disposition
+    historical=claim_disposition(prelaunch_census,claim,run_id)
+    if historical is not None:
+        need(terminal_directory==claim.parent.parent,'Historical terminal directory moved')
+        return historical
     prepared=read_json(claim/'PREPARED.json');job=text(claim/'PARENT_JOB_ID').strip()
     need(job.isdecimal() and prepared['request_sha256']==claim.name,'Unknown/incorrect accepted request claim')
     profile=ORIGINAL_OBSERVERS.get(run_id)
@@ -400,8 +406,13 @@ def verify_claim_terminal(claim,terminal_directory,run_id):
             parent_id=claim/'PARENT_JOB_ID',observer_id=claim/'CAPTURE_JOB_ID',pending_native=claim/'CAPTURE_PENDING_NATIVE.txt',
             running_admission=claim/'RUNNING_ADMISSION.json',terminal_native=root/('native-'+job+'.txt'),
             query_exit=root/('native-'+job+'.exit'),query_stderr=root/('native-'+job+'.stderr'))
-        if profile['has_smoke_identity']:paths['smoke_identity']=root/'smoke-identity.json'
-        proof=dict(schema='native-elmo-original-observer-v1',**{k:binding(v) for k,v in paths.items()})
+        if profile.get('protocol') == 'qk-full-v5-early-authenticated-v1':
+            from scripts.train.joint.qk_original_observer import proof_paths, SCHEMA
+            paths=proof_paths();schema=SCHEMA
+        else:
+            if profile['has_smoke_identity']:paths['smoke_identity']=root/'smoke-identity.json'
+            schema='native-elmo-original-observer-v1'
+        proof=dict(schema=schema,**{k:binding(v) for k,v in paths.items()})
         q=dict(parent_request={'path':profile['request_path'],'sha256':profile['request_sha256']},parent_job_id=job,
             parent_native=expected,parent_running_admission=running_binding,original_observer=proof)
         from scripts.train.joint.original_observer import validate_original