diff --git a/benchmarks/piper-probe-evidence-v01/README.md b/benchmarks/piper-probe-evidence-v01/README.md new file mode 100644 index 0000000..dc24c8f --- /dev/null +++ b/benchmarks/piper-probe-evidence-v01/README.md @@ -0,0 +1,56 @@ +# PiPER probe evidence v0.1 + +This local CPU simulation study compares two authored short-lift admission rules on the same 12 synthetic scenes: an evidence gate and a fixed timer gate. Each scene is run once per variant (24 scheduled attempts total). The object masses are 40 g and 60 g, friction settings are 0.80, 0.55 and 0.15, and two authored XY layouts are used. + +Both policies command a TCP probe target up to 10 mm above probe start at 20 mm/s, then request a hold at measured arm joints and the last issued jaw target. Actual object/TCP motion can continue under native dynamics and target rate limits. They do not continue to a full transport or placement. The evidence policy admits only after bilateral native pad force above 0.01 N, no table support, at least 6 mm object rise, and no more than 3 mm TCP-local offset drift continuously across at least 100 ms of sampled control boundaries. The timer policy admits at 600 ms after probe start without consulting that predicate. Unexpected contacts and bounded timeouts still stop the attempt. One-millisecond native records audit the policy claims; they are privileged simulator state, not physical sensors. + +The paired attempts are closed-loop runs. Early holds, alarms and supported retries can change subsequent paths, so differences are not an isolated classifier-causality estimate. All scheduled attempts and claim disagreements remain in the report. Each final planning gate retains declared assumptions; the shared 10 N per-finger value is an arbitrary synthetic admission ceiling, not a measured damage limit or force controller. Unknown quantities remain unknown. + +## Frozen run + +All 24/24 native attempts were valid. Evidence admission occurred in **6/12** attempts; reconstruction over the recorded 1 ms rows classified **zero false initial qualifications** and **zero conservative abstentions** among the supplied claims. Four of those six correctly admitted probes later lost the required probe height; two ended in a qualified hold. The timer admitted **12/12** attempts, with **8 false initial qualifications** and zero conservative abstentions. Ten timer attempts were invalidated in total: the eight false admissions plus two of the four initially correct admissions. Six invalidations followed support return and four followed loss of probe height. Two attempts ended in a timer hold. At the final observation, each variant had 2/12 `ready_for_simulated_transport` planning gates and 10/12 `test_lift_required`. A planning status is not transport execution or actuation permission. + +An independent readback audit reconstructed the native predicate from the saved arrays without importing receipt derivation or rerunning physics. It verified all 24 attempts and 480,000 native rows, with frozen source and raw evidence unchanged. Eight timer admissions lacked a qualifying 100 ms native window at the decision; four initially valid evidence admissions later lost the hold condition. Both variants finish with 2/12 qualified. Closed-loop trajectories differ, so these results do not isolate classifier causality. The audit receipt SHA-256 is `b333f788d388f5e43a035b6bcd85bb6517e197d24a18c92c12efbd6ae3ccbe8f`. + +| Variant | Primary admissions | False qualifications | Conservative abstentions | Later invalidations | Ended in admission hold | Final planning gate | +| --- | ---: | ---: | ---: | ---: | ---: | --- | +| Evidence | 6/12 | 0 | 0 | 4/6 admitted | 2/12 | 2 ready · 10 test lift required | +| Timer | 12/12 | 8 | 0 | 10 total (2/4 correct admissions) | 2/12 | 2 ready · 10 test lift required | + +The audit categories keep false positive admission separate from conservative abstention. These are finite synthetic scene outcomes, not estimated rates for physical objects or robots. + +Across the twelve attempts per variant, mean policy-call CPU compute was 0.045 ms for evidence and 0.044 ms for timer. Mean whole-step CPU compute was 1.707 ms and 1.722 ms, respectively; mean per-attempt step-compute p95 was 2.872 ms and 2.559 ms. Across recorded step wall samples, 5/12,000 exceeded 20 ms for each variant. These are offline timings on an unpaced shared host; complete-step values include native step work and dense recording, and exclude evidence serialization, physical sensing, transport and actuation. They are not real-time guarantees. + +This is not a learned policy, new learning result, arbitrary-video-to-robot transfer, complete placement benchmark or hardware qualification. An earlier PiPER learned policy result of 5/12 remains unchanged; this study does not train or reevaluate it. + +## Reproduce and inspect + +Use Python 3.12 or newer. This CPU-only study needs no GPU, account or API key. Provisioning downloads the pinned public PiPER assets once; subsequent runs are offline. The experimental source branch leaves the stable release unchanged. + +```sh +git clone --branch feat/piper-probe-evidence-v01 https://github.com/Dvidia-Inference/dvidia-training.git +cd dvidia-training +python3.12 -m venv .venv +.venv/bin/python -m pip install '.[arm]' +.venv/bin/dvidia-robot-sim fetch piper --output runs/piper-assets +.venv/bin/python -m dvidia_training.piper_probe_benchmark prepare \ + --assets runs/piper-assets --output runs/my-probe-study +.venv/bin/python -m dvidia_training.piper_probe_benchmark run \ + --assets runs/piper-assets --output runs/my-probe-study +.venv/bin/python -m dvidia_training.piper_probe_benchmark inspect \ + --output runs/my-probe-study +.venv/bin/python benchmarks/piper-probe-evidence-v01/render_report.py \ + runs/my-probe-study/report.json +``` + +Use the corresponding `.venv/Scripts/` executables on Windows. Each experiment requires a fresh output directory; the runner refuses to overwrite or resume a frozen run. The protocol binds local source and asset bytes. Reproduction timings and compiled model hashes may differ by machine. The published study used Python 3.12.9, NumPy 2.5.3 and MuJoCo 3.15.0. About 113 MB of raw evidence was produced; no minimum RAM or real-time target has been qualified. + +The renderer uses only Python's standard library and writes a self-contained `report.html` beside the report JSON. The report, protocol, independent audit and inspection links in the rendered page resolve to colocated files. + +## Publication links + +- Research article: [piper-probe-evidence-v01](https://research.dvidia.org/papers/piper-probe-evidence-v01/) +- Source release: [piper-probe-evidence-v0.1-preview](https://github.com/Dvidia-Inference/dvidia-training/releases/tag/piper-probe-evidence-v0.1-preview) +- Evidence dataset: [Dvidia/piper-probe-evidence-v01](https://huggingface.co/datasets/Dvidia/piper-probe-evidence-v01) + +Original DVIDIA source and authored synthetic records are MIT. Simulation assets and dependencies retain their upstream licenses and notices. diff --git a/benchmarks/piper-probe-evidence-v01/audit_guarded.py b/benchmarks/piper-probe-evidence-v01/audit_guarded.py new file mode 100644 index 0000000..75f91d8 --- /dev/null +++ b/benchmarks/piper-probe-evidence-v01/audit_guarded.py @@ -0,0 +1,50 @@ +"""Run the immutable original probe audit with assertions explicitly enabled. + +The original recorded audit uses assertions. This wrapper refuses optimized +Python before importing or executing it, then pins its exact reviewed bytes. +""" +if not __debug__: + raise RuntimeError('Probe audit refuses optimized Python; run without -O/-OO.') + +import argparse +from hashlib import sha256 +import json +from pathlib import Path +import runpy +import sys + +ORIGINAL_SHA256 = '491ac28f8e9026eb0673b15356dbc7fa164402e873a5da3c5e2e9303fd3733d7' + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument('--audit-script', type=Path, required=True) + parser.add_argument('--guard-receipt', type=Path, required=True) + args, remaining = parser.parse_known_args() + if sys.flags.optimize or args.audit_script.is_symlink() or not args.audit_script.is_file(): + raise RuntimeError('Expected normal Python and a plain original audit file.') + raw = args.audit_script.read_bytes() + if sha256(raw).hexdigest() != ORIGINAL_SHA256: + raise RuntimeError('Original audit source identity changed.') + if args.guard_receipt.exists() or args.guard_receipt.is_symlink(): + raise RuntimeError('Guard receipt must be new.') + output = Path(remaining[remaining.index('--output')+1]) + prior = sys.argv + try: + sys.argv = [str(args.audit_script), *remaining] + runpy.run_path(str(args.audit_script), run_name='__main__') + finally: + sys.argv = prior + result = output.read_bytes() + receipt = {'schema': 'dvidia.probe-audit-normal-mode/v1', 'optimization_level': sys.flags.optimize, + 'assertions_enabled': __debug__, 'original_audit_sha256': ORIGINAL_SHA256, + 'guard_sha256': sha256(Path(__file__).read_bytes()).hexdigest(), + 'readback_output': {'bytes': len(result), 'sha256': sha256(result).hexdigest()}, + 'scope': 'A guarded readback of the same frozen records; no native physics rerun.'} + with args.guard_receipt.open('x') as stream: + json.dump(receipt, stream, sort_keys=True, indent=2, allow_nan=False) + stream.write('\n') + + +if __name__ == '__main__': + main() diff --git a/benchmarks/piper-probe-evidence-v01/package_evidence.py b/benchmarks/piper-probe-evidence-v01/package_evidence.py new file mode 100644 index 0000000..ea2173a --- /dev/null +++ b/benchmarks/piper-probe-evidence-v01/package_evidence.py @@ -0,0 +1,415 @@ +"""Deterministically package reviewed PiPER records, offline and without physics. + +Every frozen study file is bound to the independent audit before indexing and +again after copying. Archive readback verifies every inventory digest. No code +from frozen src is imported, no credentials are read, and nothing is published. +""" +from __future__ import annotations + +import argparse +import gzip +from hashlib import sha256 +import io +import json +from pathlib import Path +import re +import stat +import zipfile + +AUDIT_SHA256 = 'b333f788d388f5e43a035b6bcd85bb6517e197d24a18c92c12efbd6ae3ccbe8f' +SOURCE_COMMIT = 'a6a1d15db905214da52eb3942ebead3f85254762' +ARCHIVE_NAME = 'piper-probe-evidence-v01.zip' +HF_NAMES = {'README.md', 'LICENSE', ARCHIVE_NAME, 'report.json', 'protocol.json', 'inspection.json'} +MAX_FILE = 32*1024*1024 +MAX_TOTAL = 256*1024*1024 +PRIVATE = (b'/'+b'Users/', b'/private/'+b'tmp/', b'/opt/'+b'homebrew/', b'/var/'+b'folders/') +SECRETS = tuple(re.compile(value) for value in (rb'hf_[A-Za-z0-9]{20,}', + rb'gh[pousr]_[A-Za-z0-9]{20,}', rb'apikey_[A-Za-z0-9_]{20,}', rb'AKIA[A-Z0-9]{16}')) + +VERIFY_SOURCE = r'''"""Verify/reconstruct fixed DVIDIA evidence; no code imports or pickle reads.""" +import argparse +from hashlib import sha256 +import json +from pathlib import Path +import re +import stat +import zipfile + +MAX_FILE = 32*1024*1024 +MAX_TOTAL = 256*1024*1024 + +def safe_path(name): + if (type(name) is not str or not re.fullmatch(r'[A-Za-z0-9][A-Za-z0-9._/-]{0,399}', name) + or any(part in ('', '.', '..') for part in name.split('/'))): + raise ValueError('Unsafe artifact path') + return name + +def unique(pairs): + result = {} + for key,value in pairs: + if key in result: raise ValueError('Duplicate JSON field') + result[key] = value + return result + +def verify_archive(path, output=None): + with zipfile.ZipFile(path) as bundle: + entries = bundle.infolist() + names = [entry.filename for entry in entries] + if len(names) != len(set(names)) or not 1 <= len(names) <= 512: + raise ValueError('Duplicate or excessive archive members') + if sum(entry.file_size for entry in entries) > MAX_TOTAL: + raise ValueError('Archive exceeds expanded byte budget') + for entry in entries: + safe_path(entry.filename) + if (entry.file_size > MAX_FILE or entry.flag_bits & 1 + or stat.S_IFMT(entry.external_attr >> 16) != stat.S_IFREG + or entry.compress_type not in (zipfile.ZIP_STORED, zipfile.ZIP_DEFLATED)): + raise ValueError('Unsafe archive member') + if 'inventory.json' not in names or bundle.getinfo('inventory.json').file_size > 1024*1024: + raise ValueError('Missing or oversized inventory') + inventory = json.loads(bundle.read('inventory.json'), object_pairs_hook=unique) + if inventory.get('schema') != 'dvidia.probe-evidence-inventory/v1' or type(inventory.get('files')) is not list: + raise ValueError('Unknown inventory') + records = {} + for row in inventory['files']: + if type(row) is not dict or set(row) != {'path', 'bytes', 'sha256'}: + raise ValueError('Invalid inventory row') + name = safe_path(row['path']) + if (name in records or name == 'inventory.json' or type(row['bytes']) is not int + or not 0 <= row['bytes'] <= MAX_FILE + or type(row['sha256']) is not str or not re.fullmatch(r'[0-9a-f]{64}',row['sha256'])): + raise ValueError('Invalid or duplicate identity') + records[name] = row + if set(names) != set(records) | {'inventory.json'}: + raise ValueError('Inventory roster differs') + for name in records: + parts = name.split('/') + if any('/'.join(parts[:index]) in records for index in range(1,len(parts))): + raise ValueError('File and directory paths collide') + for name,row in records.items(): + if bundle.getinfo(name).file_size != row['bytes']: + raise ValueError('Archive size differs from inventory') + digest,size = sha256(),0 + with bundle.open(name) as stream: + while chunk := stream.read(65536): + size += len(chunk) + if size > row['bytes']: raise ValueError('Member exceeds expected size') + digest.update(chunk) + if size != row['bytes'] or digest.hexdigest() != row['sha256']: + raise ValueError('Archive digest differs from inventory') + if output is not None: + root = Path(output) + if root.exists() or root.is_symlink(): raise ValueError('Output must be new') + root.mkdir(parents=True) + resolved = root.resolve() + for name in sorted(names): + target = root/name + target.parent.mkdir(parents=True,exist_ok=True) + if not target.parent.resolve().is_relative_to(resolved): raise ValueError('Output escape') + with bundle.open(name) as source,target.open('xb') as sink: + while chunk := source.read(65536): sink.write(chunk) + return {'verified':True,'files':len(records),'logical_bytes':sum(r['bytes'] for r in records.values()), + 'reconstructed':output is not None} + +def main(): + parser=argparse.ArgumentParser(description=__doc__) + parser.add_argument('archive',type=Path) + parser.add_argument('--output',type=Path) + args=parser.parse_args() + print(json.dumps(verify_archive(args.archive,args.output),sort_keys=True)) + +if __name__=='__main__': main() +''' + + +def encoded(value): + return json.dumps(value, sort_keys=True, indent=2, allow_nan=False).encode()+b'\n' + + +def identity(path): + if path.is_symlink() or not path.is_file(): + raise ValueError('Expected regular evidence file') + digest, size = sha256(), 0 + with path.open('rb') as stream: + while chunk := stream.read(65536): + size += len(chunk) + if size > MAX_FILE: + raise ValueError('Evidence file exceeds byte bound') + digest.update(chunk) + return {'bytes': size, 'sha256': digest.hexdigest()} + + +def safe_path(name): + if (type(name) is not str or not re.fullmatch(r'[A-Za-z0-9][A-Za-z0-9._/-]{0,399}', name) + or any(part in ('', '.', '..') for part in name.split('/'))): + raise ValueError('Unsafe evidence path') + return name + + +def tree(root): + if root.is_symlink() or not root.is_dir(): + raise ValueError('Evidence root must be a regular directory') + result = {} + for path in sorted(root.rglob('*')): + if path.is_symlink(): + raise ValueError('Evidence links are forbidden') + if path.is_dir(): + continue + name = safe_path(path.relative_to(root).as_posix()) + result[name] = identity(path) + return result + + +def scan_stream(stream): + size, tail = 0, b'' + while chunk := stream.read(65536): + size += len(chunk) + if size > MAX_FILE: + raise ValueError('Decoded scan exceeds its byte bound') + data = tail+chunk + if any(prefix in data for prefix in PRIVATE) or any(pattern.search(data) for pattern in SECRETS): + raise ValueError('Private path or credential-shaped bytes in public evidence') + tail = data[-256:] + return size + + +def scan_file(path): + with path.open('rb') as stream: + scan_stream(stream) + if path.suffix == '.gz': + with gzip.open(path, 'rb') as stream: + raw = stream.read(MAX_FILE+1) + if len(raw) > MAX_FILE: + raise ValueError('Gzip expands beyond byte bound') + scan_stream(io.BytesIO(raw)) + decoded = json.dumps(json.loads(raw), ensure_ascii=False, allow_nan=False).encode() + scan_stream(io.BytesIO(decoded)) + elif path.suffix == '.json': + decoded = json.dumps(json.loads(path.read_bytes()), ensure_ascii=False, allow_nan=False).encode() + scan_stream(io.BytesIO(decoded)) + elif path.suffix == '.npz': + import numpy as np + with zipfile.ZipFile(path) as archive: + entries = archive.infolist() + if (len(entries) != 22 or len({row.filename for row in entries}) != 22 + or sum(row.file_size for row in entries) > MAX_FILE + or any(not re.fullmatch(r'[a-z0-9_]+\.npy', row.filename) or row.file_size > 2*1024*1024 + for row in entries)): + raise ValueError('Unsupported numeric archive') + for row in entries: + with archive.open(row) as stream: + scan_stream(stream) + with np.load(path, allow_pickle=False) as arrays: + for name in arrays.files: + values = arrays[name] + if (values.dtype != np.dtype('float64') or values.ndim not in (1, 2) + or not np.isfinite(values).all() or not 1 <= values.shape[0] <= 30000): + raise ValueError('Expected bounded finite numeric-only sidecar') + + +def frozen_inputs(repo, study, audit): + audit_raw = (audit/'audit.json').read_bytes() + if sha256(audit_raw).hexdigest() != AUDIT_SHA256: + raise ValueError('Independent audit identity changed') + proof = json.loads(audit_raw) + if not proof['passed'] or not proof['raw_files_unchanged'] or not proof['source_files_unchanged']: + raise ValueError('Independent audit did not pass') + if tree(study) != proof['frozen_raw_files']: + raise ValueError('Study roster or bytes differ from the pinned independent audit') + protocol = json.loads((study/'protocol.json').read_bytes()) + if sha256((study/'protocol.json').read_bytes()).hexdigest() != proof['protocol_sha256']: + raise ValueError('Frozen protocol differs') + for name, digest in protocol['source_sha256'].items(): + if identity(repo/'src/dvidia_training'/name)['sha256'] != digest: + raise ValueError('Frozen runtime source changed') + audit_files = tree(audit) + if set(audit_files) != {'audit.py', 'audit.json', 'guarded-audit.json', 'guarded-invocation.json'}: + raise ValueError('Unexpected independent-audit roster') + if audit_files['audit.py'] != proof['script'] or audit_files['guarded-audit.json'] != audit_files['audit.json']: + raise ValueError('Original or guarded audit bytes differ') + guard = json.loads((audit/'guarded-invocation.json').read_bytes()) + if (guard['optimization_level'] != 0 or guard['assertions_enabled'] is not True + or guard['original_audit_sha256'] != proof['script']['sha256'] + or guard['guard_sha256'] != identity(repo/'benchmarks/piper-probe-evidence-v01/audit_guarded.py')['sha256'] + or guard['readback_output'] != audit_files['guarded-audit.json']): + raise ValueError('Guarded normal-mode invocation differs') + expected = {'study/'+name: record for name, record in proof['frozen_raw_files'].items()} + expected.update({'audit/'+name: record for name, record in audit_files.items()}) + expected['audit/audit.json'] = {'bytes': len(audit_raw), 'sha256': AUDIT_SHA256} + return proof, expected + + +def write_archive(path, files, generated, expected): + records = [] + for name, source in sorted(files.items()): + safe_path(name) + record = identity(source) + if name in expected and record != expected[name]: + raise ValueError('Frozen evidence changed before indexing: '+name) + scan_file(source) + records.append({'path': name, **record}) + for name, raw in sorted(generated.items()): + safe_path(name) + if name in files: + raise ValueError('Generated path collision') + scan_stream(io.BytesIO(raw)) + records.append({'path': name, 'bytes': len(raw), 'sha256': sha256(raw).hexdigest()}) + records.sort(key=lambda row: row['path']) + if len(records) > 511 or sum(row['bytes'] for row in records) > MAX_TOTAL-1024*1024: + raise ValueError('Archive exceeds total evidence bound') + inventory = {'schema': 'dvidia.probe-evidence-inventory/v1', 'files': records, + 'source_commit': SOURCE_COMMIT, + 'scope': 'Inventory hashes every file except inventory.json itself. The external package receipt hashes the complete ZIP.'} + content = {**files, **generated, 'inventory.json': encoded(inventory)} + with zipfile.ZipFile(path, 'x', compression=zipfile.ZIP_DEFLATED, compresslevel=9) as archive: + for name, source in sorted(content.items()): + info = zipfile.ZipInfo(name, date_time=(1980, 1, 1, 0, 0, 0)) + info.create_system = 3 + info.external_attr = (stat.S_IFREG | 0o644) << 16 + info.compress_type = zipfile.ZIP_STORED if isinstance(source, Path) and source.suffix in ('.gz', '.npz') else zipfile.ZIP_DEFLATED + digest, size = sha256(), 0 + with archive.open(info, 'w', force_zip64=True) as sink: + with source.open('rb') if isinstance(source, Path) else io.BytesIO(source) as stream: + while chunk := stream.read(65536): + size += len(chunk) + if size > MAX_FILE: + raise ValueError('File grew during packaging') + digest.update(chunk) + sink.write(chunk) + if name != 'inventory.json': + record = next(row for row in records if row['path'] == name) + if record['bytes'] != size or record['sha256'] != digest.hexdigest(): + raise ValueError('File changed while copying: '+name) + return inventory + + +def archive_identity(path): + digest, size = sha256(), 0 + with path.open('rb') as stream: + while chunk := stream.read(65536): + size += len(chunk) + if size > MAX_TOTAL: + raise ValueError('ZIP exceeds its publication byte bound') + digest.update(chunk) + return {'bytes': size, 'sha256': digest.hexdigest()} + + +def prepare(repo, study, audit, output): + if output.exists() or output.is_symlink(): + raise ValueError('Package output must be new') + proof, expected = frozen_inputs(repo, study, audit) + files = {'study/'+name: study/name for name in proof['frozen_raw_files']} + files.update({'audit/'+name: audit/name for name in ('audit.py', 'audit.json', 'guarded-audit.json', 'guarded-invocation.json')}) + files.update({'benchmarks/piper-probe-evidence-v01/README.md': repo/'benchmarks/piper-probe-evidence-v01/README.md', + 'benchmarks/piper-probe-evidence-v01/render_report.py': repo/'benchmarks/piper-probe-evidence-v01/render_report.py', + 'tools/package_evidence.py': Path(__file__), + 'tools/test_package_evidence.py': repo/'benchmarks/piper-probe-evidence-v01/test_package_evidence.py', + 'tools/audit_guarded.py': repo/'benchmarks/piper-probe-evidence-v01/audit_guarded.py', + 'licenses/agilex_piper-LICENSE.txt': repo/'third_party/robot-models/agilex_piper-LICENSE.txt', + 'licenses/mujoco-LICENSE.txt': repo/'third_party/mujoco/LICENSE.txt', + 'licenses/mujoco-THIRD_PARTY_NOTICES.txt': repo/'third_party/mujoco/THIRD_PARTY_NOTICES.txt'}) + readme = b'''# DVIDIA | PiPER probe evidence v0.1 + +Simulation-only authored short-lift/hold research on 12 fixed synthetic scenes, +once per controller (24 attempts). Independent reconstruction checked 480,000 +native 1 ms rows. Evidence gate: 6 admissions, 0/6 false initial qualifications; +four later hold losses, final 2/12 qualified. Timer: 12 admissions, 8/12 false +initial qualifications; final 2/12 qualified. Closed-loop trajectories can differ. +These counts do not establish general accuracy, speed or hardware safety. + +The historical learned 5/12 placement result is unchanged. No training, model +weights, full transport/placement or physical sensor qualification is included. +The 10 N per-finger ceiling is an arbitrary synthetic assumption, not calibration. + +Paths under study/ preserve every frozen study artifact. Paths under audit/ +preserve the independent original audit and its receipt, plus a guarded normal- +mode reanalysis. The original audit used assertions and was executed with normal +Python. Always use tools/audit_guarded.py for reuse: it refuses -O/-OO and pins +the original script. The guarded reanalysis was byte-identical to the original. + +After checking the release ZIP checksum, copy tools/verify_archive.py from the +ZIP or reviewed repository and verify/reconstruct it with standard Python: + + python verify_archive.py piper-probe-evidence-v01.zip --output evidence + +With the project dependencies and frozen checkout available, re-read the saved +records (no physics rerun), writing new outputs outside the archived study: + + python evidence/tools/audit_guarded.py --audit-script evidence/audit/audit.py --guard-receipt new-guard.json --study evidence/study --repo /path/to/frozen-checkout --output new-audit.json + +Frozen source: https://github.com/Dvidia-Inference/dvidia-training/tree/a6a1d15db905214da52eb3942ebead3f85254762 +Research article (publication target): https://research.dvidia.org/papers/piper-probe-evidence-v01/ +Source release (publication target): https://github.com/Dvidia-Inference/dvidia-training/releases/tag/piper-probe-evidence-v0.1-preview +Evidence dataset (publication target): https://huggingface.co/datasets/Dvidia/piper-probe-evidence-v01 + +Original DVIDIA code and authored synthetic records are MIT. Upstream notices +remain included. No robot meshes, simulator binaries or credentials are bundled. +DVIDIA research: https://research.dvidia.org | hello@dvidia.org +''' + generated = {'README.md': readme, 'LICENSE': (repo/'LICENSE').read_bytes(), + 'tools/verify_archive.py': VERIFY_SOURCE.encode(), + 'SOURCE.json': encoded({'frozen_commit': SOURCE_COMMIT, + 'protocol_sha256': proof['protocol_sha256'], + 'independent_audit_sha256': AUDIT_SHA256, + 'normal_mode_reanalysis': 'audit/guarded-invocation.json'})} + output.mkdir(parents=True) + stage = output/'huggingface'; stage.mkdir() + archive = stage/ARCHIVE_NAME + inventory = write_archive(archive, files, generated, expected) + verify_namespace = {'__name__': 'dvidia_evidence_verifier'} + exec(compile(VERIFY_SOURCE, 'verify_archive.py', 'exec'), verify_namespace) + verified = verify_namespace['verify_archive'](archive) + frozen_inputs(repo, study, audit) + archive_id = archive_identity(archive) + card = b'''--- +license: mit +pretty_name: DVIDIA PiPER probe evidence v0.1 +task_categories: +- robotics +tags: +- robotics +- simulation +- piper +- mujoco +- cpu +- research +--- + +'''+readme+b'\nArchive SHA-256: `'+archive_id['sha256'].encode()+b'`\n' + payload = {'README.md': card, 'LICENSE': generated['LICENSE'], + **{name: (study/name).read_bytes() for name in ('report.json', 'protocol.json', 'inspection.json')}} + for name, raw in payload.items(): + scan_stream(io.BytesIO(raw)) + if name in ('report.json', 'protocol.json', 'inspection.json') and {'bytes': len(raw), 'sha256': sha256(raw).hexdigest()} != expected['study/'+name]: + raise ValueError('Frozen top-level payload changed') + with (stage/name).open('xb') as stream: + stream.write(raw) + if set(path.name for path in stage.iterdir()) != HF_NAMES: + raise ValueError('Hugging Face payload must contain the exact six files') + receipt = {'schema': 'dvidia.probe-evidence-package/v1', 'remote_operations': False, + 'source_commit': SOURCE_COMMIT, 'target_dataset': 'Dvidia/piper-probe-evidence-v01', + 'archive': {'filename': ARCHIVE_NAME, **archive_id}, 'verified': verified, + 'frozen_study_files': len(proof['frozen_raw_files']), 'frozen_inputs_unchanged': True, + 'scan': {'passed': True, 'decoded_gzip_json': True, 'numeric_only_npz': True, + 'source_and_text_paths_and_credential_shapes': True}, + 'guarded_reanalysis': json.loads((audit/'guarded-invocation.json').read_bytes()), + 'payload': [{'path': path.name, **(archive_id if path.name == ARCHIVE_NAME else identity(path))} + for path in sorted(stage.iterdir())]} + (output/'inventory.json').write_bytes(encoded(inventory)) + (output/'package-receipt.json').write_bytes(encoded(receipt)) + return receipt + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument('--repo', type=Path, required=True) + parser.add_argument('--study', type=Path, required=True) + parser.add_argument('--audit', type=Path, required=True) + parser.add_argument('--output', type=Path, required=True) + args = parser.parse_args() + result = prepare(args.repo, args.study, args.audit, args.output) + print(json.dumps({'archive': result['archive'], 'verified': result['verified']})) + + +if __name__ == '__main__': + main() diff --git a/benchmarks/piper-probe-evidence-v01/render_report.py b/benchmarks/piper-probe-evidence-v01/render_report.py new file mode 100644 index 0000000..245f196 --- /dev/null +++ b/benchmarks/piper-probe-evidence-v01/render_report.py @@ -0,0 +1,127 @@ +#!/usr/bin/env python3 +"""Render the bounded piper-probe-study/v1 report as a self-contained HTML page.""" +import argparse +import html +import json +from pathlib import Path + +SCHEMA = 'dvidia.piper-probe-study/v1' +MAX_BYTES = 16 * 1024 * 1024 + + +def load(path): + path = Path(path) + if path.is_symlink() or not path.is_file() or path.stat().st_size > MAX_BYTES: + raise ValueError('Expected a bounded regular report.json file.') + with path.open('r', encoding='utf-8') as stream: + value = json.load(stream, parse_constant=lambda x: (_ for _ in ()).throw(ValueError('Non-finite JSON number'))) + if type(value) is not dict or value.get('schema') != SCHEMA or type(value.get('rows')) is not list: + raise ValueError('Unsupported report schema.') + return value + + +def esc(value): + return html.escape(str(value), quote=True) + + +def ms(summary, name): + value = (summary or {}).get(name) + if type(value) is not dict or type(value.get('mean_s')) not in (int, float): + return '—' + p95 = value.get('p95_s') + return f"{value['mean_s']*1000:.2f} / {p95*1000:.2f} ms" if type(p95) in (int, float) else f"{value['mean_s']*1000:.2f} ms" + + +def attempt(row): + evaluation = row.get('evaluation') or {} + receipt = evaluation.get('receipt') or {} + audit = receipt.get('audit') or {} + verification = evaluation.get('verification') or {} + admission = evaluation.get('admission') or {} + policy = row.get('policy') or {} + probe_index = ((receipt.get('context') or {}).get('probe_index')) + if type(probe_index) is not int: + probes = policy.get('probes') or [] + probe_index = probes[-1].get('probe_index') if probes else None + events = [e for e in (policy.get('events') or []) + if type(e) is dict and e.get('probe_index') == probe_index] + decisions = [e for e in events if e.get('type') in ('evidence_qualified', 'timer_admitted')] + primary = decisions[-1] if decisions else None + primary_text = (f"{primary['type']} @ {primary['t_seconds']:.2f}s" + if primary and type(primary.get('t_seconds')) in (int, float) else 'no admission') + terminal = [e for e in events if e.get('type') in ('hold_invalidated', 'abort')] + terminal_text = (f"{terminal[-1].get('type')}: {terminal[-1].get('reason', '—')}" + if terminal else f"no invalidation · {policy.get('status', 'unknown')}") + false_n = audit.get('controller_false_qualification_count', '—') + abstain_n = audit.get('controller_conservative_abstention_count', '—') + disagreements = audit.get('controller_claim_mismatches') + disagree_n = len(disagreements) if isinstance(disagreements, list) else '—' + audit_text = f"false qualification {false_n} · conservative abstention {abstain_n} · disagreements {disagree_n}" + state = 'valid' if row.get('valid') is True else 'invalid' + native = verification.get('admissible') + native_text = 'admissible' if native is True else ('not admissible' if native is False else 'not inspected') + gate = admission.get('status') or (admission.get('planning') or {}).get('status') or 'unknown' + return '' + ''.join(f'{esc(x)}' for x in ( + row.get('id', '—'), state, primary_text, terminal_text, native_text, + audit_text, gate, ms(row.get('timing'), 'policy_cpu'), ms(row.get('timing'), 'step_cpu'))) + '' + + +def render(report): + rows = report['rows'] + variants = ('evidence', 'timer') + body = [] + for variant in variants: + group = [r for r in rows if r.get('variant') == variant] + admitted = sum(any(e.get('type') in ('evidence_qualified', 'timer_admitted') + for e in (r.get('policy') or {}).get('events', [])) for r in group) + false_n = sum((((r.get('evaluation') or {}).get('receipt') or {}).get('audit') or {}).get( + 'controller_false_qualification_count', 0) for r in group) + abstain_n = sum((((r.get('evaluation') or {}).get('receipt') or {}).get('audit') or {}).get( + 'controller_conservative_abstention_count', 0) for r in group) + terminal = {} + for row in group: + status = (row.get('policy') or {}).get('status', 'unknown') + terminal[status] = terminal.get(status, 0) + 1 + outcome = ', '.join(f'{esc(k)} {v}' for k, v in sorted(terminal.items())) or 'no attempts' + body.append(f'

{esc(variant.title())} policy · {len(group)} attempts

') + body.append(f'

Primary admissions {admitted}/{len(group)} · audited false qualifications {false_n} · conservative abstentions {abstain_n} · terminal policy statuses: {outcome}.

') + body.append('
') + body.extend(attempt(r) for r in group) + body.append('
Attempt / sceneNative runPrimary admission eventTerminal event / stateFinal 1 ms evidenceAudited claims: false qual · abstention · disagreementFinal planning gatePolicy CPU mean / p95Whole-step CPU mean / p95
') + valid = report.get('valid_attempts', sum(r.get('valid') is True for r in rows)) + return f''' + +PiPER Probe Evidence v0.1 · DVIDIA Research +
DVIDIA · Research preview · PiPER
+

Probe, verify, hold.

A bounded native simulation study compares an evidence-based short-lift gate with a fixed timer gate. Both stop in a measured-position hold; neither performs full placement or transport.

+ +
{len(rows)}attempt records
{valid}/{len(rows)}valid native runs
{esc(report.get('planned_attempts', '—'))}planned attempts
+

{esc(report.get('scope', ''))}

+{''.join(body)} +

How to read this

The 1 ms evidence audit independently checks recorded native simulator rows. The policy gate is sampled at control boundaries; the report preserves any disagreement. “Final planning gate” combines the simulation receipt with caller-declared load, friction, geometry, force-limit and pose assumptions. It is a planning result, not permission to actuate.

+

CPU values show per-attempt policy-call and complete-step computation summaries, in milliseconds (mean / p95). They exclude sensors, transport, physical actuation and evidence serialization. Host timing is not a real-time guarantee.

+

The evidence receipt uses privileged simulator state and an authored synthetic force ceiling. It does not measure real material properties, establish a calibrated force limit, or demonstrate that a physical gripper can hold the object.

+ +
''' + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument('report', type=Path, help='bounded report.json') + parser.add_argument('--output', type=Path, help='HTML path (default: beside report.json)') + args = parser.parse_args() + report = load(args.report) + output = args.output or args.report.with_name('report.html') + output.write_text(render(report), encoding='utf-8') + print(output) + + +if __name__ == '__main__': + main() diff --git a/benchmarks/piper-probe-evidence-v01/test_package_evidence.py b/benchmarks/piper-probe-evidence-v01/test_package_evidence.py new file mode 100644 index 0000000..b8f748d --- /dev/null +++ b/benchmarks/piper-probe-evidence-v01/test_package_evidence.py @@ -0,0 +1,114 @@ +"""Archive/readback boundaries and immutable-input checks; no native runs.""" +from hashlib import sha256 +import importlib.util +import io +import json +import gzip +from pathlib import Path +import stat +import subprocess +import sys +import tempfile +import unittest +import zipfile + +_SPEC = importlib.util.spec_from_file_location('probe_package', Path(__file__).with_name('package_evidence.py')) +package = importlib.util.module_from_spec(_SPEC) +_SPEC.loader.exec_module(package) +verify = {'__name__': 'probe_archive_test'} +exec(compile(package.VERIFY_SOURCE, 'verify_archive.py', 'exec'), verify) + + +class PackagingTests(unittest.TestCase): + def fixture(self, root): + source = root/'source.json' + source.write_text('{"unchanged":true}\n') + return {'study/source.json': source}, {'README.md': b'Public research\n'}, {'study/source.json': package.identity(source)} + + def test_deterministic_archive_verification_reconstruction_and_no_overwrite(self): + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + files, generated, frozen = self.fixture(root) + a, b = root/'a.zip', root/'b.zip' + package.write_archive(a, files, generated, frozen) + package.write_archive(b, files, generated, frozen) + self.assertEqual(a.read_bytes(), b.read_bytes()) + result = verify['verify_archive'](a, root/'restored') + self.assertEqual(result['files'], 2) + self.assertEqual((root/'restored/study/source.json').read_bytes(), next(iter(files.values())).read_bytes()) + with self.assertRaises(ValueError): + verify['verify_archive'](a, root/'restored') + + def test_changed_scientific_bytes_refused_before_indexing(self): + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + files, generated, frozen = self.fixture(root) + next(iter(files.values())).write_text('{"unchanged":nope}\n') + with self.assertRaisesRegex(ValueError, 'Frozen evidence changed'): + package.write_archive(root/'bad.zip', files, generated, frozen) + self.assertFalse((root/'bad.zip').exists()) + + def test_traversal_symlink_unknown_blob_and_digest_mutation_rejected(self): + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + files, generated, frozen = self.fixture(root) + package.write_archive(root/'a.zip', files, generated, frozen) + with zipfile.ZipFile(root/'a.zip') as archive: + original = {name: archive.read(name) for name in archive.namelist()} + for index, mutation in enumerate(('path', 'extra', 'data', 'link')): + content = dict(original) + if mutation == 'path': + inventory = json.loads(content['inventory.json']) + inventory['files'][0]['path'] = '../escape' + content['inventory.json'] = json.dumps(inventory).encode() + elif mutation == 'extra': + content['unreviewed.txt'] = b'extra' + elif mutation == 'data': + content['study/source.json'] += b'changed' + path = root/f'bad-{index}.zip' + with zipfile.ZipFile(path, 'x') as archive: + for name, raw in content.items(): + info = zipfile.ZipInfo(name) + info.external_attr = ((stat.S_IFLNK if mutation == 'link' else stat.S_IFREG) | 0o644) << 16 + archive.writestr(info, raw) + with self.assertRaises(ValueError): + verify['verify_archive'](path) + + def test_plain_tree_and_scans_refuse_private_paths_credentials_and_links(self): + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + source = root/'original'; source.write_bytes(b'public') + (root/'link').symlink_to(source) + with self.assertRaises(ValueError): + package.tree(root) + for raw in (b'a'*65534+b'/'+b'Users/'+b'person', b'hf_'+b'x'*24): + with self.assertRaises(ValueError): + package.scan_stream(io.BytesIO(raw)) + for bad in ('../escape', '/absolute', 'a//b', 'a\\b'): + with self.assertRaises(ValueError): + package.safe_path(bad) + + def test_escaped_private_json_and_decoded_gzip_are_scanned(self): + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + raw = b'{"path":"\\u002fUsers\\u002fperson"}' + for name, content in (('data.json', raw), ('data.json.gz', gzip.compress(raw))): + path = root/name + path.write_bytes(content) + with self.assertRaises(ValueError): + package.scan_file(path) + + def test_guarded_audit_refuses_optimized_mode_before_any_output(self): + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + wrapper = Path(__file__).with_name('audit_guarded.py') + completed = subprocess.run([sys.executable, '-O', str(wrapper), '--audit-script', str(root/'missing.py'), + '--guard-receipt', str(root/'guard.json'), '--output', str(root/'audit.json')], + capture_output=True, text=True, check=False) + self.assertNotEqual(completed.returncode, 0) + self.assertIn('refuses optimized Python', completed.stderr) + self.assertEqual(list(root.iterdir()), []) + + +if __name__ == '__main__': + unittest.main() diff --git a/publishing/huggingface/publish_piper_probe_evidence_v01.py b/publishing/huggingface/publish_piper_probe_evidence_v01.py new file mode 100644 index 0000000..d31ff18 --- /dev/null +++ b/publishing/huggingface/publish_piper_probe_evidence_v01.py @@ -0,0 +1,184 @@ +"""Publish one reviewed native-probe evidence dataset; never modify a Space. + +Only publish loads the existing named Keychain credential. All six immutable +payloads are verified and held in memory first. Existing dataset content is +never overwritten; the single commit is conditional on the empty public head. +""" +from __future__ import annotations + +import argparse +from hashlib import sha256 +import json +from pathlib import Path +import re +import sys + +from publish import PublicationError, client +from publish_task_environments_v02 import digest, encoded, plain, public_client, download, scan + + +REPO = 'Dvidia/piper-probe-evidence-v01' +SCHEMA = 'dvidia.piper-probe-evidence-publication/v1' +ARCHIVE = 'piper-probe-evidence-v01.zip' +NAMES = {'README.md', 'LICENSE', ARCHIVE, 'report.json', 'protocol.json', 'inspection.json'} +PLAN_LIMIT = 4*1024*1024 +LIMITS = {name: (256*1024*1024 if name == ARCHIVE else + 16*1024*1024 if name == 'report.json' else 4*1024*1024) + for name in NAMES} +OID = re.compile(r'(?:[a-f0-9]{40}|[a-f0-9]{64})\Z') + + +def read(path, limit=None): + path = plain(path) + limit = LIMITS.get(path.name, PLAN_LIMIT) if limit is None else limit + if not path.is_file() or path.stat().st_size > limit: + raise PublicationError('Missing or oversized reviewed publication file.') + with path.open('rb') as stream: + raw = stream.read(limit+1) + if len(raw) > limit: + raise PublicationError('Publication file changed beyond its byte limit.') + return raw + + +def files(root): + root = plain(root) + if not root.is_dir(): + raise PublicationError('Expected a plain publication directory.') + paths = sorted(root.iterdir()) + if {path.name for path in paths} != NAMES: + raise PublicationError('Dataset must contain exactly the reviewed six files.') + inventory = [] + for path in paths: + raw = read(path) + scan(raw) + inventory.append({'path': path.name, **digest(raw), + 'representation': 'lfs' if path.name == ARCHIVE else 'git'}) + return inventory + + +def stage(root, plan): + root, plan = plain(root), plain(plan) + value = {'schema': SCHEMA, 'repo_id': REPO, 'repo_type': 'dataset', + 'private': False, 'root': str(root), 'files': files(root)} + with plan.open('xb') as stream: + stream.write(encoded(value)) + return {'plan_sha256': sha256(read(plan, PLAN_LIMIT)).hexdigest(), + 'files': len(value['files']), 'bytes': sum(row['bytes'] for row in value['files'])} + + +def _pairs(items): + result = {} + for key, value in items: + if key in result: + raise PublicationError('Duplicate keys in reviewed publication plan.') + result[key] = value + return result + + +def _constant(_value): + raise PublicationError('Nonfinite values in reviewed publication plan.') + + +def checked(plan, expected_digest): + raw = read(plan, PLAN_LIMIT) + if sha256(raw).hexdigest() != expected_digest: + raise PublicationError('Reviewed plan digest changed.') + try: + value = json.loads(raw, object_pairs_hook=_pairs, parse_constant=_constant) + except (ValueError, TypeError, UnicodeError) as exc: + raise PublicationError('Malformed reviewed publication plan.') from exc + if (type(value) is not dict + or set(value) != {'schema', 'repo_id', 'repo_type', 'private', 'root', 'files'} + or value['schema'] != SCHEMA or value['repo_id'] != REPO + or value['repo_type'] != 'dataset' or value['private'] is not False + or type(value['root']) is not str or type(value['files']) is not list + or len(value['files']) != len(NAMES)): + raise PublicationError('Reviewed dataset identity changed.') + for row in value['files']: + if (type(row) is not dict or set(row) != {'path', 'bytes', 'sha256', 'representation'} + or type(row['path']) is not str or row['path'] not in NAMES + or type(row['bytes']) is not int or row['bytes'] < 0 + or type(row['sha256']) is not str or not re.fullmatch(r'[a-f0-9]{64}', row['sha256']) + or row['representation'] != ('lfs' if row['path'] == ARCHIVE else 'git')): + raise PublicationError('Reviewed dataset inventory is malformed.') + if ({row['path'] for row in value['files']} != NAMES + or files(value['root']) != value['files']): + raise PublicationError('Reviewed dataset identity or bytes changed.') + return value + + +def _remote(info): + if (getattr(info, 'private', None) is not False + or not OID.fullmatch(getattr(info, 'sha', '') or '') + or not getattr(info, 'siblings', None)): + raise PublicationError('Expected an initialized public dataset with an exact commit.') + + +def publish(plan, expected_digest, cache): + value = checked(plan, expected_digest) + payloads = [] + for row in value['files']: + raw = read(Path(value['root'])/row['path']) + if digest(raw) != {key: row[key] for key in ('bytes', 'sha256')}: + raise PublicationError('Payload changed while freezing reviewed publication bytes.') + payloads.append((row['path'], raw)) + # No payload file is reopened after credential loading or remote access. + api, token = client(cache) + from huggingface_hub import CommitOperationAdd + from huggingface_hub.errors import RepositoryNotFoundError + try: + info = api.repo_info(REPO, repo_type='dataset', files_metadata=True) + except RepositoryNotFoundError: + api.create_repo(REPO, repo_type='dataset', private=False, exist_ok=False) + info = api.repo_info(REPO, repo_type='dataset', files_metadata=True) + _remote(info) + if len(info.siblings) != 1 or {item.rfilename for item in info.siblings} != {'.gitattributes'}: + raise PublicationError('Dataset is not empty. Verify the existing publication; do not overwrite it.') + operations = [CommitOperationAdd(path_in_repo=name, path_or_fileobj=raw) for name, raw in payloads] + commit = api.create_commit(REPO, repo_type='dataset', operations=operations, + parent_commit=info.sha, + commit_message='Publish native PiPER short-probe, hold and independent evidence study') + del token, api + return {'repo_id': REPO, 'commit': commit.oid, 'files_added': len(operations)} + + +def verify(plan, expected_digest, cache): + value = checked(plan, expected_digest) + api = public_client() + info = api.repo_info(REPO, repo_type='dataset', files_metadata=True, token=False) + _remote(info) + if (len(info.siblings) != len(NAMES)+1 + or {item.rfilename for item in info.siblings} != NAMES | {'.gitattributes'}): + raise PublicationError('Public dataset roster differs from the reviewed files.') + for row in value['files']: + raw = download(REPO, row['path'], 'dataset', info.sha, cache) + if digest(raw) != {key: row[key] for key in ('bytes', 'sha256')}: + raise PublicationError('Anonymous public bytes differ from the reviewed artifacts.') + return {'repo_id': REPO, 'commit': info.sha, 'anonymous_files_verified': len(value['files'])} + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + commands = parser.add_subparsers(dest='command', required=True) + command = commands.add_parser('stage') + command.add_argument('root'); command.add_argument('--plan', required=True) + for name in ('publish', 'verify'): + command = commands.add_parser(name) + command.add_argument('--plan', required=True) + command.add_argument('--sha256', required=True) + command.add_argument('--cache', required=True) + args = parser.parse_args() + try: + result = (stage(args.root, args.plan) if args.command == 'stage' else + globals()[args.command](args.plan, args.sha256, Path(args.cache))) + print(json.dumps(result, indent=2)) + return 0 + except PublicationError as exc: + print(str(exc), file=sys.stderr) + except Exception: + print('Publication operation failed; no credential or remote response details displayed.', file=sys.stderr) + return 2 + + +if __name__ == '__main__': + raise SystemExit(main()) diff --git a/publishing/huggingface/test_publish_piper_probe_evidence_v01.py b/publishing/huggingface/test_publish_piper_probe_evidence_v01.py new file mode 100644 index 0000000..46c73c8 --- /dev/null +++ b/publishing/huggingface/test_publish_piper_probe_evidence_v01.py @@ -0,0 +1,128 @@ +from copy import deepcopy +from hashlib import sha256 +import json +from pathlib import Path +from types import SimpleNamespace +import tempfile +import unittest +from unittest.mock import Mock, patch + +import publish_piper_probe_evidence_v01 as publisher + + +class PublicationTests(unittest.TestCase): + def setUp(self): + temporary = tempfile.TemporaryDirectory() + self.addCleanup(temporary.cleanup) + self.root = Path(temporary.name).resolve() + self.dataset = self.root/'dataset'; self.dataset.mkdir() + for name in publisher.NAMES: + (self.dataset/name).write_bytes(b'synthetic publication fixture') + self.plan = self.root/'plan.json' + self.receipt = publisher.stage(self.dataset, self.plan) + + def info(self, names=('.gitattributes',), private=False, revision='a'*40): + return SimpleNamespace(private=private, sha=revision, + siblings=[SimpleNamespace(rfilename=name) for name in names]) + + def test_staging_checking_and_anonymous_verification_are_credential_free(self): + api = Mock(); api.repo_info.return_value = self.info(sorted(publisher.NAMES | {'.gitattributes'})) + with patch.object(publisher, 'client', side_effect=AssertionError('Credential accessed')), \ + patch.object(publisher, 'public_client', return_value=api), \ + patch.object(publisher, 'download', side_effect=lambda repo,name,kind,revision,cache: (self.dataset/name).read_bytes()) as download: + value = publisher.checked(self.plan, self.receipt['plan_sha256']) + proof = publisher.verify(self.plan, self.receipt['plan_sha256'], self.root/'cache') + self.assertEqual(value['repo_id'], 'Dvidia/piper-probe-evidence-v01') + self.assertEqual(proof['anonymous_files_verified'], 6) + self.assertEqual(download.call_count, 6) + api.repo_info.assert_called_once_with(publisher.REPO, repo_type='dataset', files_metadata=True, token=False) + self.assertTrue(all(call.args[3] == 'a'*40 for call in download.call_args_list)) + with self.assertRaises(FileExistsError):publisher.stage(self.dataset, self.plan) + + def test_mutated_payload_and_unreviewed_roster_fail_before_credentials(self): + with patch.object(publisher, 'client') as client: + (self.dataset/'README.md').write_bytes(b'changed') + with self.assertRaises(publisher.PublicationError): + publisher.publish(self.plan, self.receipt['plan_sha256'], self.root/'cache') + (self.dataset/'README.md').write_bytes(b'synthetic publication fixture') + (self.dataset/'unreviewed.json').write_text('{}') + with self.assertRaises(publisher.PublicationError): + publisher.publish(self.plan, self.receipt['plan_sha256'], self.root/'cache') + client.assert_not_called() + + def test_resealed_destination_duplicate_and_nonfinite_plans_are_rejected(self): + original = json.loads(self.plan.read_bytes()) + for field, replacement in [('repo_id','Dvidia/robot-simulation-lab'), ('repo_type','space'), ('private',True)]: + value = deepcopy(original); value[field] = replacement + raw = publisher.encoded(value); self.plan.write_bytes(raw) + with self.subTest(field=field),self.assertRaises(publisher.PublicationError): + publisher.checked(self.plan, sha256(raw).hexdigest()) + for raw in (b'{"schema":1,"schema":2}', b'{"private":NaN}'): + self.plan.write_bytes(raw) + with self.assertRaises(publisher.PublicationError): + publisher.checked(self.plan, sha256(raw).hexdigest()) + value = deepcopy(original); value['files'][1] = value['files'][0] + raw = publisher.encoded(value); self.plan.write_bytes(raw) + with self.assertRaises(publisher.PublicationError):publisher.checked(self.plan, sha256(raw).hexdigest()) + + def test_existing_private_or_unpinned_dataset_cannot_be_overwritten(self): + for info in (self.info(['README.md']), self.info(private=True), self.info(revision='main')): + api = Mock(); api.repo_info.return_value = info + with self.subTest(info=info),patch.object(publisher, 'client', return_value=(api,'synthetic-fixture')): + with self.assertRaises(publisher.PublicationError): + publisher.publish(self.plan, self.receipt['plan_sha256'], self.root/'cache') + api.create_commit.assert_not_called() + api.create_repo.assert_not_called() + + def test_payload_freeze_detects_changes_and_keeps_bytes_across_credential_boundary(self): + original = publisher.read + reads = 0 + def changed(path, *args, **kwargs): + nonlocal reads + if Path(path).name == 'README.md': + reads += 1 + if reads == 2:return b'changed after review' + return original(path, *args, **kwargs) + with patch.object(publisher, 'read', side_effect=changed),patch.object(publisher, 'client') as client: + with self.assertRaisesRegex(publisher.PublicationError, 'freezing'): + publisher.publish(self.plan, self.receipt['plan_sha256'], self.root/'cache') + client.assert_not_called() + api = Mock(); api.repo_info.return_value = self.info() + api.create_commit.return_value = SimpleNamespace(oid='b'*40) + def credential_boundary(_cache): + (self.dataset/'README.md').write_bytes(b'late mutation must not be uploaded') + return api, 'synthetic-fixture' + with patch.object(publisher, 'client', side_effect=credential_boundary): + result = publisher.publish(self.plan, self.receipt['plan_sha256'], self.root/'cache') + self.assertEqual(result['files_added'], 6) + commit = api.create_commit.call_args + self.assertEqual(commit.kwargs['parent_commit'], 'a'*40) + self.assertEqual(commit.args, (publisher.REPO,)) + self.assertEqual(commit.kwargs['repo_type'], 'dataset') + uploaded = {operation.path_in_repo:operation.path_or_fileobj for operation in commit.kwargs['operations']} + self.assertEqual(uploaded['README.md'], b'synthetic publication fixture') + + def test_oversized_report_other_metadata_archive_and_plan_fail_before_reading(self): + for name, size in [('report.json',16*1024*1024+1), ('protocol.json',4*1024*1024+1), + (publisher.ARCHIVE,256*1024*1024+1)]: + path = self.dataset/name + with path.open('wb') as stream:stream.truncate(size) + with self.subTest(name=name),self.assertRaisesRegex(publisher.PublicationError, 'oversized'): + publisher.read(path) + with self.plan.open('wb') as stream:stream.truncate(4*1024*1024+1) + with self.assertRaisesRegex(publisher.PublicationError, 'oversized'): + publisher.checked(self.plan, '0'*64) + + def test_symlinks_and_anonymous_readback_mismatch_are_rejected(self): + api = Mock(); api.repo_info.return_value = self.info(sorted(publisher.NAMES | {'.gitattributes'})) + with patch.object(publisher, 'client', side_effect=AssertionError('Credential accessed')), \ + patch.object(publisher, 'public_client', return_value=api),patch.object(publisher, 'download', return_value=b'changed'): + with self.assertRaisesRegex(publisher.PublicationError, 'Anonymous public bytes'): + publisher.verify(self.plan, self.receipt['plan_sha256'], self.root/'cache') + target = self.root/'unreviewed'; target.write_text('synthetic') + path = self.dataset/'README.md'; path.unlink(); path.symlink_to(target) + with self.assertRaises(publisher.PublicationError):publisher.checked(self.plan, self.receipt['plan_sha256']) + + +if __name__ == '__main__': + unittest.main() diff --git a/src/dvidia_training/piper_probe.py b/src/dvidia_training/piper_probe.py new file mode 100644 index 0000000..a7bb3ad --- /dev/null +++ b/src/dvidia_training/piper_probe.py @@ -0,0 +1,302 @@ +"""Bounded native PiPER acquisition followed by a measured-position hold. + +Both variants inherit the authored guarded approach/closure/supported retry. +The evidence variant admits a sampled 100 ms loaded probe; the timer variant +admits after a fixed 600 ms probe clock, without testing probe evidence first. +Neither dispatches the teacher's full lift or transport. Every observation is +privileged simulator state, not calibrated physical telemetry. A held position +target is not an instantaneous physical stop or a force limiter. +""" +from __future__ import annotations + +from copy import deepcopy +from dataclasses import asdict +import math + +from . import piper_grip_guard as guard + + +SCHEMA = 'dvidia.piper-probe-stop/v1' +VARIANTS = ('evidence', 'timer') +ORIGIN = 'privileged_native_simulator_state' +TIMER_ADMISSION_S = .60 +DT = .02 + + +def probe_contract(): + return { + 'schema': SCHEMA, 'variants': list(VARIANTS), 'parameters': asdict(guard.GuardConfig()), + 'timer_admission_s': TIMER_ADMISSION_S, 'control_dt_s': DT, + 'observation_origin': ORIGIN, 'physical_robot_ready': False, + 'prefix': 'Original GuardedPiperTaskPolicy approach/close and at most one supported open/recenter/retry; already uses authored native state, not a fully blind baseline.', + 'evidence_admission': 'Current bilateral force >0.01 N, no table support, object rise >=6 mm and TCP-local offset drift <=3 mm continuously across sampled boundaries for elapsed >=100 ms.', + 'timer_admission': 'Same 10 mm/20 mm/s probe target law, admitted at elapsed >=600 ms from that probe start; no evidence predicate or slip alarm before admission. Unexpected contacts and bounded timeouts still stop.', + 'hold': 'Latch measured six arm joints and the last issued jaw target at admission/terminal invalidity. No full lift/transfer, no squeeze escalation, no post-admission retry.', + 'invalidation': 'At each subsequent 20 ms observation and final observation, invalidate immediately on lost contact/force, support, insufficient rise, >3 mm local drift or unexpected contact. Never requalify after terminal invalidity.', + 'sampling_limit': 'Independent 1 ms evidence may contradict sampled admission; that remains a finding. No sensor or material inference is implemented.', + 'stop_scope': 'Holding native position targets is not physical arrest or calibrated force limiting; inertia and existing target rate limits can continue motion.', + 'comparison_scope': 'Closed-loop variants may acquire different trajectories through early hold/retry. This is not an isolated classifier ablation.', + } + + +def _number(value, name, low=-1e6, high=1e6): + if type(value) not in (int, float): + raise ValueError(name+' must be a finite real scalar') + try: + number = float(value) + except (ValueError, TypeError, OverflowError) as exc: + raise ValueError(name+' must be finite') from exc + if not math.isfinite(number) or not low <= number <= high: + raise ValueError(name+' outside bounded range') + return number + + +def _vector(value, name, length, low=-1e3, high=1e3): + if type(value) not in (list, tuple) or len(value) != length: + raise ValueError(name+' has the wrong shape') + return [_number(item, name, low, high) for item in value] + + +def _observation(value): + if type(value) is not dict: + raise ValueError('Probe observation must be a dictionary') + # MuJoCo's repeated 1 ms additions can end at 20.00000000000146. + # Match the episode-clock tolerance without admitting another native tick. + _number(value.get('t_seconds'), 't_seconds', 0., 20.+1e-8) + for name, width in (('joint_position_rad', 6), ('tcp_position_m', 3), + ('object_position_m', 3), ('target_position_m', 3), + ('gripper_joint_position_m', 2)): + _vector(value.get(name), name, width) + quaternion = _vector(value.get('tcp_quaternion_wxyz'), 'tcp_quaternion_wxyz', 4, -1., 1.) + if not math.isclose(sum(item*item for item in quaternion), 1., rel_tol=0., abs_tol=1e-8): + raise ValueError('Probe TCP quaternion must be normalized wxyz') + for name in ('table_object_contact_count', 'unexpected_contact_count'): + if type(value.get(name)) is not int or not 0 <= value[name] <= 10000: + raise ValueError(name+' must be a nonnegative integer') + counts, forces = value.get('grasp_contact_counts'), value.get('pad_normal_forces_n') + if type(counts) is not dict or type(forces) is not dict or set(counts) != {'left', 'right'} or set(forces) != set(counts): + raise ValueError('Probe requires explicit left/right contact and force readings') + for side in counts: + if type(counts[side]) is not int or not 0 <= counts[side] <= 10000: + raise ValueError('Contact counts must be nonnegative integers') + _number(forces[side], 'pad force', 0., 1e6) + + +class ProbeStopPolicy(guard.GuardedPiperTaskPolicy): + """Reuse original acquisition while preventing any full-lift dispatch.""" + + def __init__(self, environment, variant='evidence'): + if type(variant) is not str or variant not in VARIANTS: + raise ValueError('Probe variant must be evidence or timer') + control_dt = _number(environment.config.control_dt, 'control_dt', DT, DT) + timeout = _number(environment.config.timeout_s, 'timeout_s', 1., 20.) + if not math.isclose(timeout/control_dt, round(timeout/control_dt), rel_tol=0., abs_tol=1e-8): + raise ValueError('Probe timeout must contain complete control boundaries') + super().__init__(environment) + self.variant, self.status = variant, 'acquiring' + self.timeout_s, self.command_budget = timeout, round(timeout/control_dt) + self.admitted, self.ever_admitted, self.admission_time_s = False, False, None + self.last_requested, self.frozen_jaw_target = None, None + self.initial_context, self.probes, self.active_probe_index = None, [], None + self.requested_controls, self.qualification_checks = [], [] + self.last_observed_s, self.final_observed = None, False + + def _clock(self, observation): + _observation(observation) + now = float(observation['t_seconds']) + expected = 0. if self.last_observed_s is None else self.last_observed_s+DT + if not math.isclose(now, expected, rel_tol=0., abs_tol=1e-8): + raise ValueError('Probe requires causal consecutive 20 ms observations starting at zero') + if now > self.timeout_s+1e-8: + raise ValueError('Probe observation exceeds its fixed episode budget') + self.last_observed_s = now + if self.initial_context is None: + self.initial_context = {'t_seconds': now, 'object_position_m': list(observation['object_position_m']), + 'tcp_position_m': list(observation['tcp_position_m']), + 'tcp_quaternion_wxyz': list(observation['tcp_quaternion_wxyz']), + 'observation_origin': ORIGIN} + + def _jaw_to_freeze(self, observation): + if self.last_requested is not None: + return self.last_requested['gripper_position_m'] + controls = observation.get('native_actuator_controls') + if controls is not None: + controls = _vector(controls, 'native_actuator_controls', 7) + return _number(controls[6], 'jaw control', 0., .035) + # At the first observation no policy target exists yet. Use the explicit + # native single-finger coordinate, not a synthesized full jaw gap. + return _number(observation['gripper_joint_position_m'][0], 'initial jaw coordinate', 0., .035) + + def _hold(self, jaw=None): + if self.stop_targets is None: + raise ValueError('Measured stop targets have not been latched') + if jaw is None: + jaw = (self.frozen_jaw_target if self.frozen_jaw_target is not None else + self.last_requested['gripper_position_m'] if self.last_requested is not None else .035) + return {'joint_targets_rad': list(self.stop_targets), 'gripper_position_m': jaw} + + def _event(self, kind, observation, reason): + if kind == 'probe_started': + self.active_probe_index = len(self.probes) + self.probes.append({ + 'probe_index': self.active_probe_index, 'start_time_s': observation['t_seconds'], + 'object_origin_m': list(self.probe_origin['object']), 'tcp_origin_m': list(self.probe_origin['tcp']), + 'tcp_quaternion_wxyz': list(self.probe_quaternion), 'local_offset_origin_m': list(self.anchor), + 'observation_origin': ORIGIN, 'end_time_s': None, 'disposition': 'probing', 'reason': None, + }) + if kind == 'alarm' and self.active_probe_index is not None: + self.probes[self.active_probe_index].update(end_time_s=observation['t_seconds'], + disposition='interrupted', reason=reason) + if kind == 'retry_started': self.status = 'acquiring_retry' + if kind == 'retry_completed': self.status = 'acquiring' + super()._event(kind, observation, reason) + self.events[-1].update(variant=self.variant, status=self.status, + probe_index=self.active_probe_index, observation_origin=ORIGIN) + if kind == 'retry_completed': self.active_probe_index = None + + def _terminal_hold(self, observation, reason, *, timeout=False, event='hold_invalidated'): + self.stop_targets = list(observation['joint_position_rad']) + self.frozen_jaw_target = self._jaw_to_freeze(observation) + self.state, self.issued_phase = 'abort', 'probe_stop' + self.status = 'timed_out_hold' if timeout else 'invalidated_hold' + self.admitted, self.qualified, self.aborted, self.abort_reason = False, False, True, reason + self.diagnostic.update(decision='hold_measured_arm_and_freeze_issued_jaw', qualified=False) + if self.active_probe_index is not None: + self.probes[self.active_probe_index].update(end_time_s=observation['t_seconds'], + disposition=self.status, reason=reason) + self._event(event, observation, reason) + return self._hold() + + def _abort(self, observation, reason): + return self._terminal_hold(observation, reason, event='abort') + + def _eligible(self, observation): + if self.probe_origin is None: + return False + return (all(observation['grasp_contact_counts'].values()) + and min(observation['pad_normal_forces_n'].values()) > .01 + and observation['table_object_contact_count'] == 0 + and observation['unexpected_contact_count'] == 0 + and self.diagnostic['object_height_delta_m'] >= self.config.probe_min_object_rise_m + and self.diagnostic['relative_offset_error_m'] <= self.config.relative_drift_limit_m) + + def _admit(self, observation, kind): + self.stop_targets = list(observation['joint_position_rad']) + self.frozen_jaw_target = self._jaw_to_freeze(observation) + self.state, self.issued_phase = 'probe_hold', 'probe_stop' + self.status = 'qualified_hold' if self.variant == 'evidence' else 'timer_hold' + self.admitted, self.ever_admitted, self.qualified = True, True, True + self.admission_time_s = observation['t_seconds'] + self.qualification_count += 1 + self.diagnostic.update(decision='hold_measured_arm_and_freeze_issued_jaw', qualified=True) + self.probes[self.active_probe_index].update(disposition=self.status, + admission_time_s=self.admission_time_s) + self._event(kind, observation, 'sampled_probe_evidence' if self.variant == 'evidence' else 'fixed_probe_clock_only') + return self._hold() + + def _probe(self, observation): + self.status, self.issued_phase = 'probing', 'guard_probe' + if self.probe_started is None: + self.probe_started = observation['t_seconds'] + self.probe_origin = {'object': list(observation['object_position_m']), 'tcp': list(observation['tcp_position_m'])} + self.probe_quaternion = list(observation['tcp_quaternion_wxyz']) + self.anchor = guard._local_offset(observation) + self.diagnostic = self._diagnose(observation) + self._event('probe_started', observation, 'slow_load_bearing_check') + elapsed = observation['t_seconds']-self.probe_started + if self.variant == 'evidence': + stopped = self._monitor(observation, probe=True) + if stopped is not None: return stopped + eligible = self._eligible(observation) + self.qualification_since = ((observation['t_seconds'] if self.qualification_since is None else self.qualification_since) + if eligible else None) + dwell = 0. if self.qualification_since is None else observation['t_seconds']-self.qualification_since + self.diagnostic['qualification_dwell_seconds'] = dwell + if dwell+1e-9 >= self.config.qualification_dwell_s: + return self._admit(observation, 'evidence_qualified') + elif elapsed+1e-9 >= TIMER_ADMISSION_S: + return self._admit(observation, 'timer_admitted') + if elapsed > self.config.probe_timeout_s: + return self._alarm(observation, 'probe_timeout') + goal = list(self.probe_origin['tcp']) + goal[2] += min(self.config.probe_height_m, + self.config.probe_speed_m_s*(elapsed+self.env.config.control_dt)) + return self.env.pose_action(goal, self.probe_quaternion, 0.) + + def _hold_check(self, observation): + if not self.admitted: + return self._hold() + if not self._eligible(observation): + reason = ('unexpected_contact' if observation['unexpected_contact_count'] else + 'bilateral_contact_or_force_loss' if not all(observation['grasp_contact_counts'].values()) or min(observation['pad_normal_forces_n'].values()) <= .01 else + 'support_returned' if observation['table_object_contact_count'] else + 'relative_slip' if self.diagnostic['relative_offset_error_m'] > self.config.relative_drift_limit_m else + 'probe_height_not_retained') + return self._terminal_hold(observation, reason) + self.issued_phase = 'probe_stop' + self.diagnostic.update(decision='maintain_latched_arm_and_jaw', qualified=True) + return self._hold() + + def _claim(self, observation): + if self.active_probe_index is not None: + self.qualification_checks.append({'t_seconds': observation['t_seconds'], 'qualified': self.admitted, + 'probe_index': self.active_probe_index}) + self.diagnostic.update(variant=self.variant, status=self.status, admitted=self.admitted, + probe_index=self.active_probe_index, observation_origin=ORIGIN) + + def __call__(self, observation): + if self.final_observed or len(self.requested_controls) >= self.command_budget: + raise ValueError('Probe command budget exhausted or final observation already sealed') + self._clock(observation) + self.diagnostic_count += 1 + self.diagnostic = self._diagnose(observation) + if self.status in ('invalidated_hold', 'timed_out_hold'): + action = self._hold() + elif observation['unexpected_contact_count']: + action = self._terminal_hold(observation, 'unexpected_contact') + elif self.admitted: + action = self._hold_check(observation) + elif observation['t_seconds']+DT >= self.timeout_s-1e-9: + action = self._terminal_hold(observation, 'episode_acquisition_timeout', timeout=True) + elif self.state == 'probe': + action = self._probe(observation) + else: + if self.state == 'normal' and self.teacher.phase not in ('approach', 'descend', 'close'): + action = self._terminal_hold(observation, 'forbidden_full_lift_dispatch') + else: + action = super().__call__(observation) + if self.state == 'probe': self.status = 'probe_pending' + _vector(action.get('joint_targets_rad'), 'requested joint targets', 6) + _number(action.get('gripper_position_m'), 'requested jaw target', 0., .035) + self._claim(observation) + self.last_requested = deepcopy(action) + self.requested_controls.append({'index': len(self.requested_controls), 't_seconds': observation['t_seconds'], + 'phase': self.issued_phase, 'status': self.status, + 'probe_index': self.active_probe_index, **deepcopy(action)}) + return deepcopy(action) + + def observe_final(self, observation): + """Check the last recorded state without emitting another control vector.""" + if self.final_observed: raise ValueError('Final observation is already sealed') + self._clock(observation) + self.diagnostic_count += 1 + self.diagnostic = self._diagnose(observation) + if self.admitted: self._hold_check(observation) + elif self.status not in ('invalidated_hold', 'timed_out_hold'): + self._terminal_hold(observation, 'episode_ended_without_admission', timeout=True) + self._claim(observation) + self.final_observed = True + return self.summary() + + def summary(self): + return deepcopy({'schema': SCHEMA, 'variant': self.variant, 'status': self.status, + 'observation_origin': ORIGIN, 'physical_robot_ready': False, + 'contract': probe_contract(), 'admitted': self.admitted, + 'ever_admitted': self.ever_admitted, 'admission_time_s': self.admission_time_s, + 'initial_context': self.initial_context, 'probes': self.probes, + 'active_probe_index': self.active_probe_index, + 'events': self.events, 'requested_controls': self.requested_controls, + 'qualification_checks': self.qualification_checks, + 'stop_targets_rad': self.stop_targets, 'frozen_jaw_target_m': self.frozen_jaw_target, + 'retry_count': self.retry_count, 'abort_reason': self.abort_reason, + 'last_observed_s': self.last_observed_s, 'final_observed': self.final_observed}) diff --git a/src/dvidia_training/piper_probe_benchmark.py b/src/dvidia_training/piper_probe_benchmark.py new file mode 100644 index 0000000..311735f --- /dev/null +++ b/src/dvidia_training/piper_probe_benchmark.py @@ -0,0 +1,371 @@ +"""Frozen local PiPER short-lift study; authored control and simulator evidence. + +No learning, physical sensing, force calibration or hardware commands occur. +The two closed-loop controllers may produce different trajectories: this is not +an isolated classifier comparison. All scheduled attempts remain in the report. +""" +from __future__ import annotations + +import argparse +from dataclasses import asdict, replace +from datetime import datetime, timezone +from hashlib import sha256 +import gzip +import io +import json +from pathlib import Path +import platform +import sys +import time +import zipfile + +import numpy as np + +SCHEMA = 'dvidia.piper-probe-study/v1' +VARIANTS = ('evidence', 'timer') +SCOPE = ('Authored short-lift/hold controllers with privileged native simulator state. ' + 'No full placement, learned policy, camera tracking, calibrated physical sensing or hardware qualification.') + + +def encoded(value): + return (json.dumps(value, sort_keys=True, indent=2, allow_nan=False)+'\n').encode() + + +def digest(value): + return sha256(encoded(value)).hexdigest() + + +def write(path, value): + with Path(path).open('xb') as stream: + stream.write(encoded(value)) + + +def read(path, limit=16*1024*1024): + path = Path(path) + if path.is_symlink() or not path.is_file() or path.stat().st_size > limit: + raise ValueError('Expected a bounded plain evidence file.') + with path.open('rb') as stream: + raw = stream.read(limit+1) + if len(raw) > limit: + raise ValueError('Evidence exceeds its byte limit.') + return json.loads(raw) + + +def sources(): + root = Path(__file__).parent + return {p.name: sha256(p.read_bytes()).hexdigest() + for p in sorted([*root.glob('*.py'), *root.glob('*.lock.json')])} + + +def runtime(): + import mujoco + return dict(python=platform.python_version(), numpy=np.__version__, mujoco=mujoco.__version__, + system=platform.system(), machine=platform.machine(), + executable_sha256=sha256(Path(sys.executable).resolve().read_bytes()).hexdigest()) + + +def make_protocol(assets): + from .piper_task import PiperTaskConfig + from .robot_profiles import verify_assets, LOCK_PATH + from .piper_probe import probe_contract + provenance = verify_assets('piper', assets) + base, cases = PiperTaskConfig(), [] + for mass in (.04, .06): + for friction in (.8, .55, .15): + for index, start in enumerate(((.30, -.03), (.309, -.04))): + config = replace(base, mass_kg=mass, friction=friction, + object_position_m=(*start, .082)) + cases.append(dict(id=f'm{round(mass*1000):02}-f{round(friction*100):02}-p{index}', + seed=311000+len(cases), config=asdict(config))) + protocol = dict(schema=SCHEMA, scope=SCOPE, physical_robot_ready=False, learned_policy=False, + cohort='development_method_check', cases=cases, variants=list(VARIANTS), + planned_attempts=24, source_sha256=sources(), runtime=runtime(), + asset_provenance=provenance, asset_lock_sha256=sha256(LOCK_PATH.read_bytes()).hexdigest(), + controller_contract=probe_contract(), + evidence='Independent reconstruction from every post-physics 1 ms native row; continuous 100 ms grip, not a caller passed flag.', + force_ceiling_n=10., force_ceiling_scope='Arbitrary synthetic admission ceiling; not a measured damage limit or force controller.', + comparison='Same scene and authored acquisition prefix, followed by two closed-loop probe/hold controllers. ' + 'Early holds, alarms and supported retries can change the trajectories. Each admission is judged ' + 'against its own native history, not as an isolated causal effect of a classifier.', + timing='Policy-call and complete control-step wall/CPU compute, including dense recording for the step. ' + 'Unpaced shared CPU; excludes physical sensors, transport, hardware actuation and evidence serialization.', + selection='No parameter selection after these development attempts. Each scene runs once per variant; ' + 'deterministic cells are not independent hardware trials or a held-out learning test.') + return protocol + + +def prepare(root, assets): + root = Path(root) + if root.exists() or root.is_symlink(): + raise ValueError('Choose a fresh output directory; existing evidence is never overwritten.') + protocol = make_protocol(assets) + root.mkdir(parents=True) + write(root/'protocol.json', protocol) + write(root/'frozen.json', {'created_at_utc':datetime.now(timezone.utc).isoformat(), + 'protocol_sha256':sha256((root/'protocol.json').read_bytes()).hexdigest()}) + (root/'sources').mkdir() + for name in ('piper_probe.py', 'probe_evidence.py', 'piper_probe_benchmark.py'): + (root/'sources'/name).write_bytes(Path(__file__).with_name(name).read_bytes()) + return protocol + + +def _timing(values): + if not values: + return None + return dict(samples=len(values), mean_s=float(np.mean(values)), p50_s=float(np.percentile(values, 50)), + p95_s=float(np.percentile(values, 95)), maximum_s=float(max(values)), + over_20ms=sum(x > .02 for x in values)) + + +def _feasibility(case): + from .task_feasibility import template + value = template() + value.update(object_width_interval_m=[.04, .04], gripper_max_opening_interval_m=[.07, .07], + gripper_min_closure_m=0., mass_interval_kg=[case['config']['mass_kg']]*2, + friction_interval=[case['config']['friction']]*2, + per_finger_normal_force_budget_n=10., pose_specific_payload_limit_kg=.10, + planned_upward_accel_m_s2=.1, safety_factor=1.5, + pose_uncertainty_m=0., allowed_pose_tolerance_m=.003, + calibration_id='synthetic_probe_v01', test_lift_passed=False) + return value + + +def _context(result, protocol, probe, planning): + from .probe_evidence import feasibility_binding + info = result['metrics'] + return dict(schema='dvidia.probe-context/v1', run_id=result['id'], object_id='task_block', + grasp_id=result['id']+'-probe-'+str(probe['probe_index']), probe_index=probe['probe_index'], robot_profile='piper', + model_sha256=result['compiled_model_sha256'], source_sha256=digest(protocol['source_sha256']), + asset_lock_sha256=protocol['asset_lock_sha256'], command_sha256=digest(result['actions']), + pose_binding_sha256=digest({'config':result['config'], 'probe':probe}), + feasibility_sha256=feasibility_binding(planning), calibration_id='synthetic_probe_v01', + calibration_scope='simulation_assumptions_not_hardware', + initial_object_position_m=result['initial_object_position_m'], probe_start_s=probe['start_time_s'], + probe_origin_object_position_m=probe['object_origin_m'], + probe_origin_tcp_local_offset_m=probe['local_offset_origin_m'], + per_finger_force_ceiling_n=planning['per_finger_normal_force_budget_n'], + native_status={'valid':info['valid'], 'warnings':info['warnings'], + 'unexpected_contact_native_ticks':info['unexpected_contact_native_ticks'], + 'simulation_time_s':info['simulation_time']}) + + +def _attempt(assets, case, variant, protocol): + from .piper_task import PiperTaskConfig + from .piper_feedback_env import FeedbackPiperTaskEnv + from .piper_probe import ProbeStopPolicy + from .reference_sim import model_digest + env = FeedbackPiperTaskEnv(assets, PiperTaskConfig(**case['config'])) + result = dict(schema=SCHEMA, id=case['id']+'-'+variant, scene_id=case['id'], variant=variant, + config=case['config'], valid=False, physical_robot_ready=False, + trajectory=[], actions=[], error=None, metrics={}, policy=None) + timers = {name:[] for name in ('policy_wall', 'policy_cpu', 'step_wall', 'step_cpu')} + began = time.perf_counter() + try: + obs, info = env.reset(case['seed']) + result.update(initial_object_position_m=env.initial_object_position.tolist(), + compiled_model_sha256=model_digest(env.model), + native_force_limits=env.model.actuator_forcerange.tolist()) + policy = ProbeStopPolicy(env, variant=variant) + result['trajectory'].append(obs) + for _ in range(round(env.config.timeout_s/env.config.control_dt)): + wall, cpu = time.perf_counter(), time.process_time() + action = policy(obs) + timers['policy_wall'].append(time.perf_counter()-wall) + timers['policy_cpu'].append(time.process_time()-cpu) + record = dict(t_seconds=obs['t_seconds'], **action) + result['actions'].append(record) + obs['command'] = record + wall, cpu = time.perf_counter(), time.process_time() + obs, info = env.step(action) + timers['step_wall'].append(time.perf_counter()-wall) + timers['step_cpu'].append(time.process_time()-cpu) + if obs is not None: + result['trajectory'].append(obs) + if env.done: + break + if obs is not None: + policy.observe_final(obs) + result.update(valid=info['valid'], metrics=info, policy=policy.summary()) + except Exception as exc: + result['error'] = {'type':type(exc).__name__, 'message':str(exc)[:300]} + if env.ready: + result['metrics'] = env.info() + result['timing'] = dict(attempt_wall_s=time.perf_counter()-began, + summaries={k:_timing(v) for k,v in timers.items()}, samples=timers) + return result, env.native_arrays(), env.native_metadata() + + +def _evaluate(result, arrays, metadata, protocol): + from .probe_evidence import derive_probe_evidence, verify_probe_evidence, assess_with_probe + planning = _feasibility({'config':result['config']}) + probes = (result.get('policy') or {}).get('probes', []) + if not result['valid'] or not probes: + return dict(status='not_qualified', reason='invalid_attempt' if not result['valid'] else 'no_probe', + receipt=None, planning_input=planning, physical_robot_ready=False) + probe = probes[-1] + context = _context(result, protocol, probe, planning) + claims = [dict(t_seconds=e['t_seconds'], qualified=True, probe_index=e['probe_index']) + for e in result['policy']['events'] + if e['type'] in ('evidence_qualified', 'timer_admitted') and e['probe_index']==probe['probe_index']] + # A historical admission cannot resurrect a controller's terminal refusal. + terminal = result['policy']['qualification_checks'][-1] + if terminal['probe_index'] == probe['probe_index']: + if claims and abs(claims[-1]['t_seconds']-terminal['t_seconds']) < 1e-8: + claims[-1] = terminal + else: + claims.append(terminal) + receipt = derive_probe_evidence(arrays, metadata, context, controller_qualifications=claims) + now = result['metrics']['simulation_time'] + verification = verify_probe_evidence(receipt, arrays, metadata, context, now_s=now) + admission = assess_with_probe(planning, receipt, arrays, metadata, context, now_s=now) + return dict(status='inspected', planning_input=planning, context=context, receipt=receipt, + verification=verification, admission=admission, physical_robot_ready=False) + + +def inspect(root): + """Re-read seals, applied slew-limited commands and probe receipts; no physics.""" + from .piper_grip_benchmark import decode_attempt + from .probe_evidence import _arrays + root = Path(root) + protocol, report = read(root/'protocol.json'), read(root/'report.json') + if protocol['source_sha256'] != sources() or report['source_sha256'] != protocol['source_sha256']: + raise ValueError('Inspect with the exact frozen study implementation.') + roster = [(case, variant) for case in protocol['cases'] for variant in protocol['variants']] + ids = [case['id']+'-'+variant for case, variant in roster] + if ([r['id'] for r in report['rows']] != ids or len(ids) != protocol['planned_attempts'] + or report['attempted'] != len(ids)): + raise ValueError('Recorded attempt denominator/order differs from the protocol.') + checked = [] + for row, (case, variant) in zip(report['rows'], roster): + folder = root/row['id'] + if set(row['files']) != {'started.json', 'episode.json.gz', 'native.npz', 'native.json', 'evaluation.json'}: + raise ValueError('Unexpected attempt evidence roster.') + for name, expected in row['files'].items(): + path = folder/name + if path.is_symlink() or not path.is_file() or path.stat().st_size > 32*1024*1024: + raise ValueError('Invalid bounded evidence file.') + raw = path.read_bytes() + if {'bytes':len(raw),'sha256':sha256(raw).hexdigest()} != expected: + raise ValueError('Recorded evidence digest changed: '+row['id']+'/'+name) + episode = json.loads(decode_attempt((folder/'episode.json.gz').read_bytes())) + if (episode['id'] != row['id'] or episode['config'] != case['config'] or episode['variant'] != variant + or episode['valid'] != row['valid'] or episode['metrics'] != row['metrics']): + raise ValueError('Episode and protocol/report identity differ.') + with zipfile.ZipFile(folder/'native.npz') as archive: + entries = archive.infolist() + if (len({p.filename for p in entries}) != len(entries) or len(entries) != 22 + or sum(p.file_size for p in entries) > 32*1024*1024 + or any(p.file_size > 2*1024*1024 or '/' in p.filename or not p.filename.endswith('.npy') for p in entries)): + raise ValueError('Native NPZ archive exceeds the numeric-only contract.') + with np.load(folder/'native.npz', allow_pickle=False) as native: + arrays = {name:native[name] for name in native.files} + metadata = read(folder/'native.json') + _arrays(arrays, metadata) # Also validate refused/no-probe native trajectories. + expected_evaluation = _evaluate(episode, arrays, metadata, protocol) + if (encoded(expected_evaluation) != encoded(row['evaluation']) + or encoded(read(folder/'evaluation.json')) != encoded(row['evaluation'])): + raise ValueError('Recomputed native probe evidence differs: '+row['id']) + actions, trajectory = episode['actions'], episode['trajectory'] + count = len(actions) + if (not episode['valid'] or count != 1000 or len(trajectory) != count+1 + or len(arrays['t_seconds']) != count*20): + raise ValueError('Incomplete/invalid native attempt retained; it cannot qualify.') + requested = episode['policy']['requested_controls'] + if len(requested) != count: + raise ValueError('Missing requested-control records.') + control = np.array(trajectory[0]['native_actuator_controls'], dtype=float) + step = np.array([episode['config']['joint_target_rate_rad_s']*.001]*6 + +[episode['config']['jaw_target_rate_m_s']*.001]) + for i, (action, request) in enumerate(zip(actions, requested)): + if (abs(action['t_seconds']-i*.02) > 1e-8 or request['index'] != i + or any(request[k] != action[k] for k in ('t_seconds','joint_targets_rad','gripper_position_m'))): + raise ValueError('Issued commands differ from the policy records/cadence.') + target = np.array(action['joint_targets_rad']+[action['gripper_position_m']]) + for tick in range(i*20, (i+1)*20): + control += np.clip(target-control, -step, step) + actual = np.r_[arrays['arm_actuator_controls_rad'][tick], arrays['applied_jaw_target_m'][tick]] + if (not np.allclose(control, actual, atol=1e-12, rtol=0) + or arrays['requested_jaw_target_m'][tick] != action['gripper_position_m']): + raise ValueError('Applied native controls differ from the issued tape and fixed slew law.') + for field in ('object_position_m', 'joint_position_rad', 'tcp_position_m'): + if not np.allclose(trajectory[i+1][field], arrays[field][(i+1)*20-1], atol=1e-9, rtol=0): + raise ValueError('Boundary observation differs from native evidence.') + checked.append(dict(id=row['id'], native_rows=len(arrays['t_seconds']), commands=count, + evaluation_recomputed=True, applied_controls_recomputed=True)) + return dict(schema='dvidia.piper-probe-inspection/v1', attempts=len(checked), + native_rows=sum(r['native_rows'] for r in checked), rows=checked, + scope='Recorded consistency and predicate reconstruction; not independent physics or authenticated sensors.', + physical_robot_ready=False) + + +def run(root, assets): + root = Path(root) + protocol = read(root/'protocol.json') + if encoded(protocol) != encoded(make_protocol(assets)): + raise ValueError('Frozen protocol/source/runtime/assets no longer match; create a new study.') + # One launch per study; a failed run remains evidence, never an implicit resume. + write(root/'started.json', {'protocol_sha256':sha256((root/'protocol.json').read_bytes()).hexdigest(), + 'started_at_utc':datetime.now(timezone.utc).isoformat()}) + rows = [] + for case in protocol['cases']: + for variant in protocol['variants']: + if sources() != protocol['source_sha256']: + raise ValueError('Source changed during the frozen study.') + folder = root/(case['id']+'-'+variant); folder.mkdir() + write(folder/'started.json', {'scene_id':case['id'], 'variant':variant, + 'protocol_sha256':sha256((root/'protocol.json').read_bytes()).hexdigest()}) + result, arrays, metadata = _attempt(assets, case, variant, protocol) + raw = encoded(result) + (folder/'episode.json.gz').write_bytes(gzip.compress(raw, mtime=0)) + buffer = io.BytesIO(); np.savez_compressed(buffer, **arrays) + (folder/'native.npz').write_bytes(buffer.getvalue()) + write(folder/'native.json', metadata) + try: + evaluation = _evaluate(result, arrays, metadata, protocol) + except Exception as exc: + evaluation = dict(status='audit_refused', error_type=type(exc).__name__, reason=str(exc)[:300], + physical_robot_ready=False) + write(folder/'evaluation.json', evaluation) + files = {p.name:{'bytes':p.stat().st_size,'sha256':sha256(p.read_bytes()).hexdigest()} for p in sorted(folder.iterdir())} + row = dict(id=result['id'], scene_id=case['id'], variant=variant, + mass_kg=case['config']['mass_kg'], friction=case['config']['friction'], + valid=result['valid'], error=result['error'], metrics=result['metrics'], + policy={k:v for k,v in (result.get('policy') or {}).items() if k != 'requested_controls'}, + timing=result['timing']['summaries'], evaluation=evaluation, files=files) + rows.append(row) + write(folder/'summary.json', row) + print(json.dumps({'attempt':result['id'],'valid':result['valid'], 'audit':evaluation['status']}), flush=True) + if sources() != protocol['source_sha256'] or runtime() != protocol['runtime']: + raise ValueError('Frozen source/runtime changed during the study.') + report = dict(schema=SCHEMA, scope=SCOPE, planned_attempts=protocol['planned_attempts'], attempted=len(rows), + valid_attempts=sum(r['valid'] for r in rows), rows=rows, + previous_learned_heldout_result='5/12 unchanged; this study contains no training or full-placement evaluation.', + physical_robot_ready=False, source_sha256=protocol['source_sha256']) + write(root/'report.json', report) + return report + + +def main(argv=None): + parser = argparse.ArgumentParser(description=__doc__) + sub = parser.add_subparsers(dest='command', required=True) + for name in ('prepare', 'run'): + command = sub.add_parser(name) + command.add_argument('--output', type=Path, required=True) + command.add_argument('--assets', type=Path, required=True) + command = sub.add_parser('inspect') + command.add_argument('--output', type=Path, required=True) + args = parser.parse_args(argv) + try: + value = (inspect(args.output) if args.command == 'inspect' + else globals()[args.command](args.output, args.assets)) + print(json.dumps(value if args.command == 'inspect' else + {'schema':SCHEMA,'planned_attempts':value['planned_attempts'], + 'physical_robot_ready':False},indent=2)) + return 0 + except (ValueError, OSError, KeyError, ImportError) as exc: + print('Probe study refused: '+str(exc), file=sys.stderr) + return 2 + + +if __name__ == '__main__': + raise SystemExit(main()) diff --git a/src/dvidia_training/probe_evidence.py b/src/dvidia_training/probe_evidence.py new file mode 100644 index 0000000..ef53459 --- /dev/null +++ b/src/dvidia_training/probe_evidence.py @@ -0,0 +1,370 @@ +"""Bounded post-run evidence for one simulation-only PiPER probe. + +Native sidecars are privileged simulator records, not physical sensor evidence. +Hashes detect corruption and bind context; they do not authenticate an external +claim. The unchanged feasibility v1 gate retains every unknown load/friction/ +force/payload assumption. A successful probe never estimates those quantities. +""" +from __future__ import annotations + +from copy import deepcopy +from hashlib import sha256 +import json +import math +import re + +import numpy as np + +from . import task_feasibility + +SCHEMA = 'dvidia.probe-evidence/v1' +CONTEXT_SCHEMA = 'dvidia.probe-context/v1' +MAX_ROWS = 30000 +MAX_JSON_BYTES = 256*1024 +DT_S = .001 +MIN_FORCE_N = .01 +MIN_RISE_M = .006 +MAX_DRIFT_M = .003 +MIN_DWELL_S = .1 +CRITERIA = {'native_dt_s': DT_S, 'minimum_per_finger_force_n_exclusive': MIN_FORCE_N, + 'minimum_rise_m': MIN_RISE_M, 'maximum_tcp_local_drift_m': MAX_DRIFT_M, + 'minimum_continuous_dwell_s': MIN_DWELL_S, + 'distance_comparison_epsilon': 1e-12, 'origin_tolerance_m': 1e-9, + 'initial_sample_tolerance_m': .001, 'time_tolerance_s': 1e-8} +FIELDS = (('t_seconds', 1), ('native_tick', 1), ('object_position_m', 3), + ('object_quaternion_wxyz', 4), ('object_linear_velocity_m_s', 3), + ('object_angular_velocity_rad_s', 3), ('tcp_position_m', 3), + ('tcp_quaternion_wxyz', 4), ('object_minus_tcp_local_m', 3), + ('grasp_contact_counts', 2), ('pad_normal_forces_n', 2), + ('table_object_contact_count', 1), ('unexpected_contact_count', 1), + ('gripper_joint_position_m', 2), ('joint_position_rad', 6), + ('arm_actuator_controls_rad', 6), ('requested_jaw_target_m', 1), + ('normal_rate_limited_jaw_target_m', 1), ('applied_jaw_target_m', 1), + ('pulse_active', 1), ('pulse_minimum_opening_m', 1), ('pulse_closing_tail', 1)) +CONTEXT_FIELDS = {'schema', 'run_id', 'object_id', 'grasp_id', 'probe_index', 'robot_profile', + 'model_sha256', 'source_sha256', 'asset_lock_sha256', 'command_sha256', + 'pose_binding_sha256', 'feasibility_sha256', 'calibration_id', 'calibration_scope', + 'initial_object_position_m', 'probe_start_s', 'probe_origin_object_position_m', + 'probe_origin_tcp_local_offset_m', 'per_finger_force_ceiling_n', 'native_status'} +DIGEST_FIELDS = ('model_sha256', 'source_sha256', 'asset_lock_sha256', 'command_sha256', + 'pose_binding_sha256', 'feasibility_sha256') +SCOPE = ('Post-run privileged native simulation evidence for one bound probe. Synthetic assumptions ' + 'are not hardware calibration. Hashes detect corruption, not authenticity. No physical ' + 'actuation, load/friction inference or physical robot qualification.') + + +class ProbeEvidenceError(ValueError): + """Malformed, unbounded or internally inconsistent evidence input.""" + + +def _canonical(value): + try: + return json.dumps(value, sort_keys=True, separators=(',', ':'), allow_nan=False).encode() + except (TypeError, ValueError, OverflowError, RecursionError) as exc: + raise ProbeEvidenceError('Evidence must be bounded finite plain JSON.') from exc + + +def _digest(value): + return sha256(_canonical(value)).hexdigest() + + +def _number(value, name, low=0., high=30.): + if type(value) not in (int, float): + raise ProbeEvidenceError(name+' must be a finite number.') + try: + number = float(value) + except (ValueError, OverflowError) as exc: + raise ProbeEvidenceError(name+' must be finite.') from exc + if not math.isfinite(number) or not low <= number <= high: + raise ProbeEvidenceError(name+' is outside its bound.') + return number + + +def _vector(value, name): + if type(value) not in (list, tuple) or len(value) != 3: + raise ProbeEvidenceError(name+' must have three values.') + return [_number(item, name, -10., 10.) for item in value] + + +def _context(value): + if type(value) is not dict or set(value) != CONTEXT_FIELDS: + raise ProbeEvidenceError('Probe context fields differ from schema.') + if (value['schema'] != CONTEXT_SCHEMA or value['robot_profile'] != 'piper' + or value['calibration_scope'] != 'simulation_assumptions_not_hardware'): + raise ProbeEvidenceError('Unsupported probe/context scope.') + result = deepcopy(value) + for name in ('run_id', 'object_id', 'grasp_id'): + if (type(value[name]) is not str or not re.fullmatch(r'[A-Za-z0-9][A-Za-z0-9._:-]{0,159}', value[name])): + raise ProbeEvidenceError('Expected a bounded identifier: '+name) + if type(value['probe_index']) is not int or not 0 <= value['probe_index'] <= 8: + raise ProbeEvidenceError('probe_index must be an integer in [0,8].') + for name in DIGEST_FIELDS: + if type(value[name]) is not str or not re.fullmatch('[0-9a-f]{64}', value[name]): + raise ProbeEvidenceError('Expected lowercase SHA256: '+name) + calibration = value['calibration_id'] + if calibration is not None and (type(calibration) is not str + or not re.fullmatch(r'synthetic_[A-Za-z0-9._:-]{1,140}', calibration)): + raise ProbeEvidenceError('Calibration must be a synthetic_ identifier or unknown.') + for name in ('initial_object_position_m', 'probe_origin_object_position_m', + 'probe_origin_tcp_local_offset_m'): + result[name] = _vector(value[name], name) + result['probe_start_s'] = _number(value['probe_start_s'], 'probe_start_s', .001, 30.) + ceiling = value['per_finger_force_ceiling_n'] + result['per_finger_force_ceiling_n'] = None if ceiling is None else _number(ceiling, 'force ceiling', 0., 1e6) + status = value['native_status'] + if type(status) is not dict or set(status) != {'valid', 'warnings', 'unexpected_contact_native_ticks', 'simulation_time_s'}: + raise ProbeEvidenceError('Native status envelope fields differ.') + if (type(status['valid']) is not bool or type(status['warnings']) is not list + or len(status['warnings']) > 100 + or any(type(item) is not str or len(item) > 1000 for item in status['warnings']) + or type(status['unexpected_contact_native_ticks']) is not int + or not 0 <= status['unexpected_contact_native_ticks'] <= MAX_ROWS): + raise ProbeEvidenceError('Malformed native validity envelope.') + result['native_status']['simulation_time_s'] = _number(status['simulation_time_s'], 'simulation_time_s') + if len(_canonical(result)) > MAX_JSON_BYTES: + raise ProbeEvidenceError('Context exceeds its byte bound.') + return result + + +def feasibility_binding(document): + """Bind all planning assumptions; sensing flags are enforced separately.""" + task_feasibility.assess(document) + if type(document) is not dict: + raise ProbeEvidenceError('Feasibility binding requires a plain v1 dictionary.') + return _digest({name: value for name, value in document.items() + if name not in ('test_lift_passed', 'observations_fresh', 'slip_detected')}) + + +def _arrays(arrays, metadata): + if type(arrays) is not dict or set(arrays) != {name for name, _ in FIELDS}: + raise ProbeEvidenceError('Native sidecar fields differ from the fixed schema.') + if type(metadata) is not dict or len(_canonical(metadata)) > MAX_JSON_BYTES: + raise ProbeEvidenceError('Metadata must be a bounded plain dictionary.') + if (metadata.get('schema') != 'dvidia.piper-feedback-native/v1' + or metadata.get('native_timestep_s') != DT_S + or metadata.get('physical_robot_ready') is not False + or metadata.get('side_order') != ['left', 'right'] + or metadata.get('fields') != [{'name': name, 'width': width} for name, width in FIELDS]): + raise ProbeEvidenceError('Unsupported native telemetry contract.') + first = arrays['t_seconds'] + if not isinstance(first, np.ndarray) or first.ndim != 1 or not 1 <= first.size <= MAX_ROWS: + raise ProbeEvidenceError('Expected 1..30000 native samples.') + count = len(first) + if type(metadata.get('rows')) is not int or metadata['rows'] != count: + raise ProbeEvidenceError('Metadata row count differs.') + result = {} + digest = sha256() + for name, width in FIELDS: + value = arrays[name] + shape = (count,) if width == 1 else (count, width) + if (not isinstance(value, np.ndarray) or value.dtype != np.dtype('float64') + or value.shape != shape or not np.isfinite(value).all()): + raise ProbeEvidenceError('Expected finite float64 native array: '+name) + # Copy once: a caller edit during verification cannot alter later reads. + result[name] = np.array(value, dtype=np.float64, order='C', copy=True) + digest.update(_canonical({'name': name, 'shape': list(shape), 'dtype': 'float64'})) + digest.update(result[name].tobytes()) + ticks = np.arange(1, count+1, dtype=np.float64) + if (not np.array_equal(result['native_tick'], ticks) + or np.max(np.abs(result['t_seconds']-ticks*DT_S)) > 1e-8): + raise ProbeEvidenceError('Native cadence must be contiguous exact 1 ms ticks from reset.') + for name in ('grasp_contact_counts', 'table_object_contact_count', 'unexpected_contact_count'): + values = result[name] + if np.any(values < 0) or np.any(values > 10000) or np.any(values != np.floor(values)): + raise ProbeEvidenceError('Contact counts must be bounded nonnegative integers.') + if np.any(result['pad_normal_forces_n'] < 0) or np.any(result['pad_normal_forces_n'] > 1e6): + raise ProbeEvidenceError('Native pad forces exceed the supported range.') + for name in ('tcp_quaternion_wxyz', 'object_quaternion_wxyz'): + if np.max(np.abs(np.linalg.norm(result[name], axis=1)-1.)) > 1e-8: + raise ProbeEvidenceError('Native quaternion is not normalized.') + # Check the local-offset field independently against world positions/rotation. + q = result['tcp_quaternion_wxyz'] + vector = result['object_position_m']-result['tcp_position_m'] + xyz = -q[:, 1:] + cross = np.cross(xyz, vector) + local = vector+2*q[:, :1]*cross+2*np.cross(xyz, cross) + if np.max(np.abs(local-result['object_minus_tcp_local_m'])) > 1e-9: + raise ProbeEvidenceError('TCP-local offset disagrees with world pose.') + return result, digest.hexdigest() + + +def _claims(value, context, times, qualified): + if type(value) not in (list, tuple) or len(value) > 1500: + raise ProbeEvidenceError('Controller qualifications must be a bounded sequence.') + records, mismatches, prior = [], [], -1. + for row in value: + if type(row) is not dict or set(row) != {'t_seconds', 'qualified', 'probe_index'}: + raise ProbeEvidenceError('Controller qualification fields differ.') + timestamp = _number(row['t_seconds'], 'qualification time') + if (type(row['qualified']) is not bool or type(row['probe_index']) is not int + or row['probe_index'] != context['probe_index'] or timestamp <= prior): + raise ProbeEvidenceError('Qualification order, boolean or probe binding differs.') + tick = round(timestamp/DT_S) + if (not 1 <= tick <= len(times) or abs(times[tick-1]-timestamp) > 1e-8 + or timestamp+1e-8 < context['probe_start_s'] + or abs(timestamp/.02-round(timestamp/.02)) > 1e-8): + raise ProbeEvidenceError('Controller claims must use causal observed 20 ms boundaries.') + record = {'t_seconds': timestamp, 'qualified': row['qualified'], 'probe_index': row['probe_index']} + records.append(record) + if row['qualified'] != bool(qualified[tick-1]): + mismatches.append({'t_seconds': timestamp, 'controller_qualified': row['qualified'], + 'native_prefix_qualified': bool(qualified[tick-1]), + 'classification': ('false_qualification' if row['qualified'] + else 'conservative_abstention')}) + prior = timestamp + return records, mismatches + + +def derive_probe_evidence(arrays, metadata, context, *, controller_qualifications=()): + """Derive one probe receipt from the whole causal native prefix supplied. + + Supply only the final selected probe's claims. Missing probes have no valid + origins and should be retained as refused attempts without a receipt. + """ + context = _context(context) + data, array_hash = _arrays(arrays, metadata) + times = data['t_seconds'] + start_tick = round(context['probe_start_s']/DT_S) + if not 1 <= start_tick <= len(times) or abs(times[start_tick-1]-context['probe_start_s']) > 1e-8: + raise ProbeEvidenceError('Probe origin is not an observed native sample.') + origin = start_tick-1 + if (np.linalg.norm(data['object_position_m'][origin]-context['probe_origin_object_position_m']) > 1e-9 + or np.linalg.norm(data['object_minus_tcp_local_m'][origin]-context['probe_origin_tcp_local_offset_m']) > 1e-9): + raise ProbeEvidenceError('Probe origin object/local pose does not match its native row.') + if np.linalg.norm(data['object_position_m'][0]-context['initial_object_position_m']) > .001: + raise ProbeEvidenceError('Initial object binding differs from the first post-reset sample.') + if abs(context['native_status']['simulation_time_s']-times[-1]) > 1e-8: + raise ProbeEvidenceError('Native status time differs from latest actual sample.') + rise = data['object_position_m'][:, 2]-context['probe_origin_object_position_m'][2] + drift = np.linalg.norm(data['object_minus_tcp_local_m']-context['probe_origin_tcp_local_offset_m'], axis=1) + condition = ((np.arange(len(times)) > origin) + & np.all(data['grasp_contact_counts'] > 0, axis=1) + & np.all(data['pad_normal_forces_n'] > MIN_FORCE_N, axis=1) + & (data['table_object_contact_count'] == 0) + & (data['unexpected_contact_count'] == 0) + & (rise+1e-12 >= MIN_RISE_M) & (drift <= MAX_DRIFT_M+1e-12)) + streak = np.zeros(len(times), dtype=np.int64) + count = 0 + for index, good in enumerate(condition): + count = count+1 if good else 0 + streak[index] = count + qualified = streak >= round(MIN_DWELL_S/DT_S) + claims, mismatches = _claims(controller_qualifications, context, times, qualified) + reasons = [] + status = context['native_status'] + critical_ticks = int(np.count_nonzero(data['unexpected_contact_count'])) + if not status['valid']: + reasons.append('native_dynamics_invalid') + if status['warnings']: + reasons.append('native_solver_warnings') + if critical_ticks or status['unexpected_contact_native_ticks']: + reasons.append('unexpected_native_contact') + if critical_ticks != status['unexpected_contact_native_ticks']: + reasons.append('native_contact_total_disagrees') + ceiling = context['per_finger_force_ceiling_n'] + maximum_force = float(np.max(data['pad_normal_forces_n'])) + if ceiling is not None and maximum_force > ceiling: + reasons.append('declared_force_ceiling_exceeded') + false_qualifications = [row for row in mismatches if row['controller_qualified']] + abstentions = [row for row in mismatches if not row['controller_qualified']] + if false_qualifications: + reasons.append('controller_native_qualification_disagrees') + first = np.flatnonzero(qualified) + positive = [row for row in claims if row['qualified']] + receipt = {'schema': SCHEMA, 'scope': SCOPE, 'physical_robot_ready': False, + 'context': context, 'context_sha256': _digest(context), + 'native_arrays_sha256': array_hash, 'metadata_sha256': _digest(metadata), + 'criteria': dict(CRITERIA), 'controller_qualifications': claims, + 'audit': {'valid': not reasons, 'reasons': reasons, 'rows': len(times), + 'controller_claims_checked': len(claims), + 'controller_positive_claims': len(positive), + 'controller_claim_mismatches': mismatches, + 'controller_false_qualification_count': len(false_qualifications), + 'controller_conservative_abstention_count': len(abstentions), + 'controller_coverage': 'Only the supplied claims are checked; omitted decisions are not inferred.', + 'force_ceiling_known': ceiling is not None, 'maximum_observed_per_finger_force_n': maximum_force, + 'unexpected_contact_native_ticks': critical_ticks}, + 'history': {'ever_qualified': bool(first.size), + 'first_qualified_s': float(times[first[0]]) if first.size else None, + 'maximum_continuous_qualifying_s': int(np.max(streak))*DT_S}, + 'latest': {'time_s': float(times[-1]), 'native_tick': len(times), + 'condition_valid': bool(condition[-1]), 'qualified': bool(qualified[-1]), + 'continuous_qualifying_s': int(streak[-1])*DT_S, + 'rise_m': float(rise[-1]), 'tcp_local_drift_m': float(drift[-1]), + 'controller_claim': deepcopy(claims[-1]) if claims else None}} + receipt['receipt_sha256'] = _digest(receipt) + if len(_canonical(receipt)) > MAX_JSON_BYTES: + raise ProbeEvidenceError('Receipt exceeds its byte bound.') + return receipt + + +def verify_probe_evidence(receipt, arrays, metadata, current_context, *, now_s): + """Recompute proof and require the latest sample in the unchanged context.""" + verdict = {'schema': 'dvidia.probe-evidence-verification/v1', 'verified': False, + 'admissible': False, 'historical_qualified': False, 'latest_qualified': False, + 'physical_robot_ready': False, 'reasons': [], 'scope': SCOPE} + try: + current = _context(current_context) + now = _number(now_s, 'now_s') + if type(receipt) is not dict or len(_canonical(receipt)) > MAX_JSON_BYTES: + raise ProbeEvidenceError('Receipt must be a bounded plain dictionary.') + expected = derive_probe_evidence(arrays, metadata, receipt.get('context'), + controller_qualifications=receipt.get('controller_qualifications')) + if _canonical(receipt) != _canonical(expected): + verdict['reasons'].append('receipt_or_payload_changed') + return verdict + verdict.update(verified=True, historical_qualified=expected['history']['ever_qualified'], + latest_qualified=expected['latest']['qualified'], + receipt_sha256=expected['receipt_sha256']) + if _canonical(current) != _canonical(expected['context']): + verdict['reasons'].append('probe_context_changed') + latest = expected['latest']['time_s'] + if now < latest-1e-8: + verdict['reasons'].append('future_evidence') + elif now > latest+1e-8: + verdict['reasons'].append('stale_evidence') + verdict['reasons'].extend(expected['audit']['reasons']) + if not expected['latest']['qualified']: + verdict['reasons'].append('latest_native_window_not_qualified') + if not expected['audit']['controller_positive_claims']: + verdict['reasons'].append('no_controller_qualification_for_this_probe') + claim = expected['latest']['controller_claim'] + if claim is None or abs(claim['t_seconds']-latest) > 1e-8: + verdict['reasons'].append('latest_controller_claim_missing_or_stale') + elif not claim['qualified']: + # A previous admission cannot undo a later controller invalidation, + # even if native contact subsequently happens to settle again. + verdict['reasons'].append('latest_controller_not_qualified') + verdict['admissible'] = not verdict['reasons'] + except (ProbeEvidenceError, KeyError, TypeError, ValueError, OverflowError) as exc: + verdict['reasons'].append('malformed_evidence: '+str(exc)) + return verdict + + +def assess_with_probe(document, receipt, arrays, metadata, current_context, *, now_s): + """Replace the caller lift flag with recomputed, context-bound native proof. + + All unknown assumptions and caller stale/slip vetoes remain conservative. + The returned v2 wrapper does not change task_feasibility.py or its v1 API. + """ + task_feasibility.assess(document) + if type(document) is not dict: + raise ProbeEvidenceError('Integration requires a plain feasibility v1 dictionary.') + verdict = verify_probe_evidence(receipt, arrays, metadata, current_context, now_s=now_s) + current = current_context if type(current_context) is dict else {} + if current.get('feasibility_sha256') != feasibility_binding(document): + verdict['admissible'] = False + verdict['reasons'].append('planning_assumptions_changed') + if (document['calibration_id'] != current.get('calibration_id') + or document['per_finger_normal_force_budget_n'] != current.get('per_finger_force_ceiling_n')): + verdict['admissible'] = False + verdict['reasons'].append('calibration_or_force_ceiling_changed') + checked = deepcopy(document) + checked['test_lift_passed'] = verdict['admissible'] + gate = task_feasibility.assess(checked) + return {'schema': 'dvidia.task-feasibility-with-probe/v1', 'status': gate['status'], + 'physical_robot_ready': False, 'probe': verdict, 'planning': gate, + 'observation_origin': 'privileged_native_simulation_plus_caller_planning_assumptions', + 'scope': SCOPE+' Unknown planning values remain unknown. The unchanged approximate vertical-lift ' + 'planning gate describes declared loads; it supplies no transport path, geospatial-goal ' + 'contract or actuator permission. This wrapper never issues controls.'} diff --git a/tests/test_piper_probe.py b/tests/test_piper_probe.py new file mode 100644 index 0000000..c97eaf5 --- /dev/null +++ b/tests/test_piper_probe.py @@ -0,0 +1,233 @@ +"""Synthetic decision-boundary tests; no native simulation or hardware motion.""" +from copy import deepcopy +import math +from types import SimpleNamespace +import unittest +from unittest.mock import patch + +from dvidia_training import piper_probe as probe +from dvidia_training import piper_task + + +class Environment: + def __init__(self, jaw=.012): + self.config = SimpleNamespace(control_dt=.02, timeout_s=20., + tool_orientation_wxyz=[1., 0., 0., 0.], + goal_position_tolerance_m=.025) + self.requests, self.jaw = [], jaw + + def pose_action(self, goal, quaternion, jaw): + self.requests.append((list(goal), list(quaternion), jaw)) + # A deliberately nonzero last-issued jaw tests freezing rather than + # silently replacing it with the base guard's default zero target. + return {'joint_targets_rad': [.1]*6, 'gripper_position_m': self.jaw} + + +class Teacher: + def __init__(self, env): + self.env, self.phase, self.issued_phase, self.ticks = env, 'close', 'close', 35 + self.held_offset, self.grasp_position = None, [.3, 0., .082] + self.calls = [] + + def __call__(self, observation): + self.calls.append(self.phase) + if self.phase not in ('approach', 'descend', 'close'): + raise AssertionError('Full-lift teacher must never be dispatched') + self.issued_phase = self.phase + if self.phase == 'close': self.phase, self.ticks = 'lift', 0 + return {'joint_targets_rad': [.1]*6, 'gripper_position_m': self.env.jaw} + + +def observation(t, *, rise=None, supported=None, both=True, force=.4, slip=0., unexpected=0, joints=.02): + rise = max(0., min(.010, .020*(t-.02))) if rise is None else rise + return {'t_seconds': t, 'tcp_position_m': [.3, 0., .082+rise], + 'tcp_quaternion_wxyz': [1., 0., 0., 0.], 'object_position_m': [.3+slip, 0., .08+rise], + 'target_position_m': [.35, .08, .08], 'joint_position_rad': [joints]*6, + 'gripper_joint_position_m': [.02, -.02], 'grasp_contact_counts': {'left': int(both), 'right': int(both)}, + 'pad_normal_forces_n': {'left': force if both else 0., 'right': force if both else 0.}, + 'table_object_contact_count': int(rise < .006 if supported is None else supported), + 'unexpected_contact_count': unexpected, 'native_actuator_controls': [0.]*6+[.027]} + + +def policy(variant='evidence', jaw=.012): + env = Environment(jaw) + with patch.object(piper_task, 'PiperTaskPolicy', Teacher): + value = probe.ProbeStopPolicy(env, variant) + return value, env + + +def until_admitted(value): + start = 0 if value.last_observed_s is None else round(value.last_observed_s/.02)+1 + for index in range(start, 40): + action = value(observation(index*.02)) + if value.admitted: return action + raise AssertionError('Fixture did not admit within bounded time') + + +class ProbeStopTests(unittest.TestCase): + def test_contract_labels_native_scope_and_fair_timer_limit(self): + contract = probe.probe_contract() + self.assertEqual(contract['timer_admission_s'], .6) + self.assertEqual(contract['parameters']['probe_height_m'], .010) + self.assertEqual(contract['parameters']['probe_speed_m_s'], .020) + self.assertFalse(contract['physical_robot_ready']) + self.assertIn('not a fully blind', contract['prefix']) + for variant in ('blind', None, True): + with self.assertRaises(ValueError):policy(variant) + + def test_probe_context_is_exact_and_every_goal_stays_within_ten_mm(self): + value, env = policy() + value(observation(0.));value(observation(.02));value(observation(.04)) + self.assertEqual(value.probes[0]['start_time_s'], .02) + self.assertEqual(value.probes[0]['object_origin_m'], observation(.02)['object_position_m']) + self.assertEqual(value.probes[0]['tcp_origin_m'], observation(.02)['tcp_position_m']) + self.assertAlmostEqual(env.requests[0][0][2]-.082, .0004) + self.assertAlmostEqual(env.requests[1][0][2]-.082, .0008) + self.assertTrue(all(request[0][2] <= .092+1e-12 for request in env.requests)) + + def test_evidence_dwell_stops_at_measured_joints_and_freezes_nonzero_jaw(self): + value, env = policy() + action = until_admitted(value) + self.assertEqual(value.status, 'qualified_hold') + self.assertEqual(action, {'joint_targets_rad': [.02]*6, 'gripper_position_m': .012}) + self.assertGreaterEqual(value.admission_time_s-value.qualification_since+1e-9, .10) + self.assertEqual(value.teacher.calls, ['close']) + before_requests = len(env.requests) + now = value.last_observed_s+.02 + held = value(observation(now, joints=.04)) + self.assertEqual(held, action) + self.assertEqual(len(env.requests), before_requests) + self.assertEqual(value.events[-1]['type'], 'evidence_qualified') + self.assertEqual(value.qualification_checks[-1], {'t_seconds': now, 'qualified': True, 'probe_index': 0}) + + def test_timer_admits_bad_probe_at_fixed_clock_then_hold_invalidates(self): + value, _ = policy('timer') + value(observation(0.)) + for index in range(1, 32): + value(observation(index*.02, rise=0., supported=True, both=False)) + self.assertEqual(value.status, 'timer_hold') + self.assertAlmostEqual(value.admission_time_s-value.probes[0]['start_time_s'], .6) + self.assertEqual(value.events[-1]['type'], 'timer_admitted') + self.assertTrue(value.qualification_checks[-1]['qualified']) + action = value(observation(.64, rise=0., supported=True, both=False)) + self.assertEqual(value.status, 'invalidated_hold') + self.assertFalse(value.admitted) + self.assertEqual(action['gripper_position_m'], .012) + self.assertEqual(value.events[-1]['reason'], 'bilateral_contact_or_force_loss') + + def test_qualification_window_resets_on_weak_force(self): + value, _ = policy() + for index in range(20): + value(observation(index*.02, force=.01 if index == 19 else .4)) + self.assertFalse(value.admitted) + self.assertIsNone(value.qualification_since) + until_admitted(value) + self.assertGreaterEqual(value.qualification_since, .4) + self.assertGreaterEqual(value.admission_time_s-value.qualification_since+1e-9, .1) + + def test_hold_invalidates_on_first_contact_force_support_slip_or_collision_failure(self): + for changes, reason in (({'both': False}, 'bilateral_contact_or_force_loss'), + ({'force': .01}, 'bilateral_contact_or_force_loss'), + ({'supported': True}, 'support_returned'), + ({'slip': .0031}, 'relative_slip'), + ({'unexpected': 1}, 'unexpected_contact'), + ({'rise': .005, 'supported': False}, 'probe_height_not_retained')): + with self.subTest(changes=changes): + value, env = policy();until_admitted(value) + action = value(observation(value.last_observed_s+.02, joints=.05, **changes)) + self.assertEqual(value.abort_reason, reason) + self.assertEqual(action, {'joint_targets_rad': [.05]*6, 'gripper_position_m': .012}) + self.assertFalse(value.admitted) + held = value(observation(value.last_observed_s+.02)) + self.assertEqual(held, action) + self.assertEqual(value.teacher.calls, ['close']) + + def test_initial_unexpected_contact_uses_actual_control_not_zero(self): + value, _ = policy() + action = value(observation(0., unexpected=1)) + self.assertEqual(action['joint_targets_rad'], [.02]*6) + self.assertEqual(action['gripper_position_m'], .027) + self.assertEqual(value.teacher.calls, []) + self.assertEqual(value.status, 'invalidated_hold') + + def test_original_supported_retry_is_bounded_and_preserves_probe_context(self): + value, _ = policy() + value(observation(0.));value(observation(.02)) + value.diagnostic = value._diagnose(observation(.02)) + first = value._alarm(observation(.02), 'probe_timeout') + self.assertEqual(value.retry_count, 1) + self.assertEqual(value.state, 'retry_open') + self.assertEqual(first['gripper_position_m'], .012) + self.assertEqual(value.probes[0]['disposition'], 'interrupted') + second = value._alarm(observation(.02), 'probe_timeout') + self.assertEqual(value.retry_count, 1) + self.assertEqual(value.status, 'invalidated_hold') + self.assertEqual(second['gripper_position_m'], .012) + self.assertIn('retry_budget_exhausted', value.abort_reason) + + def test_final_observation_checks_validity_without_an_extra_control(self): + value, _ = policy();until_admitted(value) + count = len(value.requested_controls) + result = value.observe_final(observation(value.last_observed_s+.02, both=False)) + self.assertEqual(len(value.requested_controls), count) + self.assertEqual(result['status'], 'invalidated_hold') + self.assertTrue(result['final_observed']) + with self.assertRaises(ValueError):value(observation(value.last_observed_s+.02)) + with self.assertRaises(ValueError):value.observe_final(observation(value.last_observed_s+.02)) + + def test_fixed_budget_holds_only_and_accepts_native_terminal_roundoff(self): + value, env = policy();until_admitted(value) + first_held = deepcopy(value.last_requested) + pose_request_count = len(env.requests) + for index in range(round(value.last_observed_s/.02)+1, 1000): + self.assertEqual(value(observation(index*.02)), first_held) + self.assertEqual(len(value.requested_controls), 1000) + self.assertEqual(len(env.requests), pose_request_count) + self.assertEqual(value.teacher.calls, ['close']) + with self.assertRaises(ValueError):value(observation(20.)) + result = value.observe_final(observation(20.00000000000146)) + self.assertTrue(result['admitted']) + self.assertTrue(result['final_observed']) + self.assertEqual(len(result['requested_controls']), 1000) + # The numeric tolerance is much smaller than even one extra 1 ms tick. + other, _ = policy() + other.last_observed_s = 19.98 + with self.assertRaises(ValueError):other.observe_final(observation(20.00000002)) + + def test_no_admission_times_out_without_a_full_lift_or_more_squeeze(self): + value, _ = policy() + value.timeout_s, value.command_budget = 1., 50 + for index in range(50): + action = value(observation(index*.02, rise=0., both=False, supported=True)) + self.assertEqual(value.status, 'timed_out_hold') + self.assertEqual(value.abort_reason, 'episode_acquisition_timeout') + self.assertFalse(value.ever_admitted) + self.assertEqual(action['joint_targets_rad'], [.02]*6) + self.assertEqual(action['gripper_position_m'], .012) + self.assertEqual(value.teacher.calls, ['close']) + final = value.observe_final(observation(1., rise=0., both=False, supported=True)) + self.assertEqual(final['status'], 'timed_out_hold') + self.assertEqual(len(final['requested_controls']), 50) + + def test_observation_clock_and_nonfinite_or_malformed_inputs_fail_closed(self): + for changes in ({'t_seconds': True}, {'t_seconds': float('nan')}, + {'joint_position_rad': [0.]*5}, {'pad_normal_forces_n': {'left': .4}}, + {'grasp_contact_counts': {'left': True, 'right': 1}}, + {'tcp_quaternion_wxyz': [0.]*4}): + value, _ = policy();bad=observation(0.);bad.update(changes) + with self.subTest(changes=changes),self.assertRaises(ValueError):value(bad) + self.assertEqual(value.requested_controls, []) + value, _ = policy();value(observation(0.)) + for timestamp in (0., .01, .04): + with self.subTest(timestamp=timestamp),self.assertRaises(ValueError):value(observation(timestamp)) + self.assertEqual(len(value.requested_controls), 1) + + def test_summary_and_returned_controls_do_not_alias_internal_records(self): + value, _ = policy();action=value(observation(0.));action['joint_targets_rad'][0]=99. + self.assertEqual(value.requested_controls[0]['joint_targets_rad'][0], .1) + result=value.summary();result['events'].append({'fake':True});result['requested_controls'][0]['gripper_position_m']=0. + self.assertEqual(value.events, []) + self.assertEqual(value.requested_controls[0]['gripper_position_m'], .012) + + +if __name__ == '__main__':unittest.main() diff --git a/tests/test_piper_probe_benchmark.py b/tests/test_piper_probe_benchmark.py new file mode 100644 index 0000000..812dae5 --- /dev/null +++ b/tests/test_piper_probe_benchmark.py @@ -0,0 +1,51 @@ +"""Study isolation and refusal behavior; no native experiment hidden in tests.""" +from pathlib import Path +from tempfile import TemporaryDirectory +import unittest +from unittest.mock import patch + +from dvidia_training import piper_probe_benchmark as study + + +class ProbeStudyTests(unittest.TestCase): + def test_existing_output_is_not_overwritten_or_resumed(self): + with TemporaryDirectory() as folder, patch.object(study, 'make_protocol') as protocol: + marker = Path(folder)/'original.txt' + marker.write_text('preserve this failed attempt') + with self.assertRaises(ValueError): + study.prepare(folder, 'unused-assets') + self.assertEqual(marker.read_text(), 'preserve this failed attempt') + protocol.assert_not_called() + + def test_changed_frozen_roster_refuses_before_any_native_attempt(self): + original = {'schema':study.SCHEMA, 'planned_attempts':24, 'cases':[{'id':'original'}]} + changed = {**original, 'planned_attempts':23} + with TemporaryDirectory() as folder, patch.object(study, 'make_protocol', return_value=original), \ + patch.object(study, '_attempt') as attempt: + study.write(Path(folder)/'protocol.json', changed) + with self.assertRaises(ValueError): + study.run(folder, 'unused-assets') + attempt.assert_not_called() + self.assertFalse((Path(folder)/'started.json').exists()) + + def test_started_study_refuses_second_launch(self): + protocol = {'schema':study.SCHEMA, 'cases':[], 'planned_attempts':24} + with TemporaryDirectory() as folder, patch.object(study, 'make_protocol', return_value=protocol), \ + patch.object(study, '_attempt') as attempt: + study.write(Path(folder)/'protocol.json', protocol) + study.write(Path(folder)/'started.json', {'original':'retained'}) + with self.assertRaises(FileExistsError): + study.run(folder, 'unused-assets') + attempt.assert_not_called() + self.assertEqual(study.read(Path(folder)/'started.json'), {'original':'retained'}) + + def test_input_file_bounds_and_symlink_refused(self): + with TemporaryDirectory() as folder: + data = Path(folder)/'data.json'; data.write_text('{"x":1}') + alias = Path(folder)/'alias.json'; alias.symlink_to(data) + for path, limit in ((alias, 100), (data, 2)): + with self.assertRaises(ValueError): study.read(path, limit) + + +if __name__ == '__main__': + unittest.main() diff --git a/tests/test_probe_evidence.py b/tests/test_probe_evidence.py new file mode 100644 index 0000000..d7f88ec --- /dev/null +++ b/tests/test_probe_evidence.py @@ -0,0 +1,281 @@ +"""Native-shaped falsification fixtures; no native simulator or hardware runs.""" +from copy import deepcopy +import unittest + +import numpy as np + +from dvidia_training import probe_evidence as proof +from dvidia_training import task_feasibility + + +def fixture(rows=300): + arrays = {name: np.zeros((rows,) if width == 1 else (rows, width), dtype=np.float64) + for name, width in proof.FIELDS} + arrays['native_tick'][:] = np.arange(1, rows+1) + arrays['t_seconds'][:] = arrays['native_tick']*.001 + arrays['object_position_m'][:] = [.3, -.03, .08] + arrays['tcp_position_m'][:] = [.3, -.03, .18] + lifted = arrays['t_seconds'] >= .04 + arrays['object_position_m'][lifted, 2] += .01 + arrays['tcp_position_m'][lifted, 2] += .01 + arrays['object_minus_tcp_local_m'][:] = arrays['object_position_m']-arrays['tcp_position_m'] + arrays['object_quaternion_wxyz'][:, 0] = 1. + arrays['tcp_quaternion_wxyz'][:, 0] = 1. + arrays['grasp_contact_counts'][:] = [1., 1.] + arrays['pad_normal_forces_n'][:] = [.2, .3] + arrays['table_object_contact_count'][:] = ~lifted + metadata = {'schema': 'dvidia.piper-feedback-native/v1', 'native_timestep_s': .001, + 'rows': rows, 'fields': [{'name': name, 'width': width} for name, width in proof.FIELDS], + 'side_order': ['left', 'right'], 'physical_robot_ready': False} + document = task_feasibility.template() + document['calibration_id'] = 'synthetic_probe_v01' + context = {'schema': proof.CONTEXT_SCHEMA, 'run_id': 'run-00', 'object_id': 'task_block', + 'grasp_id': 'run-00-probe-0', 'probe_index': 0, 'robot_profile': 'piper', + **{name: 'a'*64 for name in proof.DIGEST_FIELDS}, + 'calibration_id': document['calibration_id'], + 'calibration_scope': 'simulation_assumptions_not_hardware', + 'initial_object_position_m': [.3, -.03, .08], 'probe_start_s': .02, + 'probe_origin_object_position_m': arrays['object_position_m'][19].tolist(), + 'probe_origin_tcp_local_offset_m': arrays['object_minus_tcp_local_m'][19].tolist(), + 'per_finger_force_ceiling_n': document['per_finger_normal_force_budget_n'], + 'native_status': {'valid': True, 'warnings': [], 'unexpected_contact_native_ticks': 0, + 'simulation_time_s': float(arrays['t_seconds'][-1])}} + context['feasibility_sha256'] = proof.feasibility_binding(document) + claims = [{'t_seconds': .14, 'qualified': True, 'probe_index': 0}] if rows >= 140 else [] + if rows > 140 and rows % 20 == 0: + claims.append({'t_seconds': rows*.001, 'qualified': True, 'probe_index': 0}) + return arrays, metadata, context, document, claims + + +def derive(arrays, metadata, context, claims): + return proof.derive_probe_evidence(arrays, metadata, context, controller_qualifications=claims) + + +class ProbeEvidenceTests(unittest.TestCase): + def test_fresh_bound_native_proof_replaces_caller_lift_flag_without_mutation(self): + arrays, metadata, context, document, claims = fixture() + originals = deepcopy((arrays, metadata, context, document, claims)) + receipt = derive(arrays, metadata, context, claims) + self.assertTrue(receipt['history']['ever_qualified']) + self.assertTrue(receipt['latest']['qualified']) + result = proof.assess_with_probe(document, receipt, arrays, metadata, context, now_s=.3) + self.assertEqual(result['status'], 'ready_for_simulated_transport') + self.assertTrue(result['probe']['admissible']) + self.assertFalse(result['physical_robot_ready']) + for name in arrays: + np.testing.assert_array_equal(arrays[name], originals[0][name]) + self.assertEqual((metadata, context, document, claims), originals[1:]) + + def test_exact_100_native_tick_window_not_99_ticks(self): + for rows, qualified in ((138, False), (139, True)): + arrays, metadata, context, _, claims = fixture(rows) + receipt = derive(arrays, metadata, context, claims) + self.assertEqual(receipt['latest']['qualified'], qualified) + self.assertEqual(receipt['latest']['continuous_qualifying_s'], .1) + + def test_single_hidden_force_gap_invalidates_20ms_controller_claim(self): + arrays, metadata, context, _, claims = fixture() + arrays['pad_normal_forces_n'][79, 0] = .01 # Exactly threshold is insufficient. + receipt = derive(arrays, metadata, context, claims) + self.assertFalse(receipt['audit']['valid']) + self.assertEqual(receipt['audit']['controller_claim_mismatches'][0]['t_seconds'], .14) + self.assertTrue(receipt['latest']['qualified']) # Later recovery cannot fix an earlier false claim. + verdict = proof.verify_probe_evidence(receipt, arrays, metadata, context, now_s=.3) + self.assertFalse(verdict['admissible']) + + def test_prior_qualification_does_not_substitute_for_latest_live_window(self): + for field in ('pad_normal_forces_n', 'grasp_contact_counts', 'table_object_contact_count'): + arrays, metadata, context, _, claims = fixture() + if field == 'table_object_contact_count': + arrays[field][249] = 1. + else: + arrays[field][249, 0] = 0. + receipt = derive(arrays, metadata, context, claims) + verdict = proof.verify_probe_evidence(receipt, arrays, metadata, context, now_s=.3) + self.assertTrue(verdict['historical_qualified']) + self.assertFalse(verdict['latest_qualified']) + self.assertFalse(verdict['admissible']) + + def test_terminal_controller_invalidation_cannot_be_resurrected_by_native_recovery(self): + arrays, metadata, context, document, claims = fixture() + claims[-1]['qualified'] = False + receipt = derive(arrays, metadata, context, claims) + self.assertTrue(receipt['history']['ever_qualified']) + self.assertTrue(receipt['latest']['qualified']) + self.assertTrue(receipt['audit']['valid']) # Abstaining is not a false positive. + self.assertEqual(receipt['audit']['controller_false_qualification_count'], 0) + self.assertEqual(receipt['audit']['controller_conservative_abstention_count'], 1) + self.assertEqual(receipt['audit']['controller_claim_mismatches'][0]['classification'], 'conservative_abstention') + result = proof.assess_with_probe(document, receipt, arrays, metadata, context, now_s=.3) + self.assertFalse(result['probe']['admissible']) + self.assertIn('latest_controller_not_qualified', result['probe']['reasons']) + self.assertEqual(result['status'], 'test_lift_required') + + def test_latest_controller_claim_must_match_latest_native_sample(self): + arrays, metadata, context, _, claims = fixture() + claims.pop() + receipt = derive(arrays, metadata, context, claims) + verdict = proof.verify_probe_evidence(receipt, arrays, metadata, context, now_s=.3) + self.assertTrue(verdict['historical_qualified']) + self.assertTrue(verdict['latest_qualified']) + self.assertFalse(verdict['admissible']) + self.assertIn('latest_controller_claim_missing_or_stale', verdict['reasons']) + + def test_world_pose_redundancy_and_tcp_local_drift(self): + arrays, metadata, context, _, claims = fixture() + arrays['object_minus_tcp_local_m'][249, 0] = .004 + with self.assertRaisesRegex(proof.ProbeEvidenceError, 'world pose'): + derive(arrays, metadata, context, claims) + arrays['tcp_position_m'][249, 0] -= .004 + receipt = derive(arrays, metadata, context, claims) + self.assertFalse(receipt['latest']['qualified']) + self.assertTrue(receipt['history']['ever_qualified']) + + def test_less_than_six_mm_rise_fails_without_inventing_success(self): + arrays, metadata, context, _, _ = fixture() + arrays['object_position_m'][39:, 2] = .0859 + arrays['tcp_position_m'][39:, 2] = .1859 + arrays['object_minus_tcp_local_m'][:] = arrays['object_position_m']-arrays['tcp_position_m'] + claims = [{'t_seconds': .14, 'qualified': False, 'probe_index': 0}] + receipt = derive(arrays, metadata, context, claims) + self.assertFalse(receipt['history']['ever_qualified']) + self.assertFalse(proof.verify_probe_evidence(receipt, arrays, metadata, context, now_s=.3)['admissible']) + + def test_stale_and_future_evidence_fail_even_after_historical_success(self): + arrays, metadata, context, _, claims = fixture() + receipt = derive(arrays, metadata, context, claims) + for now, reason in ((.32, 'stale_evidence'), (.28, 'future_evidence')): + verdict = proof.verify_probe_evidence(receipt, arrays, metadata, context, now_s=now) + self.assertFalse(verdict['admissible']) + self.assertIn(reason, verdict['reasons']) + + def test_changed_run_grasp_object_pose_command_calibration_or_model_invalidates(self): + arrays, metadata, context, _, claims = fixture() + receipt = derive(arrays, metadata, context, claims) + for field, value in (('run_id', 'run-other'), ('object_id', 'other'), ('grasp_id', 'another-grasp'), + ('pose_binding_sha256', 'b'*64), ('command_sha256', 'b'*64), + ('calibration_id', 'synthetic_other'), ('model_sha256', 'b'*64)): + current = deepcopy(context) + current[field] = value + verdict = proof.verify_probe_evidence(receipt, arrays, metadata, current, now_s=.3) + self.assertFalse(verdict['admissible'], field) + self.assertIn('probe_context_changed', verdict['reasons']) + + def test_receipt_metadata_or_array_mutations_fail_recomputation(self): + arrays, metadata, context, _, claims = fixture() + receipt = derive(arrays, metadata, context, claims) + changed = deepcopy(receipt) + changed['latest']['qualified'] = False + self.assertFalse(proof.verify_probe_evidence(changed, arrays, metadata, context, now_s=.3)['admissible']) + changed_meta = deepcopy(metadata) + changed_meta['new_note'] = 'changed' + self.assertFalse(proof.verify_probe_evidence(receipt, arrays, changed_meta, context, now_s=.3)['admissible']) + arrays['joint_position_rad'][149, 0] += .001 + self.assertFalse(proof.verify_probe_evidence(receipt, arrays, metadata, context, now_s=.3)['admissible']) + + def test_invalid_dynamics_warning_unexpected_contact_and_force_ceiling_refuse(self): + for mutation in ('invalid', 'warning', 'contact', 'ceiling'): + arrays, metadata, context, _, claims = fixture() + if mutation == 'invalid': + context['native_status']['valid'] = False + elif mutation == 'warning': + context['native_status']['warnings'] = ['solver warning'] + elif mutation == 'contact': + arrays['unexpected_contact_count'][249] = 1. + context['native_status']['unexpected_contact_native_ticks'] = 1 + else: + context['per_finger_force_ceiling_n'] = .25 + receipt = derive(arrays, metadata, context, claims) + self.assertFalse(receipt['audit']['valid'], mutation) + self.assertFalse(proof.verify_probe_evidence(receipt, arrays, metadata, context, now_s=.3)['admissible']) + + def test_unknown_mass_friction_budget_payload_and_calibration_remain_unknown(self): + for field in ('mass_interval_kg', 'friction_interval', 'per_finger_normal_force_budget_n', + 'pose_specific_payload_limit_kg', 'calibration_id'): + arrays, metadata, context, document, claims = fixture() + document[field] = None + if field == 'calibration_id': + context['calibration_id'] = None + if field == 'per_finger_normal_force_budget_n': + context['per_finger_force_ceiling_n'] = None + context['feasibility_sha256'] = proof.feasibility_binding(document) + receipt = derive(arrays, metadata, context, claims) + result = proof.assess_with_probe(document, receipt, arrays, metadata, context, now_s=.3) + self.assertTrue(result['probe']['admissible']) + self.assertEqual(result['status'], 'probe_required', field) + self.assertIsNone(document[field]) + arrays, metadata, context, document, claims = fixture() + context['per_finger_force_ceiling_n'] = None + receipt = derive(arrays, metadata, context, claims) + self.assertFalse(proof.assess_with_probe(document, receipt, arrays, metadata, context, now_s=.3)['probe']['admissible']) + + def test_native_critical_total_counts_ticks_not_sum_of_contact_counts(self): + arrays, metadata, context, _, claims = fixture() + arrays['unexpected_contact_count'][249] = 5. + context['native_status']['unexpected_contact_native_ticks'] = 1 + receipt = derive(arrays, metadata, context, claims) + self.assertEqual(receipt['audit']['unexpected_contact_native_ticks'], 1) + self.assertIn('unexpected_native_contact', receipt['audit']['reasons']) + self.assertNotIn('native_contact_total_disagrees', receipt['audit']['reasons']) + context['native_status']['unexpected_contact_native_ticks'] = 5 + receipt = derive(arrays, metadata, context, claims) + self.assertIn('native_contact_total_disagrees', receipt['audit']['reasons']) + + def test_planning_change_and_caller_stale_slip_vetoes_preserved(self): + arrays, metadata, context, document, claims = fixture() + receipt = derive(arrays, metadata, context, claims) + document['planned_upward_accel_m_s2'] = .6 + result = proof.assess_with_probe(document, receipt, arrays, metadata, context, now_s=.3) + self.assertFalse(result['probe']['admissible']) + self.assertIn('planning_assumptions_changed', result['probe']['reasons']) + self.assertEqual(result['status'], 'test_lift_required') + document['planned_upward_accel_m_s2'] = .5 + for field, value in (('observations_fresh', False), ('slip_detected', True)): + changed = deepcopy(document) + changed[field] = value + self.assertEqual(proof.assess_with_probe(changed, receipt, arrays, metadata, context, now_s=.3)['status'], 'blocked') + + def test_caller_test_lift_boolean_cannot_bypass_missing_receipt_or_claim(self): + arrays, metadata, context, document, _ = fixture() + document['test_lift_passed'] = True + self.assertEqual(proof.assess_with_probe(document, None, arrays, metadata, context, now_s=.3)['status'], 'test_lift_required') + receipt = derive(arrays, metadata, context, []) + self.assertTrue(receipt['latest']['qualified']) + self.assertFalse(proof.verify_probe_evidence(receipt, arrays, metadata, context, now_s=.3)['admissible']) + self.assertEqual(proof.assess_with_probe(document, receipt, arrays, metadata, None, now_s=.3)['status'], 'test_lift_required') + + def test_inclusive_rise_and_drift_boundaries_and_old_probe_claim_refusal(self): + arrays, metadata, context, _, claims = fixture() + lifted = arrays['t_seconds'] >= .04 + arrays['object_position_m'][lifted, 2] = .086 + arrays['tcp_position_m'][lifted, 2] = .186 + arrays['tcp_position_m'][lifted, 0] -= .003 + arrays['object_minus_tcp_local_m'][:] = arrays['object_position_m']-arrays['tcp_position_m'] + receipt = derive(arrays, metadata, context, claims) + self.assertTrue(proof.verify_probe_evidence(receipt, arrays, metadata, context, now_s=.3)['admissible']) + new_context = deepcopy(context) + new_context['probe_index'] = 1 + new_context['grasp_id'] = 'run-00-probe-1' + self.assertFalse(proof.verify_probe_evidence(receipt, arrays, metadata, new_context, now_s=.3)['admissible']) + + def test_malformed_cadence_shape_nan_origin_and_controller_binding_reject(self): + for mutation in ('tick', 'nan', 'dtype', 'origin', 'future_claim', 'grasp_claim'): + arrays, metadata, context, _, claims = fixture() + if mutation == 'tick': + arrays['native_tick'][49] += 1 + elif mutation == 'nan': + arrays['qpos' if 'qpos' in arrays else 'joint_position_rad'][49, 0] = np.nan + elif mutation == 'dtype': + arrays['joint_position_rad'] = arrays['joint_position_rad'].astype(np.float32) + elif mutation == 'origin': + context['probe_origin_object_position_m'][0] += .001 + elif mutation == 'future_claim': + claims[0]['t_seconds'] = .32 + else: + claims[0]['probe_index'] = 1 + with self.assertRaises(proof.ProbeEvidenceError, msg=mutation): + derive(arrays, metadata, context, claims) + + +if __name__ == '__main__': + unittest.main()