diff --git a/benchmarks/piper-rgbd-tracker-v01/README.md b/benchmarks/piper-rgbd-tracker-v01/README.md new file mode 100644 index 0000000..f117589 --- /dev/null +++ b/benchmarks/piper-rgbd-tracker-v01/README.md @@ -0,0 +1,102 @@ +# DVIDIA · Known-block RGB-D tracker v0.1 + +A local NumPy estimator for one upright, orange, 40 mm cube. It consumes the +current RGB image, aligned optical-Z depth, camera calibration and frame timing. +It supplies robot-base position, yaw modulo 90 degrees, and explicitly +uncalibrated geometric uncertainty scales—or an explicit missing result. + +This is a static simulated perception benchmark. It does not recover motor +actions from human video, train a policy, execute a robot, or qualify hardware. +The native action recorder and strict learning export are unchanged. Successful +perception is one prerequisite for the proposed observation-to-action study. + +## Local use + +Use Python 3.12 and the repository's pinned NumPy dependency. Estimation itself +uses the CPU and requires no network, hosted model, API key or dedicated GPU. +Generating these fixtures additionally requires MuJoCo and a working OpenGL +context. macOS arm64/CGL is the evaluated rendering host; other graphics backends +and minimum memory/CPU specifications remain unqualified. Install dependencies +and pinned robot assets once before an offline run. + +```sh +git clone --branch piper-rgbd-tracker-v0.1-preview https://github.com/Dvidia-Inference/dvidia-training.git +cd dvidia-training +python3.12 -m venv .venv +. .venv/bin/activate +python -m pip install '.[arm]' +dvidia-robot-sim fetch piper --output runs/robot-profile-upstream/assets +python -m dvidia_training.piper_rgbd_benchmark prepare --output runs/rgbd-study +python -m dvidia_training.piper_rgbd_benchmark render --output runs/rgbd-study \ + --assets /absolute/path/to/robot-profile-upstream/assets --split development +python -m dvidia_training.piper_rgbd_benchmark estimate --output runs/rgbd-study --split development +python -m dvidia_training.piper_rgbd_benchmark score --output runs/rgbd-study --split development +python -m dvidia_training.piper_rgbd_benchmark freeze --output runs/rgbd-study +python -m dvidia_training.piper_rgbd_benchmark render --output runs/rgbd-study \ + --assets /absolute/path/to/robot-profile-upstream/assets --split confirmation +python -m dvidia_training.piper_rgbd_benchmark estimate --output runs/rgbd-study --split confirmation +python -m dvidia_training.piper_rgbd_benchmark score --output runs/rgbd-study --split confirmation +python benchmarks/piper-rgbd-tracker-v01/render_report.py runs/rgbd-study +python -m http.server 8361 --bind 127.0.0.1 --directory runs/rgbd-study +``` + +Every output is exclusive: use a new directory for a new experiment. Read the +report's coverage, refusals and gates together. A returned pose is not proof +that it is correct. `freeze` prevents subsequent changes to the candidate, +configuration, protocol and evaluated renderer binding during confirmation; +it does not certify the estimator or automatically allow learning export. + +## Registered evaluation + +Seed 311301 assigns eight development and sixteen confirmation pose groups. +Each group is rendered at 160×120 and 320×240 with ten conditions: clear, +partial occlusion, 15-degree tilt, 30-degree tilt, absent, fully occluded, an +ambiguous second orange cube, invalid depth, stale frame and future availability. +All variants and resolutions from a group remain in the same split. Positive +coverage is measured separately from absent/unsupported false availability. + +The provisional gate requires at least 90% clear coverage, available-estimate +position p95 ≤10 mm and upright yaw p95 ≤10 degrees, zero negative false +availability, complete metrics and no pipeline failures. These are task-specific +research thresholds, not a confidence guarantee or a manipulation qualification. +Partial-occlusion results are reported separately. The two resolutions are +explicit comparisons; there is no hidden adaptive resolution or retry. + +The fixtures use the actual native PiPER renderer. Object poses are authored +without physics advancing; some elevated blocks are intentionally unsupported +render poses. Occlusion cards and extra cubes are render-only geometry with +valid segmentation IDs so they also exist in depth. Native truth is written +separately and joined only after prediction receipts are sealed. + +Synthetic 0.2-second exposure and stale/future times test clock rules; they are +not physical delivery measurements. CPU and wall estimator times exclude file +I/O and are descriptive measurements on a shared host. CGL renderer warnings +remain in the evidence; `depth_accuracy_verified` remains false. Error within +a heuristic scale is reported as a descriptive count, not calibrated coverage. + +Original DVIDIA code and generated records are MIT. Upstream PiPER assets and +MuJoCo retain their notices; no robot meshes, private footage, credentials or +model weights are bundled in this evidence release. + +## Frozen result + +Source freeze: `846a1d9b04745a516ff060b4efa931e5e9898fbf`. Sixteen confirmation +pose groups × two resolutions × ten conditions: all 320 fixtures rendered and +scored once. + +| Resolution | Clear estimates | Center error p95 | Yaw error p95 | Estimator wall p95 | Gate | +| --- | ---: | ---: | ---: | ---: | --- | +| 160×120 | 12/16 | 2.34 mm | 11.31° | 7.73 ms | Coverage and yaw failed | +| 320×240 | 16/16 | 1.98 mm | 6.36° | 7.58 ms | Clear-scene gate passed | + +Both resolutions withheld all 256 negative estimates. Partial occlusion returned +0/32 estimates and remains unsupported. These are correlated static fixtures, +not 320 independent trials, hardware accuracy or learned placement. + +Independent audit: zero mismatches across 320 rows, 20 summaries and two gates; +2,578 evidence files and 32 occluder-depth checks. The initial sandbox CGL failure +and development sources/results are retained. The first candidate dropped +shadowed orange faces; brightness floors 65/20 → 25/8 recovered coverage without +changing geometry gates. No confirmation tuning followed. The native action +recorder, strict learning export and earlier learned placement 5/12 versus its +authored teacher 12/12 remain unchanged. diff --git a/benchmarks/piper-rgbd-tracker-v01/independent_audit.py b/benchmarks/piper-rgbd-tracker-v01/independent_audit.py new file mode 100644 index 0000000..bf8d87a --- /dev/null +++ b/benchmarks/piper-rgbd-tracker-v01/independent_audit.py @@ -0,0 +1,302 @@ +"""Read-only audit of saved RGB-D confirmation evidence (stdlib + NumPy). + +No simulator, tracker or benchmark implementation is imported or executed. +Truth is used only to audit sealed estimates and rendered fixture geometry. +""" +from __future__ import annotations + +import argparse +from hashlib import sha256 +import json +import math +from pathlib import Path +import sys + +import numpy as np + + +VARIANTS = ('clear', 'partial_occlusion', 'tilt15', 'tilt30', 'absent', + 'occluded', 'ambiguous', 'invalid_depth', 'stale', 'future') +SIGNALS = ('object_position_estimate_m', 'object_quaternion_estimate_wxyz', + 'object_position_uncertainty_m', 'object_orientation_uncertainty_rad') +INPUTS = ('rgb.npy', 'depth.npy', 'input.json') +SCOPE = ('Independent readback of static synthetic known-cube perception only. ' + 'No simulation, estimator, action learning or robot qualification was run. ' + 'Depth accuracy and heuristic uncertainty scales remain uncalibrated.') + + +def canonical(value): + return json.dumps(value, sort_keys=True, separators=(',', ':'), allow_nan=False).encode() + + +class Audit: + def __init__(self, study): + self.study = Path(study); self.errors = []; self.identities = {} + + def check(self, condition, label): + if not condition: + self.errors.append(label) + return bool(condition) + + def identity(self, path): + path = Path(path) + if path.is_symlink() or not path.is_file(): + raise ValueError('Missing/nonregular evidence file: '+path.name) + body = path.read_bytes() + result = dict(bytes=len(body), sha256=sha256(body).hexdigest()) + try: name = str(path.relative_to(self.study)) + except ValueError: name = 'external-source/'+path.name + self.identities[name] = result + return result + + def load(self, path): + self.identity(path) + def bad(value): raise ValueError('Nonfinite JSON evidence') + return json.loads(Path(path).read_text(), parse_constant=bad) + + def equal(self, actual, expected, label): + if isinstance(expected, bool) or expected is None or isinstance(expected, str): + return self.check(type(actual) is type(expected) and actual == expected, label) + if isinstance(expected, int): + return self.check(type(actual) is int and actual == expected, label) + if isinstance(expected, float): + return self.check(type(actual) in (int, float) and math.isfinite(actual) + and math.isclose(actual, expected, abs_tol=1e-12, rel_tol=1e-9), label) + if isinstance(expected, list): + if not self.check(isinstance(actual, list) and len(actual) == len(expected), label): return False + return all([self.equal(a, b, label+f'[{i}]') for i, (a, b) in enumerate(zip(actual, expected))]) + if isinstance(expected, dict): + if not self.check(isinstance(actual, dict) and set(actual) == set(expected), label): return False + return all([self.equal(actual[k], value, label+'.'+k) for k, value in expected.items()]) + raise TypeError('Unsupported audit comparison') + + +def percentile(rows, key, percent): + values = [r[key] for r in rows if r[key] is not None] + # Explicit linear interpolation rather than calling the runner's reducer. + if not values: return None + values.sort(); location = (len(values)-1)*percent/100 + lower = int(math.floor(location)); upper = int(math.ceil(location)) + return float(values[lower]+(values[upper]-values[lower])*(location-lower)) + + +def _signal_contract(audit, result, timing, label): + audit.check(isinstance(result, dict), label+'.result_mapping') + audit.check(result.get('status') in ('available', 'missing', 'stale', 'refused'), label+'.status') + for key, value in timing.items(): audit.equal(result.get(key), value, label+'.'+key) + causal = timing['captured_at_seconds'] <= timing['available_at_seconds'] <= timing['decision_at_seconds'] + age = timing['decision_at_seconds']-timing['captured_at_seconds'] if causal else None + audit.equal(result.get('age_seconds'), age, label+'.age') + audit.check(result.get('base_frame') == 'robot_base', label+'.base_frame') + audit.check(result.get('depth_accuracy_verified') is False, label+'.depth_accuracy_flag') + signals = result.get('signals', {}) + audit.check(set(signals) == set(SIGNALS), label+'.signal_names') + if result['status'] == 'available': + audit.check(causal and age <= .1+1e-12, label+'.fresh_causal_available') + for name, size in zip(SIGNALS[:2], (3, 4)): + value = result.get(name) + if not (isinstance(value, list) and len(value) == size + and all(type(v) in (int, float) and math.isfinite(v) for v in value)): + raise ValueError('Malformed available pose') + quaternion = result[SIGNALS[1]] + audit.check(math.isclose(sum(x*x for x in quaternion), 1., abs_tol=1e-6, rel_tol=0), label+'.quaternion_norm') + audit.check(abs(quaternion[1]) <= 1e-8 and abs(quaternion[2]) <= 1e-8, label+'.upright_quaternion') + for name in SIGNALS[2:]: + value = result.get(name) + if not (type(value) in (int, float) and math.isfinite(value) and value >= 0): + raise ValueError('Malformed available uncertainty') + for name in SIGNALS: + value = result[name] if isinstance(result[name], list) else [result[name]] + audit.equal(signals.get(name), dict(status='available', value=value, + captured_at_seconds=timing['captured_at_seconds'], available_at_seconds=timing['available_at_seconds'], + age_seconds=age), label+'.signals.'+name) + else: + for name in SIGNALS: + audit.check(result.get(name) is None, label+'.abstain.'+name) + audit.equal(signals.get(name), dict(status='missing', value=None, + captured_at_seconds=None, available_at_seconds=None, age_seconds=None), label+'.signals.'+name) + + +def _occluder_check(audit, folder, group_id, width): + """Independently project the cube centre and inspect saved depth, not RGB labels.""" + clear = folder/f'{group_id}-{width}-clear'; hidden = folder/f'{group_id}-{width}-occluded' + calibration = audit.load(clear/'input.json')['calibration'] + other = audit.load(hidden/'input.json')['calibration'] + audit.equal(other, calibration, group_id+f'-{width}.occluder_calibration') + truth = audit.load(clear/'truth.json'); hidden_truth = audit.load(hidden/'truth.json') + audit.equal(hidden_truth['position_m'], truth['position_m'], group_id+'.occluder_same_object') + transform = np.asarray(calibration['base_from_camera'], dtype=float).reshape(4, 4) + optical = transform[:3, :3].T @ (np.asarray(truth['position_m'])-transform[:3, 3]) + intrinsics = calibration['intrinsics']; height = intrinsics['height'] + if optical[2] <= .1: raise ValueError('Occluder fixture projects behind/inside camera') + column = int(math.floor(intrinsics['fx']*optical[0]/optical[2]+intrinsics['cx'])) + row = int(math.floor(intrinsics['fy']*optical[1]/optical[2]+intrinsics['cy'])) + if not 0 <= column < width or not 0 <= row < height: raise ValueError('Object centre lies outside saved frame') + original = np.load(clear/'depth.npy', allow_pickle=False) + covered = np.load(hidden/'depth.npy', allow_pickle=False) + if original.shape != (height, width) or covered.shape != (height, width): raise ValueError('Occluder depth alignment failed') + before = float(original[row, column]); after = float(covered[row, column]) + if not math.isfinite(before) or not math.isfinite(after): + raise ValueError('Nonfinite occluder comparison depth') + # Fixture card centre is 60mm toward the camera, with 2mm half-thickness. + # The front face is constant optical Z, even away from the pixel principal ray. + expected = float(optical[2]-.062) + okay = math.isfinite(before) and math.isfinite(after) and abs(after-expected) <= .002 and before-after >= .015 + audit.check(okay, group_id+f'-{width}.occluder_depth_present') + return dict(group_id=group_id, width=width, pixel=[column, row], + clear_depth_m=before, occluded_depth_m=after, expected_card_depth_m=expected, + card_depth_absolute_error_m=abs(after-expected), foreground_separation_m=before-after, passed=okay) + + +def audit_study(study, source_root=None): + audit = Audit(study); folder = audit.study/'confirmation'; summaries = []; gates = []; occluders = [] + processed = 0 + try: + protocol = audit.load(audit.study/'protocol.json'); frozen = audit.load(audit.study/'freeze.json') + manifest = audit.load(folder/'manifest.json'); scheduled = audit.load(folder/'scheduled.json') + predictions = audit.load(folder/'predictions.json'); report = audit.load(folder/'report.json') + audit.equal(frozen['protocol_sha256'], audit.identity(audit.study/'protocol.json')['sha256'], 'freeze.protocol') + for name, identity in frozen['development_evidence'].items(): + if Path(name).name != name: raise ValueError('Unsafe development evidence name') + audit.equal(audit.identity(audit.study/'development'/name), identity, 'development.'+name) + if source_root is not None: + for name, digest in frozen['sources'].items(): + if Path(name).name != name: raise ValueError('Unsafe source filename') + audit.equal(audit.identity(Path(source_root)/name)['sha256'], digest, 'source.'+name) + groups = protocol['groups']; confirmation = [g for g in groups if g['split'] == 'confirmation'] + audit.check(len(groups) == 24 and len(confirmation) == 16 and len({g['id'] for g in groups}) == 24, + 'exact_24_groups_16_confirmation') + if any(not isinstance(g['id'], str) or not g['id'].isalnum() for g in groups): raise ValueError('Unsafe group id') + audit.equal(protocol['resolutions'], [[160, 120], [320, 240]], 'resolutions') + audit.equal(protocol['variants'], list(VARIANTS), 'variants') + audit.equal(protocol['expected_available'], ['clear'], 'available_protocol') + audit.equal(protocol['challenge_positive'], ['partial_occlusion'], 'partial_protocol') + negatives = [v for v in VARIANTS if v not in ('clear', 'partial_occlusion')] + audit.equal(protocol['expected_abstain'], negatives, 'negative_protocol') + expected = [] + for width, height in ((160, 120), (320, 240)): + for group in confirmation: + for variant in VARIANTS: + expected.append(dict(id=f'{group["id"]}-{width}-{variant}', group_id=group['id'], + variant=variant, width=width, height=height, frame_index=len(expected))) + audit.equal(scheduled['records'], expected, 'scheduled_records') + audit.check(len(expected) == 320, 'exact_320_expected_ids') + for document, name in ((manifest, 'manifest'), (predictions, 'predictions'), (scheduled, 'scheduled')): + audit.equal(document['protocol_sha256'], frozen['protocol_sha256'], name+'.protocol') + audit.equal(document['split'], 'confirmation', name+'.split') + audit.equal(manifest['scheduled_sha256'], audit.identity(folder/'scheduled.json')['sha256'], 'manifest.schedule') + audit.equal(predictions['manifest_sha256'], audit.identity(folder/'manifest.json')['sha256'], 'predictions.manifest') + audit.equal(report['manifest_sha256'], predictions['manifest_sha256'], 'report.manifest') + audit.equal(report['predictions_sha256'], audit.identity(folder/'predictions.json')['sha256'], 'report.predictions') + audit.equal(predictions['freeze_sha256'], audit.identity(audit.study/'freeze.json')['sha256'], 'predictions.freeze') + audit.check(manifest['source_stable'] is True, 'render_source_stable') + for document, name in ((manifest, 'manifest'), (predictions, 'predictions')): + audit.equal(document['source_hashes'], frozen['sources'], name+'.frozen_sources') + audit.equal(document['runtime'], frozen['runtime'], name+'.frozen_runtime') + audit.equal(manifest['renderer_bindings'], frozen['renderer_bindings'], 'renderer_bindings') + items = manifest['records']; estimates = predictions['predictions']; published_rows = report['rows'] + ids = [row['id'] for row in expected] + for rows, name in ((items, 'manifest'), (estimates, 'predictions'), (published_rows, 'report')): + if [row['id'] for row in rows] != ids: raise ValueError(name+' missing/duplicate/unknown/reordered scheduled ids') + computed = [] + for registered, item, pred, published in zip(expected, items, estimates, published_rows): + label = item['id']; fixture = folder/label + for name, value in registered.items(): audit.equal(item.get(name), value, label+'.registration.'+name) + receipt = audit.load(fixture/'receipt.json') + audit.equal(audit.identity(fixture/'receipt.json')['sha256'], item['receipt_sha256'], label+'.render_receipt_hash') + audit.equal(receipt, {k: v for k, v in item.items() if k != 'receipt_sha256'}, label+'.render_receipt') + for name, identity in item['files'].items(): + if Path(name).name != name or name in ('', '.', '..'): raise ValueError('Unsafe artifact name') + audit.equal(audit.identity(fixture/name), identity, label+'.artifact.'+name) + sealed = audit.load(folder/'estimates'/(label+'.json')) + audit.equal(audit.identity(folder/'estimates'/(label+'.json'))['sha256'], pred['receipt_sha256'], label+'.estimate_receipt_hash') + audit.equal(sealed, {k: v for k, v in pred.items() if k != 'receipt_sha256'}, label+'.estimate_receipt') + status = pred['estimate_status']; available = False; error = None; yaw_error = None; pos_scale = None; angle_scale = None + audit.check(status in ('complete', 'failed', 'unattempted'), label+'.estimate_status') + if status == 'complete': + audit.check(item['render_status'] == 'rendered', label+'.completed_fixture') + audit.equal(pred['inputs'], {name: item['files'][name] for name in INPUTS}, label+'.input_ids') + inputs = audit.load(fixture/'input.json'); result = pred['result'] + _signal_contract(audit, result, inputs['timing'], label) + truth = audit.load(fixture/'truth.json'); available = result['status'] == 'available' + if available and truth['present']: + error = math.sqrt(sum((a-b)**2 for a, b in zip(result[SIGNALS[0]], truth['position_m']))) + q = result[SIGNALS[1]]; angle = 2*math.atan2(q[3], q[0]) + if item['variant'] not in ('tilt15', 'tilt30') and abs(truth['tilt_rad']) <= 1e-6: + difference = angle-truth['yaw_rad'] + yaw_error = abs((difference+math.pi/4) % (math.pi/2)-math.pi/4) + pos_scale, angle_scale = result[SIGNALS[2]], result[SIGNALS[3]] + else: + audit.check(pred['result'] is None, label+'.failed_result_null') + metrics = dict(available=available, position_error_m=error, yaw_error_rad=yaw_error, + position_heuristic_scale_m=pos_scale, orientation_heuristic_scale_rad=angle_scale, + position_error_within_heuristic_scale=error <= pos_scale if error is not None else None, + yaw_error_within_heuristic_scale=yaw_error <= angle_scale if yaw_error is not None else None, + wall_seconds=pred['wall_seconds'], cpu_seconds=pred['cpu_seconds']) + for name, value in metrics.items(): audit.equal(published.get(name), value, label+'.report.'+name) + for name in ('wall_seconds', 'cpu_seconds'): + audit.check(metrics[name] is None or (type(metrics[name]) in (int, float) and math.isfinite(metrics[name]) and metrics[name] >= 0), label+'.'+name) + computed.append(dict(**registered, **metrics, estimate_status=status)); processed += 1 + for width, _ in ((160, 120), (320, 240)): + for variant in VARIANTS: + rows = [r for r in computed if r['width'] == width and r['variant'] == variant] + summary = dict(width=width, variant=variant, scheduled=len(rows), available=sum(r['available'] for r in rows), + pipeline_failures=sum(r['estimate_status'] != 'complete' for r in rows), + position_evaluated=sum(r['position_error_m'] is not None for r in rows), + yaw_evaluated=sum(r['yaw_error_rad'] is not None for r in rows), + position_p50_m=percentile(rows,'position_error_m',50), position_p95_m=percentile(rows,'position_error_m',95), + yaw_p95_rad=percentile(rows,'yaw_error_rad',95), wall_p50_s=percentile(rows,'wall_seconds',50), + wall_p95_s=percentile(rows,'wall_seconds',95), cpu_p95_s=percentile(rows,'cpu_seconds',95), + heuristic_comparisons=sum(r['position_error_m'] is not None for r in rows), + yaw_heuristic_comparisons=sum(r['yaw_error_rad'] is not None for r in rows), + position_error_within_heuristic_scale=sum(r['position_error_within_heuristic_scale'] is True for r in rows), + yaw_error_within_heuristic_scale=sum(r['yaw_error_within_heuristic_scale'] is True for r in rows)) + summaries.append(summary) + rows = [r for r in computed if r['width'] == width]; clear = [r for r in rows if r['variant'] == 'clear'] + negative = [r for r in rows if r['variant'] in negatives]; limits = protocol['provisional_gate'] + coverage = sum(r['available'] for r in clear)/len(clear) + position = percentile(clear, 'position_error_m', 95); yaw = percentile(clear, 'yaw_error_rad', 95) + false = sum(r['available'] for r in negative) + checks = dict(all_scheduled_fixtures_complete=not manifest['session_errors'] and all(r['estimate_status'] == 'complete' for r in rows), + clear_pose_metrics_complete=all(r['position_error_m'] is not None and r['yaw_error_rad'] is not None for r in clear if r['available']), + clear_coverage=coverage >= limits['clear_coverage_min'], + clear_position=position is not None and position <= limits['clear_position_p95_m_max'], + clear_yaw=yaw is not None and yaw <= limits['clear_yaw_p95_rad_max'], no_negative_false_available=false <= limits['negative_false_available_max']) + gates.append(dict(width=width, passed=all(checks.values()), checks=checks, clear_scheduled=len(clear), + clear_coverage=coverage, negative_scheduled=len(negative), negative_false_available=false)) + for group in confirmation: occluders.append(_occluder_check(audit, folder, group['id'], width)) + audit.equal(report['summary'], summaries, 'report.summary') + audit.equal(report['gates'], gates, 'report.gates') + audit.equal(report['scheduled'], 320, 'report.scheduled') + audit.check(report['physical_robot_ready'] is False and report['depth_accuracy_verified'] is False, 'report.scope_flags') + except Exception as error: + message = str(error).replace(str(audit.study), '') + if source_root is not None: message = message.replace(str(source_root), '') + audit.errors.append('fatal:'+type(error).__name__+':'+message[:180]) + inventory = dict(sorted(audit.identities.items())) + return dict(schema='dvidia.rgbd-independent-audit/v1', passed=not audit.errors, + mismatch_count=len(audit.errors), mismatches=audit.errors, processed_confirmation_ids=processed, + expected_confirmation_ids=320, source_bytes_checked=source_root is not None, + source_bytes_verified=source_root is not None and not audit.errors, + evidence_files_read=len(inventory), evidence_inventory_sha256=sha256(canonical(inventory)).hexdigest(), + evidence_inventory=inventory, summaries=summaries, gates=gates, occluder_depth_checks=occluders, scope=SCOPE, + simulator_executed=False, estimator_executed=False, physical_robot_ready=False) + + +def main(argv=None): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument('--study', required=True, type=Path); parser.add_argument('--output', required=True, type=Path) + parser.add_argument('--source-root', type=Path); parser.add_argument('--repo', type=Path) + args = parser.parse_args(argv) + if args.source_root is not None and args.repo is not None: parser.error('Choose --source-root or --repo') + if args.output.exists(): raise FileExistsError('Audit output already exists; refusing overwrite') + source = args.source_root or (args.repo/'src/dvidia_training' if args.repo else None) + report = audit_study(args.study, source) + body = canonical(report)+b'\n' + with args.output.open('xb') as stream: stream.write(body) + print(json.dumps(dict(passed=report['passed'], mismatch_count=report['mismatch_count'], + processed_confirmation_ids=report['processed_confirmation_ids'], audit_sha256=sha256(body).hexdigest()))) + return 0 if report['passed'] else 1 + + +if __name__ == '__main__': sys.exit(main()) diff --git a/benchmarks/piper-rgbd-tracker-v01/package_evidence.py b/benchmarks/piper-rgbd-tracker-v01/package_evidence.py new file mode 100644 index 0000000..30268ed --- /dev/null +++ b/benchmarks/piper-rgbd-tracker-v01/package_evidence.py @@ -0,0 +1,254 @@ +"""Package reviewed static perception evidence, preserving development failures. + +A passed audit is tied to the bytes actually copied and read back from the ZIP. +This verifies local identities, not publisher authentication or hardware safety. +""" +import argparse +from hashlib import sha256 +import json +import os +from pathlib import Path, PurePosixPath +import re +import stat +import zipfile + +MAX_FILE = 16*1024*1024 +MAX_ARCHIVE = 512*1024*1024 +COUNTS = {'development':160, 'confirmation':320} +VARIANTS = ('clear','partial_occlusion','tilt15','tilt30','absent','occluded','ambiguous','invalid_depth','stale','future') +FIXTURE_FILES = {'started.json','rgb.png','rgb.npy','depth.npy','input.json','truth.json'} +ROOT_FILES = {'protocol.json','freeze.json','development-lineage.json','index.html'} +SPLIT_FILES = {'manifest.json','predictions.json','report.json','scheduled.json','renderer-160.json','renderer-320.json'} + + +def canonical(value): + return json.dumps(value,sort_keys=True,separators=(',',':'),allow_nan=False).encode() + + +def decode(raw): + def reject(value):raise ValueError('Nonfinite JSON evidence') + return json.loads(raw,parse_constant=reject) + + +def _plain(path): + path=Path(path) + if any(p.is_symlink() for p in (path,*path.parents)): + raise ValueError('Symlinks are forbidden in evidence paths') + return path + + +def _snapshot(path, maximum, *, capture): + path=_plain(path) + descriptor=os.open(path,os.O_RDONLY|os.O_NOFOLLOW) + with os.fdopen(descriptor,'rb') as stream: + before=os.fstat(stream.fileno()) + if not stat.S_ISREG(before.st_mode) or before.st_size>maximum: + raise ValueError('Expected bounded plain evidence file') + chunks=[];digest=sha256();size=0 + while True: + raw=stream.read(min(1024*1024,maximum-size+1)) + if not raw:break + size+=len(raw) + if size>maximum:raise ValueError('Evidence exceeded size bound') + digest.update(raw) + if capture:chunks.append(raw) + after=os.fstat(stream.fileno()) + fields=('st_dev','st_ino','st_size','st_mtime_ns','st_ctime_ns') + if any(getattr(before,k)!=getattr(after,k) for k in fields) or size!=before.st_size: + raise ValueError('Evidence changed while being read') + return b''.join(chunks) if capture else dict(bytes=size,sha256=digest.hexdigest()) + + +def read(path):return _snapshot(path,MAX_FILE,capture=True) +def identity(raw):return dict(bytes=len(raw),sha256=sha256(raw).hexdigest()) +def archive_identity(path):return _snapshot(path,MAX_ARCHIVE,capture=False) + + +def safe_name(value): + if not isinstance(value,str) or '\\' in value or value!=PurePosixPath(value).as_posix(): + raise ValueError('Unsafe evidence path') + parts=PurePosixPath(value).parts + if not parts or value.startswith('/') or any(p in ('','.','..') or p.startswith('.') for p in parts): + raise ValueError('Unsafe evidence path') + return value + + +def checked_identity(value): + if (not isinstance(value,dict) or set(value)!={'bytes','sha256'} + or type(value['bytes']) is not int or not 0MAX_FILE or identity(zipped.read(info))!=zip_roster[info.filename]: + raise ValueError('Archived bytes differ from sealed evidence') + archive_record=archive_identity(archive) + payload=output/'huggingface';payload.mkdir() + with archive.open('rb') as source,(payload/archive.name).open('xb') as target: + while raw:=source.read(1024*1024):target.write(raw) + if archive_identity(archive)!=archive_record or archive_identity(payload/archive.name)!=archive_record: + raise ValueError('Release archive changed during payload copy') + for source_name,name in (('confirmation/report.json','report.json'),('protocol.json','protocol.json'), + ('independent-audit.json','inspection.json'),('LICENSE.txt','LICENSE')): + raw=bound_read(tree/source_name,roster[source_name]) + with (payload/name).open('xb') as stream:stream.write(raw) + if read(payload/name)!=raw:raise ValueError('Payload copy differs from archived evidence') + with (payload/'README.md').open('xb') as stream:stream.write(card_raw) + if read(payload/'README.md')!=card_raw:raise ValueError('Payload card changed') + result=dict(archive=archive.name,**archive_record,inventory_files=len(roster), + independent_audit_sha256=identity(audit_raw)['sha256'],zip_readback_verified=True) + with (output/'package.json').open('x') as stream:stream.write(json.dumps(result,indent=2)+'\n') + return result + + +if __name__=='__main__': + p=argparse.ArgumentParser(description=__doc__) + for name in ('study','audit','output','baseline','earlier','failed','card'):p.add_argument('--'+name,required=True,type=Path) + a=p.parse_args();print(json.dumps(package(a.study,a.audit,a.output,baseline=a.baseline,earlier=a.earlier,failed=a.failed,card=a.card),indent=2)) diff --git a/benchmarks/piper-rgbd-tracker-v01/render_report.py b/benchmarks/piper-rgbd-tracker-v01/render_report.py new file mode 100644 index 0000000..f93ecf4 --- /dev/null +++ b/benchmarks/piper-rgbd-tracker-v01/render_report.py @@ -0,0 +1,35 @@ +"""Build an offline, image-first reader of sealed RGB-D study results.""" +import argparse +import html +import json +from pathlib import Path + + +def render(study, output): + study, output = Path(study), Path(output) + report = json.loads((study/'confirmation/report.json').read_text()) + rows = report['rows'] + summaries = report['summary'] + esc = lambda value: html.escape(str(value), quote=True) + def fmt(value, scale=1, suffix=''): + return '—' if value is None else f'{value*scale:.2f}{suffix}' + table = ''.join(''+''.join(''+esc(value)+'' for value in ( + f'{r["width"]} px',r['variant'].replace('_',' '),f'{r["available"]}/{r["scheduled"]}', + fmt(r['position_p95_m'],1000,' mm'),fmt(r['yaw_p95_rad'],180/3.141592653589793,'°'), + fmt(r['wall_p95_s'],1000,' ms')) )+'' for r in summaries) + gate_text = '; '.join(f'{g["width"]} px: '+('clear-scene gate passed' if g['passed'] else 'gate failed') for g in report['gates']) + payload = json.dumps(rows,allow_nan=False).replace('<','\\u003c') + page='''From pixels to position · DVIDIA research

From pixels to position.

A small local estimator looks for one known block in color and depth. This experiment measures when it can estimate a pose—and when it should say it cannot.

Static rendered fixtures. One orange, upright, 40 mm cube. No new learned manipulation, task success, or hardware qualification. “Available” means the estimator returned a pose; it does not mean that pose is correct.

Rendered PiPER workstation fixture

All confirmation conditions

'''+esc(gate_text)+'''. Partial occlusion is reported separately and remains unsupported.

Each row contains 16 held-out pose groups. Variants and resolutions from one group are correlated; they are not extra independent examples. Error percentiles use available estimates only, so read them alongside the coverage.

'''+table+'''
Image widthConditionAvailablePosition error p95Yaw error p95Estimator time p95

What this measures

RGB and depth come from the same paused native renderer. The estimator receives pixels, optical-Z depth, declared camera calibration and timing metadata. Native object pose enters only the offline evaluation after predictions are saved. Occlusion cards and duplicate objects are explicit render-only challenge geometry; these are perception fixtures, not physical manipulation trials.

Cube yaw is scored modulo 90°. Tilted objects are outside the estimator’s upright profile. Uncertainty values are geometric heuristics, not calibrated confidence bounds. Synthetic stale/future timestamps test the data contract; measured compute time does not establish sensor delivery latency or a real-time robot deadline. The renderer’s depth-accuracy limitation remains explicit.

''' + with output.open('x') as stream:stream.write(page) + + +if __name__=='__main__': + parser=argparse.ArgumentParser(description=__doc__);parser.add_argument('study',type=Path);parser.add_argument('--output',type=Path) + args=parser.parse_args();render(args.study,args.output or args.study/'index.html') diff --git a/publishing/huggingface/publish_piper_rgbd_tracker_v01.py b/publishing/huggingface/publish_piper_rgbd_tracker_v01.py new file mode 100644 index 0000000..7fd3f5b --- /dev/null +++ b/publishing/huggingface/publish_piper_rgbd_tracker_v01.py @@ -0,0 +1,184 @@ +"""Stage or publish the reviewed PiPER RGB-D tracker v0.1 evidence dataset. + +This publisher is intentionally inert until its explicit ``publish`` command. +It freezes exactly six reviewed files and writes one commit only to an empty, +public dataset head. It does not modify any Space. +""" +from __future__ import annotations + +import argparse +from hashlib import sha256 +import json +from pathlib import Path +import re +import sys + +from publish import PublicationError, client +from publish_task_environments_v02 import digest, encoded, plain, public_client, download, scan + + +REPO = 'Dvidia/piper-rgbd-tracker-v01' +SCHEMA = 'dvidia.piper-incremental-hold-publication/v1' +ARCHIVE = 'piper-rgbd-tracker-v01.zip' +NAMES = {'README.md', 'LICENSE', ARCHIVE, 'report.json', 'protocol.json', 'inspection.json'} +PLAN_LIMIT = 4 * 1024 * 1024 +LIMITS = {name: (192 * 1024 * 1024 if name == ARCHIVE else + 16 * 1024 * 1024 if name == 'report.json' else 4 * 1024 * 1024) + for name in NAMES} +OID = re.compile(r'(?:[a-f0-9]{40}|[a-f0-9]{64})\Z') + + +def read(path, limit=None): + path = plain(path) + limit = LIMITS.get(path.name, PLAN_LIMIT) if limit is None else limit + if not path.is_file() or path.stat().st_size > limit: + raise PublicationError('Missing or oversized reviewed publication file.') + with path.open('rb') as stream: + raw = stream.read(limit + 1) + if len(raw) > limit: + raise PublicationError('Publication file changed beyond its byte limit.') + return raw + + +def files(root): + root = plain(root) + if not root.is_dir(): + raise PublicationError('Expected a plain publication directory.') + paths = sorted(root.iterdir()) + if {path.name for path in paths} != NAMES: + raise PublicationError('Dataset must contain exactly the reviewed six files.') + inventory = [] + for path in paths: + raw = read(path) + scan(raw) + inventory.append({'path': path.name, **digest(raw), + 'representation': 'lfs' if path.name == ARCHIVE else 'git'}) + return inventory + + +def stage(root, plan): + root, plan = plain(root), plain(plan) + value = {'schema': SCHEMA, 'repo_id': REPO, 'repo_type': 'dataset', + 'private': False, 'root': str(root), 'files': files(root)} + with plan.open('xb') as stream: + stream.write(encoded(value)) + return {'plan_sha256': sha256(read(plan, PLAN_LIMIT)).hexdigest(), + 'files': len(value['files']), 'bytes': sum(row['bytes'] for row in value['files'])} + + +def _pairs(items): + result = {} + for key, value in items: + if key in result: + raise PublicationError('Duplicate keys in reviewed publication plan.') + result[key] = value + return result + + +def _constant(_value): + raise PublicationError('Nonfinite values in reviewed publication plan.') + + +def checked(plan, expected_digest): + raw = read(plan, PLAN_LIMIT) + if sha256(raw).hexdigest() != expected_digest: + raise PublicationError('Reviewed plan digest changed.') + try: + value = json.loads(raw, object_pairs_hook=_pairs, parse_constant=_constant) + except (ValueError, TypeError, UnicodeError) as exc: + raise PublicationError('Malformed reviewed publication plan.') from exc + if (type(value) is not dict + or set(value) != {'schema', 'repo_id', 'repo_type', 'private', 'root', 'files'} + or value['schema'] != SCHEMA or value['repo_id'] != REPO + or value['repo_type'] != 'dataset' or value['private'] is not False + or type(value['root']) is not str or type(value['files']) is not list + or len(value['files']) != len(NAMES)): + raise PublicationError('Reviewed dataset identity changed.') + for row in value['files']: + if (type(row) is not dict or set(row) != {'path', 'bytes', 'sha256', 'representation'} + or type(row['path']) is not str or row['path'] not in NAMES + or type(row['bytes']) is not int or row['bytes'] < 0 + or type(row['sha256']) is not str or not re.fullmatch(r'[a-f0-9]{64}', row['sha256']) + or row['representation'] != ('lfs' if row['path'] == ARCHIVE else 'git')): + raise PublicationError('Reviewed dataset inventory is malformed.') + if ({row['path'] for row in value['files']} != NAMES + or files(value['root']) != value['files']): + raise PublicationError('Reviewed dataset identity or bytes changed.') + return value + + +def _remote(info): + if (getattr(info, 'private', None) is not False + or not OID.fullmatch(getattr(info, 'sha', '') or '') + or not getattr(info, 'siblings', None)): + raise PublicationError('Expected an initialized public dataset with an exact commit.') + + +def publish(plan, expected_digest, cache): + value = checked(plan, expected_digest) + payloads = [] + for row in value['files']: + raw = read(Path(value['root']) / row['path']) + if digest(raw) != {key: row[key] for key in ('bytes', 'sha256')}: + raise PublicationError('Payload changed while freezing reviewed publication bytes.') + payloads.append((row['path'], raw)) + # No payload file is reopened after credential loading or remote access. + api, token = client(cache) + from huggingface_hub import CommitOperationAdd + from huggingface_hub.errors import RepositoryNotFoundError + try: + info = api.repo_info(REPO, repo_type='dataset', files_metadata=True) + except RepositoryNotFoundError: + api.create_repo(REPO, repo_type='dataset', private=False, exist_ok=False) + info = api.repo_info(REPO, repo_type='dataset', files_metadata=True) + _remote(info) + if len(info.siblings) != 1 or {item.rfilename for item in info.siblings} != {'.gitattributes'}: + raise PublicationError('Dataset is not empty. Verify the existing publication; do not overwrite it.') + operations = [CommitOperationAdd(path_in_repo=name, path_or_fileobj=raw) for name, raw in payloads] + commit = api.create_commit(REPO, repo_type='dataset', operations=operations, + parent_commit=info.sha, + commit_message='Publish PiPER RGB-D tracker v0.1 simulator evidence') + del token, api + return {'repo_id': REPO, 'commit': commit.oid, 'files_added': len(operations)} + + +def verify(plan, expected_digest, cache): + value = checked(plan, expected_digest) + api = public_client() + info = api.repo_info(REPO, repo_type='dataset', files_metadata=True, token=False) + _remote(info) + if (len(info.siblings) != len(NAMES) + 1 + or {item.rfilename for item in info.siblings} != NAMES | {'.gitattributes'}): + raise PublicationError('Public dataset roster differs from the reviewed files.') + for row in value['files']: + raw = download(REPO, row['path'], 'dataset', info.sha, cache) + if digest(raw) != {key: row[key] for key in ('bytes', 'sha256')}: + raise PublicationError('Anonymous public bytes differ from the reviewed artifacts.') + return {'repo_id': REPO, 'commit': info.sha, 'anonymous_files_verified': len(value['files'])} + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + commands = parser.add_subparsers(dest='command', required=True) + command = commands.add_parser('stage') + command.add_argument('root'); command.add_argument('--plan', required=True) + for name in ('publish', 'verify'): + command = commands.add_parser(name) + command.add_argument('--plan', required=True) + command.add_argument('--sha256', required=True) + command.add_argument('--cache', required=True) + args = parser.parse_args() + try: + result = (stage(args.root, args.plan) if args.command == 'stage' else + globals()[args.command](args.plan, args.sha256, Path(args.cache))) + print(json.dumps(result, indent=2)) + return 0 + except PublicationError as exc: + print(str(exc), file=sys.stderr) + except Exception: + print('Publication operation failed; no credential or remote response details displayed.', file=sys.stderr) + return 2 + + +if __name__ == '__main__': + raise SystemExit(main()) diff --git a/publishing/huggingface/test_publish_piper_rgbd_tracker_v01.py b/publishing/huggingface/test_publish_piper_rgbd_tracker_v01.py new file mode 100644 index 0000000..553b08a --- /dev/null +++ b/publishing/huggingface/test_publish_piper_rgbd_tracker_v01.py @@ -0,0 +1,143 @@ +from copy import deepcopy +from hashlib import sha256 +import json +from pathlib import Path +import sys +from types import SimpleNamespace +import tempfile +import unittest +from unittest.mock import Mock, patch + +import publish_piper_rgbd_tracker_v01 as publisher + + +class RgbdPublicationTests(unittest.TestCase): + def setUp(self): + temporary = tempfile.TemporaryDirectory() + self.addCleanup(temporary.cleanup) + self.root = Path(temporary.name).resolve() + self.dataset = self.root / 'dataset'; self.dataset.mkdir() + for name in publisher.NAMES: + (self.dataset / name).write_bytes(b'synthetic RGB-D fixture') + self.plan = self.root / 'plan.json' + self.receipt = publisher.stage(self.dataset, self.plan) + class Operation: + def __init__(self, path_in_repo, path_or_fileobj): + self.path_in_repo, self.path_or_fileobj = path_in_repo, path_or_fileobj + class NotFound(Exception): + pass + sdk = SimpleNamespace(CommitOperationAdd=Operation) + errors = SimpleNamespace(RepositoryNotFoundError=NotFound) + patcher = patch.dict(sys.modules, {'huggingface_hub': sdk, + 'huggingface_hub.errors': errors}) + patcher.start(); self.addCleanup(patcher.stop) + + def info(self, names=('.gitattributes',), private=False, revision='a' * 40): + return SimpleNamespace(private=private, sha=revision, + siblings=[SimpleNamespace(rfilename=name) for name in names]) + + def test_staging_checking_and_public_readback_do_not_load_credentials(self): + api = Mock(); api.repo_info.return_value = self.info(sorted(publisher.NAMES | {'.gitattributes'})) + with patch.object(publisher, 'client', side_effect=AssertionError('Credential accessed')), \ + patch.object(publisher, 'public_client', return_value=api), \ + patch.object(publisher, 'download', side_effect=lambda repo, name, kind, revision, cache: + (self.dataset / name).read_bytes()) as download: + value = publisher.checked(self.plan, self.receipt['plan_sha256']) + proof = publisher.verify(self.plan, self.receipt['plan_sha256'], self.root / 'cache') + self.assertEqual(value['repo_id'], 'Dvidia/piper-rgbd-tracker-v01') + self.assertEqual(proof['anonymous_files_verified'], 6) + self.assertEqual(download.call_count, 6) + api.repo_info.assert_called_once_with(publisher.REPO, repo_type='dataset', + files_metadata=True, token=False) + self.assertTrue(all(call.args[3] == 'a' * 40 for call in download.call_args_list)) + with self.assertRaises(FileExistsError): + publisher.stage(self.dataset, self.plan) + + def test_mutated_payload_and_unreviewed_roster_fail_before_credentials(self): + with patch.object(publisher, 'client') as client: + (self.dataset / 'README.md').write_bytes(b'changed') + with self.assertRaises(publisher.PublicationError): + publisher.publish(self.plan, self.receipt['plan_sha256'], self.root / 'cache') + (self.dataset / 'README.md').write_bytes(b'synthetic RGB-D fixture') + (self.dataset / 'unreviewed.json').write_text('{}') + with self.assertRaises(publisher.PublicationError): + publisher.publish(self.plan, self.receipt['plan_sha256'], self.root / 'cache') + client.assert_not_called() + + def test_resealed_destination_duplicate_and_nonfinite_plans_are_rejected(self): + original = json.loads(self.plan.read_bytes()) + for field, replacement in [('repo_id', 'Dvidia/robot-simulation-lab'), + ('repo_type', 'space'), ('private', True)]: + value = deepcopy(original); value[field] = replacement + raw = publisher.encoded(value); self.plan.write_bytes(raw) + with self.subTest(field=field), self.assertRaises(publisher.PublicationError): + publisher.checked(self.plan, sha256(raw).hexdigest()) + for raw in (b'{"schema":1,"schema":2}', b'{"private":NaN}'): + self.plan.write_bytes(raw) + with self.assertRaises(publisher.PublicationError): + publisher.checked(self.plan, sha256(raw).hexdigest()) + value = deepcopy(original); value['files'][1] = value['files'][0] + raw = publisher.encoded(value); self.plan.write_bytes(raw) + with self.assertRaises(publisher.PublicationError): + publisher.checked(self.plan, sha256(raw).hexdigest()) + + def test_existing_private_or_unpinned_dataset_cannot_be_overwritten(self): + for info in (self.info(['README.md']), self.info(private=True), self.info(revision='main')): + api = Mock(); api.repo_info.return_value = info + with self.subTest(info=info), patch.object(publisher, 'client', return_value=(api, 'fixture')): + with self.assertRaises(publisher.PublicationError): + publisher.publish(self.plan, self.receipt['plan_sha256'], self.root / 'cache') + api.create_commit.assert_not_called() + api.create_repo.assert_not_called() + + def test_payload_freeze_and_upload_bytes_are_immutable_across_credential_boundary(self): + original = publisher.read + reads = 0 + def changed(path, *args, **kwargs): + nonlocal reads + if Path(path).name == 'README.md': + reads += 1 + if reads == 2: + return b'changed after review' + return original(path, *args, **kwargs) + with patch.object(publisher, 'read', side_effect=changed), patch.object(publisher, 'client') as client: + with self.assertRaisesRegex(publisher.PublicationError, 'freezing'): + publisher.publish(self.plan, self.receipt['plan_sha256'], self.root / 'cache') + client.assert_not_called() + api = Mock(); api.repo_info.return_value = self.info() + api.create_commit.return_value = SimpleNamespace(oid='b' * 40) + def credential_boundary(_cache): + (self.dataset / 'README.md').write_bytes(b'late mutation not uploaded') + return api, 'fixture' + with patch.object(publisher, 'client', side_effect=credential_boundary): + result = publisher.publish(self.plan, self.receipt['plan_sha256'], self.root / 'cache') + self.assertEqual(result['files_added'], 6) + commit = api.create_commit.call_args + self.assertEqual(commit.kwargs['parent_commit'], 'a' * 40) + uploaded = {operation.path_in_repo: operation.path_or_fileobj + for operation in commit.kwargs['operations']} + self.assertEqual(uploaded['README.md'], b'synthetic RGB-D fixture') + + def test_oversized_file_and_changed_anonymous_bytes_are_rejected(self): + path = self.dataset / publisher.ARCHIVE + with path.open('wb') as stream: + stream.truncate(publisher.LIMITS[publisher.ARCHIVE] + 1) + with self.assertRaisesRegex(publisher.PublicationError, 'oversized'): + publisher.read(path) + (self.dataset / publisher.ARCHIVE).write_bytes(b'synthetic RGB-D fixture') + self.plan.unlink(); self.receipt = publisher.stage(self.dataset, self.plan) + api = Mock(); api.repo_info.return_value = self.info(sorted(publisher.NAMES | {'.gitattributes'})) + with patch.object(publisher, 'client', side_effect=AssertionError('Credential accessed')), \ + patch.object(publisher, 'public_client', return_value=api), \ + patch.object(publisher, 'download', return_value=b'changed bytes'): + with self.assertRaisesRegex(publisher.PublicationError, 'Anonymous public bytes'): + publisher.verify(self.plan, self.receipt['plan_sha256'], self.root / 'cache') + target = self.root / 'unreviewed'; target.write_text('fixture') + (self.dataset / 'README.md').unlink() + (self.dataset / 'README.md').symlink_to(target) + with self.assertRaises(publisher.PublicationError): + publisher.checked(self.plan, self.receipt['plan_sha256']) + + +if __name__ == '__main__': + unittest.main() diff --git a/src/dvidia_training/piper_rgbd_benchmark.py b/src/dvidia_training/piper_rgbd_benchmark.py new file mode 100644 index 0000000..3aac52d --- /dev/null +++ b/src/dvidia_training/piper_rgbd_benchmark.py @@ -0,0 +1,562 @@ +"""Registered static RGB-D fixtures, isolated estimates, then offline scoring. + +This is a perception experiment, not a manipulation rollout. Native poses are +rendering/evaluation fixtures only. Every scheduled ID is journalled before +rendering; incomplete attempts cannot silently become a smaller denominator. +""" +from __future__ import annotations + +import argparse +from dataclasses import asdict +from hashlib import sha256 +import importlib.metadata +import io +import json +import math +from pathlib import Path +import platform +import time + +import numpy as np + +from .piper_action_capture import CaptureConfig, _NativeSession, _json, _png, _record + +SCHEMA = 'dvidia.piper-rgbd-benchmark/v1' +VARIANTS = ('clear', 'partial_occlusion', 'tilt15', 'tilt30', 'absent', + 'occluded', 'ambiguous', 'invalid_depth', 'stale', 'future') +SOURCES = ('piper_rgbd_tracker.py', 'piper_rgbd_benchmark.py', + 'piper_action_capture.py', 'action_demonstration.py', 'piper_task.py', + 'reference_task.py', 'reference_sim.py', 'robot_profiles.py', 'robot_profiles.lock.json') +INPUTS = ('rgb.npy', 'depth.npy', 'input.json') +SIGNALS = ('object_position_estimate_m', 'object_quaternion_estimate_wxyz', + 'object_position_uncertainty_m', 'object_orientation_uncertainty_rad') +TIMING_SCOPE = ('Authored timestamp-ordering fixtures at a paused synthetic scene; ' + 'wall/CPU timers measure offline estimator computation, not physical ' + 'exposure, asynchronous delivery or robot control latency.') + + +def _load(path): + def reject(value): + raise ValueError('Nonfinite JSON value: '+value) + return json.loads(Path(path).read_text(), parse_constant=reject) + + +def _digest(path): + return sha256(Path(path).read_bytes()).hexdigest() + + +def _identity(path): + path = Path(path) + return dict(bytes=path.stat().st_size, sha256=_digest(path)) + + +def hashes(): + root = Path(__file__).parent + return {name: _digest(root / name) for name in SOURCES} + + +def runtime(): + # Distribution metadata does not initialize MuJoCo or a graphics context. + distribution = importlib.metadata.distribution('mujoco') + renderer = distribution.locate_file('mujoco/rendering/classic/renderer.py') + return dict(python=platform.python_version(), numpy=np.__version__, + mujoco=distribution.version, renderer_sha256=_digest(renderer), + system=platform.system(), machine=platform.machine()) + + +def prepare(output): + output = Path(output) + output.mkdir(parents=True, exist_ok=False) + rng = np.random.default_rng(311301) + groups = [] + for index in range(24): + groups.append(dict(id=f'pose{index:02d}', split='development' if index < 8 else 'confirmation', + position_m=[float(rng.uniform(.27, .36)), float(rng.uniform(-.075, .065)), + float(rng.choice([.08, .105, .13]))], + yaw_rad=float(rng.uniform(0, math.pi/2)))) + protocol = dict(schema=SCHEMA, seed=311301, groups=groups, + resolutions=[[160,120], [320,240]], variants=list(VARIANTS), + scope='Authored static rendered fixtures in the native PiPER scene; no task execution or new training actions.', + estimator_inputs='RGB, optical-Z depth, declared camera calibration and causal frame timestamps only.', + expected_available=['clear'], challenge_positive=['partial_occlusion'], + expected_abstain=[v for v in VARIANTS if v not in ('clear','partial_occlusion')], + provisional_gate=dict(clear_coverage_min=.9, clear_position_p95_m_max=.01, + clear_yaw_p95_rad_max=math.radians(10), negative_false_available_max=0), + symmetry='Upright yaw modulo pi/2; tilted cases are unsupported challenges.', + uncertainty='Heuristic scales, not probabilistic coverage guarantees.', + split_unit='All variants and resolutions of a physical pose group stay in one split.', + depth_accuracy_verified=False) + _json(output/'protocol.json', protocol) + return protocol + + +def _schedule(protocol, split): + if split not in ('development', 'confirmation') or protocol.get('schema') != SCHEMA: + raise ValueError('Unknown split or protocol schema') + groups = protocol['groups']; variants = protocol['variants']; resolutions = protocol['resolutions'] + if (len({g['id'] for g in groups}) != len(groups) + or any(g['split'] not in ('development', 'confirmation') for g in groups) + or len(set(variants)) != len(variants) or set(variants) != set(VARIANTS) + or resolutions != [[160,120], [320,240]]): + raise ValueError('Protocol has duplicate groups, unsupported variants or resolutions') + rows = [] + for width, height in resolutions: + for group in groups: + if group['split'] != split: + continue + if not isinstance(group['id'], str) or not group['id'].isalnum(): + raise ValueError('Unsafe pose ID') + for variant in variants: + rows.append(dict(id=f'{group["id"]}-{width}-{variant}', group_id=group['id'], + variant=variant, width=width, height=height, frame_index=len(rows))) + if not rows: + raise ValueError('No scheduled fixtures for split') + return rows + + +def _npy(path, value): + buf=io.BytesIO(); np.save(buf,value,allow_pickle=False); return _record(path,buf.getvalue()) + + +def _error(error): + # Paths and unknown library text are unnecessary for the public receipt. + return dict(type=type(error).__name__, reason='fixture_or_pipeline_operation_failed') + + +def _visual_box(session, position, size, rotation, rgba): + """Render-only geometry, with a segmentation ID so it also exists in depth.""" + scene = session.renderer.scene + if scene.ngeom >= scene.maxgeom: + raise ValueError('Scene geometry capacity exhausted') + geom = scene.geoms[scene.ngeom] + session.mj.mjv_initGeom(geom, session.mj.mjtGeom.mjGEOM_BOX, + np.asarray(size, dtype=np.float64), np.asarray(position, dtype=np.float64), + np.asarray(rotation, dtype=np.float64).reshape(9), np.asarray(rgba, dtype=np.float32)) + # Classic depth rendering enables segmentation/IDCOLOR. A default segid -1 + # would make an RGB-only occluder and invalidate the aligned-depth fixture. + geom.segid = max([g.segid for g in scene.geoms[:scene.ngeom]] + [-1]) + 1 + scene.ngeom += 1 + return geom.segid + + +def _renderer_binding(session, header, metadata): + base_rotation = session.env.data.xmat[session.base].reshape(3,3) + base_position = session.env.data.xpos[session.base] + if (not np.allclose(base_rotation, np.eye(3), atol=1e-9, rtol=0) + or not np.allclose(base_position, 0., atol=1e-9, rtol=0)): + raise ValueError('Authored world-coordinate fixture requires identity fixed robot base') + geom = session.env.block_geom + return dict(header_binding=header['binding'], assets=metadata['assets'], + runtime=metadata['runtime'], camera=metadata['camera'], + calibration=session._calibration(), + base_position_m=base_position.tolist(), base_rotation=base_rotation.tolist(), + object_half_size_m=session.env.model.geom_size[geom].tolist(), + object_rgba=session.env.model.geom_rgba[geom].tolist(), + orientation_equivalence='upright yaw modulo pi/2') + + +def _check_freeze(output): + from .piper_rgbd_tracker import TrackerConfig + frozen = _load(output/'freeze.json') + if (frozen['sources'] != hashes() or frozen['protocol_sha256'] != _digest(output/'protocol.json') + or frozen['config'] != asdict(TrackerConfig()) or frozen['runtime'] != runtime()): + raise ValueError('Confirmation source, protocol, configuration or runtime differs from freeze') + for name, identity in frozen['development_evidence'].items(): + if _identity(output/'development'/name) != identity: + raise ValueError('Development evidence changed after freeze') + return frozen + + +def _render_fixture(session, item, group, folder, baseline, address, original_rgba, binding): + session.env.data.qpos[:] = baseline + session.env.data.qvel[:] = 0. + variant = item['variant']; yaw = group['yaw_rad'] + tilt = math.radians(15 if variant=='tilt15' else 30 if variant=='tilt30' else 0) + q = np.array([math.cos(yaw/2)*math.cos(tilt/2), math.cos(yaw/2)*math.sin(tilt/2), + math.sin(yaw/2)*math.sin(tilt/2), math.sin(yaw/2)*math.cos(tilt/2)]) + session.env.data.qpos[address:address+3] = group['position_m'] + session.env.data.qpos[address+3:address+7] = q + geom = session.env.block_geom + session.env.model.geom_rgba[geom] = original_rgba + if variant == 'absent': session.env.model.geom_rgba[geom,3] = 0. + session.mj.mj_forward(session.env.model, session.env.data) + native_time = float(session.env.data.time) + with session.sim.native_warnings(session.renderer_warnings): + session.renderer.disable_depth_rendering() + session.renderer.update_scene(session.env.data, camera=session.camera) + calibration = session._calibration() + if calibration != binding['calibration']: + raise ValueError('Camera calibration changed inside the fixture set') + base_rotation = session.env.data.xmat[session.base].reshape(3,3) + base_position = session.env.data.xpos[session.base].copy() + world_position = session.env.data.xpos[session.env.block].copy() + object_rotation = base_rotation.T @ session.env.data.xmat[session.env.block].reshape(3,3) + base_q = np.empty(4) + session.mj.mju_mat2Quat(base_q, object_rotation.reshape(9)) + base_yaw = math.atan2(object_rotation[1,0], object_rotation[0,0]) + base_tilt = math.acos(float(np.clip(object_rotation[2,2], -1., 1.))) + optical_world = base_rotation @ np.asarray(calibration['base_from_camera']).reshape(4,4)[:3,:3] + custom_segids = [] + if variant in ('occluded', 'partial_occlusion'): + centre = world_position-.06*optical_world[:,2] + if variant == 'partial_occlusion': centre = centre+.026*optical_world[:,0] + custom_segids.append(_visual_box(session, centre, + [.06 if variant=='occluded' else .025,.06,.002], optical_world, [.4,.43,.46,1])) + if variant == 'ambiguous': + custom_segids.append(_visual_box(session, world_position+np.array([0.,.09,0.]), + [.02]*3, np.eye(3), original_rgba)) + rgb = session.renderer.render().copy() + session.renderer.enable_depth_rendering() + try: depth = session.renderer.render().copy().astype('.1+1e-12: + raise ValueError('Available estimate is stale') + if result.get('base_frame')!='robot_base': raise ValueError('Wrong estimator frame') + for name,size in zip(SIGNALS[:2],(3,4)): + value=result.get(name) + if not isinstance(value,list) or len(value)!=size or not all(type(x) in (int,float) and math.isfinite(x) for x in value): + raise ValueError('Invalid pose output') + q=result[SIGNALS[1]] + if not math.isclose(sum(x*x for x in q),1.,abs_tol=1e-6,rel_tol=0): raise ValueError('Quaternion is not normalized') + # Only upright canonical yaw is in this estimator's declared contract. + if abs(q[1])>1e-8 or abs(q[2])>1e-8: raise ValueError('Output exceeds upright orientation contract') + for name in SIGNALS[2:]: + value=result.get(name) + if type(value) not in (int,float) or not math.isfinite(value) or value<0: raise ValueError('Invalid uncertainty scale') + elif any(result.get(name) is not None for name in SIGNALS): + raise ValueError('Abstention carries a fabricated pose or scale') + nested=result.get('signals') + if not isinstance(nested,dict) or set(nested)!=set(SIGNALS): + raise ValueError('Estimator must supply the exact four nested observation signals') + if result['status']=='available': + age=timing['decision_at_seconds']-timing['captured_at_seconds'] + if type(result.get('age_seconds')) not in (int,float) or result['age_seconds']!=age: + raise ValueError('Top-level estimator age differs from source frame') + for name in SIGNALS: + value=result[name] + expected=dict(status='available',captured_at_seconds=timing['captured_at_seconds'], + available_at_seconds=timing['available_at_seconds'],age_seconds=age, + value=value if isinstance(value,list) else [value]) + actual=nested[name] + numeric_fields=('captured_at_seconds','available_at_seconds','age_seconds') + if (not isinstance(actual,dict) or actual!=expected + or any(type(actual.get(key)) not in (int,float) for key in numeric_fields) + or not isinstance(actual.get('value'),list) + or any(type(value) not in (int,float) for value in actual['value'])): + raise ValueError('Nested observation signal differs from top-level pose, scale or source timing') + else: + expected=dict(status='missing',captured_at_seconds=None,available_at_seconds=None,age_seconds=None,value=None) + if any(nested[name]!=expected for name in SIGNALS): + raise ValueError('Abstention nested signals must be entirely missing/null') + + +def estimate(output,split): + """Only named image, depth and calibration files cross the estimator boundary.""" + from .piper_rgbd_tracker import estimate_pose + output=Path(output);folder=output/split + frozen=_check_freeze(output) if split=='confirmation' else None + _,manifest=_manifest(output,split) + if frozen is not None and manifest['renderer_bindings']!=frozen['renderer_bindings']: + raise ValueError('Confirmation rendering differs from frozen bindings') + destination=folder/'predictions.json';journal=folder/'estimates' + if destination.exists() or journal.exists(): raise FileExistsError('Estimation already started; never retry or overwrite an evaluation') + journal.mkdir();predictions=[];interrupted=False + _json(journal/'started.json',dict(schema=SCHEMA,manifest_sha256=_digest(folder/'manifest.json'),sources=hashes())) + for item in manifest['records']: + before=time.perf_counter();result=None;error=None;status='complete';input_ids={} + estimate_wall=None;estimate_cpu=None;call_started=None;cpu_started=None + if item['render_status']!='rendered' or interrupted: + status='unattempted';error=dict(type='UnavailableFixture',reason='render_failed_or_preceding_interruption') + else: + try: + _verify_files(folder,item,INPUTS) + data=folder/item['id'];inputs=_load(data/'input.json') + input_ids={name:item['files'][name] for name in INPUTS} + rgb=np.load(data/'rgb.npy',allow_pickle=False);depth=np.load(data/'depth.npy',allow_pickle=False) + _json(journal/(item['id']+'-started.json'),dict(id=item['id'],inputs=input_ids)) + call_started=time.perf_counter();cpu_started=time.process_time() + result=estimate_pose(rgb,depth,inputs['calibration'],**inputs['timing']) + estimate_wall=time.perf_counter()-call_started;estimate_cpu=time.process_time()-cpu_started + _valid_result(result,inputs['timing']) + except (Exception,KeyboardInterrupt) as exc: + status='failed';error=_error(exc);result=None + if call_started is not None and estimate_wall is None: + estimate_wall=time.perf_counter()-call_started;estimate_cpu=time.process_time()-cpu_started + if isinstance(exc,KeyboardInterrupt): interrupted=True + row=dict(id=item['id'],estimate_status=status,result=result,error=error, + wall_seconds=estimate_wall,cpu_seconds=estimate_cpu,pipeline_wall_seconds=time.perf_counter()-before,inputs=input_ids) + _json(journal/(item['id']+'.json'),row) + predictions.append(dict(**row,receipt_sha256=_digest(journal/(item['id']+'.json')))) + _json(destination,dict(schema=SCHEMA,split=split,source_hashes=hashes(),runtime=runtime(), + manifest_sha256=_digest(folder/'manifest.json'),protocol_sha256=_digest(output/'protocol.json'), + freeze_sha256=_digest(output/'freeze.json') if frozen is not None else None,predictions=predictions, + timing_scope=TIMING_SCOPE)) + return dict(scheduled=len(predictions),estimated=sum(r['estimate_status']=='complete' for r in predictions),split=split, + pipeline_complete=all(r['estimate_status']=='complete' for r in predictions)) + + +def score(output,split): + """Identity-checked offline truth join; never call the estimator or simulator.""" + output=Path(output);folder=output/split;protocol,manifest=_manifest(output,split) + scorer_sources=hashes() + prediction_path=folder/'predictions.json';predictions=_load(prediction_path) + if (predictions['schema']!=SCHEMA or predictions['split']!=split + or predictions['manifest_sha256']!=_digest(folder/'manifest.json') + or predictions['protocol_sha256']!=_digest(output/'protocol.json')): + raise ValueError('Prediction binding differs from sealed fixtures') + if split=='confirmation': + frozen=_check_freeze(output) + if (predictions['freeze_sha256']!=_digest(output/'freeze.json') + or predictions['source_hashes']!=frozen['sources'] or predictions['runtime']!=frozen['runtime'] + or manifest['source_hashes']!=frozen['sources'] or manifest['runtime']!=frozen['runtime'] + or manifest['renderer_bindings']!=frozen['renderer_bindings']): + raise ValueError('Confirmation evidence differs from frozen candidate') + if [p['id'] for p in predictions['predictions']]!=[r['id'] for r in manifest['records']]: + raise ValueError('Predictions must cover every scheduled ID exactly once in order') + rows=[] + for pred,item in zip(predictions['predictions'],manifest['records']): + receipt=folder/'estimates'/(pred['id']+'.json') + if _digest(receipt)!=pred['receipt_sha256'] or _load(receipt)!={k:v for k,v in pred.items() if k!='receipt_sha256'}: + raise ValueError('Estimate journal changed') + _verify_files(folder,item,item['files']) + if pred['estimate_status'] not in ('complete','failed','unattempted'): raise ValueError('Unknown estimate status') + result=pred['result'];available=False;error=None;yaw_error=None;pos_scale=None;angle_scale=None + if pred['estimate_status']=='complete': + if item['render_status']!='rendered' or pred['inputs']!={name:item['files'][name] for name in INPUTS}: + raise ValueError('Completed estimate lacks exact rendered input identities') + inputs=_load(folder/item['id']/'input.json');_valid_result(result,inputs['timing']) + truth=_load(folder/item['id']/'truth.json');available=result['status']=='available' + if available and truth['present']: + error=float(np.linalg.norm(np.asarray(result['object_position_estimate_m'])-truth['position_m'])) + q=result['object_quaternion_estimate_wxyz'];yaw=2*math.atan2(q[3],q[0]);delta=yaw-truth['yaw_rad'] + # Native acos(R[2,2]) can report ~1e-8 rad for an upright + # matrix because of roundoff. Classify only declared upright + # fixtures, with a one-microradian numerical tolerance. + upright=item['variant'] not in ('tilt15','tilt30') and abs(truth['tilt_rad'])<=1e-6 + yaw_error=abs((delta+math.pi/4)%(math.pi/2)-math.pi/4) if upright else None + pos_scale=result['object_position_uncertainty_m'];angle_scale=result['object_orientation_uncertainty_rad'] + elif result is not None: raise ValueError('Failed estimate contains a result') + rows.append(dict(**item,estimate_status=pred['estimate_status'],estimate_error=pred['error'], + status=result['status'] if result else 'pipeline_failure',reason=result['reason'] if result else pred['error']['reason'], + available=available,position_error_m=error,yaw_error_rad=yaw_error, + position_heuristic_scale_m=pos_scale,orientation_heuristic_scale_rad=angle_scale, + position_error_within_heuristic_scale=(error<=pos_scale) if error is not None else None, + yaw_error_within_heuristic_scale=(yaw_error<=angle_scale) if yaw_error is not None else None, + wall_seconds=pred['wall_seconds'],cpu_seconds=pred['cpu_seconds'])) + summaries=[];gates=[] + def percentile(group,key,p): + values=[r[key] for r in group if r[key] is not None] + return float(np.percentile(values,p)) if values else None + for width,_ in protocol['resolutions']: + for variant in protocol['variants']: + group=[r for r in rows if r['width']==width and r['variant']==variant] + summaries.append(dict(width=width,variant=variant,scheduled=len(group),available=sum(r['available'] for r in group), + pipeline_failures=sum(r['estimate_status']!='complete' for r in group), + position_evaluated=sum(r['position_error_m'] is not None for r in group), + yaw_evaluated=sum(r['yaw_error_rad'] is not None for r in group), + position_p50_m=percentile(group,'position_error_m',50),position_p95_m=percentile(group,'position_error_m',95), + yaw_p95_rad=percentile(group,'yaw_error_rad',95),wall_p50_s=percentile(group,'wall_seconds',50), + wall_p95_s=percentile(group,'wall_seconds',95),cpu_p95_s=percentile(group,'cpu_seconds',95), + heuristic_comparisons=sum(r['position_error_m'] is not None for r in group), + yaw_heuristic_comparisons=sum(r['yaw_error_rad'] is not None for r in group), + position_error_within_heuristic_scale=sum(r['position_error_within_heuristic_scale'] is True for r in group), + yaw_error_within_heuristic_scale=sum(r['yaw_error_within_heuristic_scale'] is True for r in group))) + clear=[r for r in rows if r['width']==width and r['variant']=='clear'] + negative=[r for r in rows if r['width']==width and r['variant'] in protocol['expected_abstain']] + limits=protocol['provisional_gate'];coverage=sum(r['available'] for r in clear)/len(clear) + position=percentile(clear,'position_error_m',95);yaw=percentile(clear,'yaw_error_rad',95) + false_available=sum(r['available'] for r in negative) + complete=not manifest['session_errors'] and all(r['estimate_status']=='complete' for r in rows if r['width']==width) + checks=dict(all_scheduled_fixtures_complete=complete, + clear_pose_metrics_complete=all(r['position_error_m'] is not None and r['yaw_error_rad'] is not None + for r in clear if r['available']), + clear_coverage=coverage>=limits['clear_coverage_min'], + clear_position=position is not None and position<=limits['clear_position_p95_m_max'], + clear_yaw=yaw is not None and yaw<=limits['clear_yaw_p95_rad_max'], + no_negative_false_available=false_available<=limits['negative_false_available_max']) + gates.append(dict(width=width,passed=all(checks.values()),checks=checks,clear_scheduled=len(clear), + clear_coverage=coverage,negative_scheduled=len(negative),negative_false_available=false_available)) + report=dict(schema=SCHEMA,split=split,scorer_source_hashes=scorer_sources, + predictions_sha256=_digest(prediction_path),manifest_sha256=_digest(folder/'manifest.json'), + rows=rows,summary=summaries,gates=gates,scheduled=len(rows),timing_scope=TIMING_SCOPE, + uncertainty_scope='Descriptive error <= heuristic scale counts only; not calibrated probability or guaranteed bounds. Yaw comparison applies only to upright present objects.', + scope='Static simulated known-cube perception; no policy, action-learning or physical-camera/robot qualification.', + physical_robot_ready=False,depth_accuracy_verified=False) + if hashes()!=scorer_sources: raise ValueError('Scorer sources changed during offline scoring') + _json(folder/'report.json',report) + return report['summary'] + + +def main(argv=None): + parser=argparse.ArgumentParser(description=__doc__);parser.add_argument('command',choices=['prepare','render','freeze','estimate','score']) + parser.add_argument('--output',required=True,type=Path);parser.add_argument('--assets',type=Path) + parser.add_argument('--split',choices=['development','confirmation'],default='development');args=parser.parse_args(argv) + if args.command=='prepare':result=prepare(args.output) + elif args.command=='render':result=render(args.output,args.assets,args.split) + elif args.command=='freeze':result=freeze(args.output) + elif args.command=='estimate':result=estimate(args.output,args.split) + else:result=score(args.output,args.split) + print(json.dumps(result,allow_nan=False)) + # Missing/refused estimates are legitimate benchmark outcomes. A pipeline + # exception or unattempted fixture is different: retain its receipt and + # denominator, but make shell automation stop instead of reporting success. + return 1 if args.command in ('render','estimate') and not result['pipeline_complete'] else 0 + + +if __name__=='__main__':raise SystemExit(main()) diff --git a/src/dvidia_training/piper_rgbd_tracker.py b/src/dvidia_training/piper_rgbd_tracker.py new file mode 100644 index 0000000..5cc1692 --- /dev/null +++ b/src/dvidia_training/piper_rgbd_tracker.py @@ -0,0 +1,338 @@ +"""A bounded, current-frame RGB-D estimator for one declared upright cube. + +This is a colour/geometry baseline, not a generic object or learned tracker. +It imports no simulation state and never carries an estimate into another frame. +Depth accuracy and the heuristic uncertainty scales are NOT calibrated. Yaw is +identifiable only modulo pi/2 for this unmarked square object. +""" +from __future__ import annotations + +from dataclasses import asdict, dataclass +import math +import time + +import numpy as np + + +SCHEMA = 'dvidia.piper-rgbd-estimate/v1' +CLOCK_TOLERANCE_SECONDS = 1e-12 +SIGNALS = ('object_position_estimate_m', 'object_quaternion_estimate_wxyz', + 'object_position_uncertainty_m', 'object_orientation_uncertainty_rad') +UNCERTAINTY = ('Heuristic geometric resolution/residual scales, not calibrated ' + 'confidence bounds; empirical coverage must be measured separately.') + + +@dataclass(frozen=True) +class TrackerConfig: + """Frozen profile for a 40 mm unmarked orange upright rigid cube.""" + side_m: float = .04 + maximum_age_seconds: float = .1 + minimum_depth_m: float = .05 + maximum_depth_m: float = 3. + minimum_component_pixels: int = 12 + maximum_colour_pixels: int = 20000 + minimum_red_intensity: int = 25 + minimum_green_intensity: int = 8 + minimum_top_pixels: int = 6 + plane_tolerance_m: float = .0007 + maximum_tilt_rad: float = math.radians(6.) + minimum_span_fraction: float = .60 + minimum_visibility: float = .86 + minimum_mask_iou: float = .70 + maximum_depth_residual_m: float = .008 + yaw_grid_steps: int = 180 + + def __post_init__(self): + # This baseline deliberately exposes no free-form colour/size fitting. + # A different profile requires a different frozen configuration/study. + for name, value in asdict(self).items(): + if not _number(value) or value <= 0: + raise ValueError('Tracker configuration values must be positive finite numbers: '+name) + for name in ('minimum_component_pixels', 'maximum_colour_pixels', 'minimum_top_pixels', + 'minimum_red_intensity', 'minimum_green_intensity', 'yaw_grid_steps'): + if type(getattr(self, name)) is not int: + raise ValueError('Tracker pixel/grid counts must be integers.') + if (self.side_m != .04 or self.maximum_age_seconds != .1 + or self.minimum_depth_m >= self.maximum_depth_m + or not 3 <= self.minimum_top_pixels <= self.minimum_component_pixels + or not 8 <= self.minimum_component_pixels <= 10000 + or not self.minimum_component_pixels <= self.maximum_colour_pixels <= 20000 + or not 1 <= self.minimum_green_intensity <= self.minimum_red_intensity <= 255 + or not 90 <= self.yaw_grid_steps <= 720 + or not 0 < self.maximum_tilt_rad <= math.radians(10) + or not 0 < self.minimum_span_fraction <= 1 + or not 0 < self.minimum_visibility <= 1 + or not 0 < self.minimum_mask_iou <= 1): + raise ValueError('Configuration exceeds the bounded upright 40 mm cube profile.') + + +def _number(value): + try: + return type(value) in (int, float) and math.isfinite(value) + except OverflowError: + return False + + +def _empty(frame_index, captured, available, decision): + # Invalid caller input is never echoed into a JSON receipt (NaN included). + result = dict(schema=SCHEMA, status='refused', reason=None, + frame_index=frame_index if type(frame_index) is int and frame_index >= 0 else None, + captured_at_seconds=float(captured) if _number(captured) else None, + available_at_seconds=float(available) if _number(available) else None, + decision_at_seconds=float(decision) if _number(decision) else None, + age_seconds=None, base_frame='robot_base', + orientation_equivalence='upright yaw modulo pi/2; canonical yaw in [0, pi/2)', + uncertainty_meaning=UNCERTAINTY, depth_accuracy_verified=False, + signals={}, diagnostics={}) + for name in SIGNALS: + result[name] = None + result['signals'][name] = dict(status='missing', captured_at_seconds=None, + available_at_seconds=None, age_seconds=None, value=None) + return result + + +def _calibration(calibration, shape, captured, decision): + if not isinstance(calibration, dict): + raise ValueError('calibration_not_mapping') + if calibration.get('depth_units') != 'm' or calibration.get('base_frame') != 'robot_base': + raise ValueError('calibration_units_or_frame') + intrinsics = calibration.get('intrinsics') + if not isinstance(intrinsics, dict): + raise ValueError('intrinsics_missing') + height, width = shape + if (type(intrinsics.get('width')) is not int or type(intrinsics.get('height')) is not int + or (intrinsics['height'], intrinsics['width']) != shape): + raise ValueError('calibration_image_dimensions') + values = [intrinsics.get(key) for key in ('fx', 'fy', 'cx', 'cy')] + if not all(_number(value) for value in values): + raise ValueError('intrinsics_nonfinite') + fx, fy, cx, cy = values + if not (0 < fx < 1e6 and 0 < fy < 1e6 and 0 <= cx <= width and 0 <= cy <= height): + raise ValueError('intrinsics_invalid') + raw = calibration.get('base_from_camera') + if not isinstance(raw, (list, tuple)) or len(raw) != 16 or not all(_number(x) for x in raw): + raise ValueError('transform_invalid') + transform = np.array(raw, dtype=float).reshape(4, 4) + rotation = transform[:3, :3] + if (not np.allclose(transform[3], [0, 0, 0, 1], atol=1e-8, rtol=0) + or not np.allclose(rotation.T @ rotation, np.eye(3), atol=1e-6, rtol=0) + or not math.isclose(float(np.linalg.det(rotation)), 1., abs_tol=1e-6, rel_tol=0) + or np.max(np.abs(transform[:3, 3])) > 100): + raise ValueError('transform_not_rigid') + start, end = calibration.get('valid_from_seconds'), calibration.get('valid_until_seconds') + if not (_number(start) and _number(end) and 0 <= start <= end + and start <= captured <= decision <= end): + raise ValueError('calibration_not_valid_at_observation') + return tuple(float(x) for x in values), transform + + +def _components(mask): + """Small bounded 4-connected components; no optional vision dependency.""" + remaining = mask.copy() + components = [] + height, width = remaining.shape + for row, column in np.argwhere(mask): + if not remaining[row, column]: + continue + remaining[row, column] = False + stack, pixels = [(int(row), int(column))], [] + while stack: + y, x = stack.pop(); pixels.append((y, x)) + for yy, xx in ((y-1, x), (y+1, x), (y, x-1), (y, x+1)): + if 0 <= yy < height and 0 <= xx < width and remaining[yy, xx]: + remaining[yy, xx] = False; stack.append((yy, xx)) + components.append(np.array(pixels, dtype=int)) + return sorted(components, key=len, reverse=True) + + +def _top_plane(points, config): + """Deterministic bounded RANSAC, then an ordinary plane fit.""" + candidates = points[points[:, 2] >= np.median(points[:, 2])] + if len(candidates) < config.minimum_top_pixels: + return None + # Fixed indices, no state, no per-frame random search or adaptation. + rng = np.random.default_rng(701) + triples = rng.integers(0, len(candidates), (96, 3)) + best = None + for indices in triples: + a, b, c = candidates[indices] + normal = np.cross(b-a, c-a) + norm = np.linalg.norm(normal) + if norm < 1e-10: + continue + normal /= norm + if normal[2] < 0: + normal *= -1 + if normal[2] < math.cos(math.radians(45)): + continue + residual = np.abs((points-a) @ normal) + selected = residual <= config.plane_tolerance_m + count = int(selected.sum()) + # The high, broad face is the cube's candidate top. A plane through + # several silhouette samples is insufficient evidence on its own. + if count >= config.minimum_top_pixels and (best is None or count > best[0]): + best = count, points[selected] + if best is None: + return None + face = best[1] + centre = face.mean(axis=0) + _, singular, vectors = np.linalg.svd(face-centre, full_matrices=False) + if len(singular) < 3 or singular[1] < .001: + return None + normal = vectors[-1] + if normal[2] < 0: + normal *= -1 + residual = np.abs((face-centre) @ normal) + return face, normal, float(np.sqrt(np.mean(residual**2))) + + +def _rectangle(face, steps): + xy = face[:, :2] + angles = np.arange(steps) * (math.pi/2/steps) + co, si = np.cos(angles), np.sin(angles) + first = xy[:, 0, None]*co + xy[:, 1, None]*si + second = -xy[:, 0, None]*si + xy[:, 1, None]*co + lo_u, hi_u = first.min(axis=0), first.max(axis=0) + lo_v, hi_v = second.min(axis=0), second.max(axis=0) + widths = hi_u-lo_u; heights = hi_v-lo_v + # Minimum rectangle area is invariant to translations and square symmetry. + index = int(np.argmin(widths*heights)) + u, v = (lo_u[index]+hi_u[index])/2, (lo_v[index]+hi_v[index])/2 + centre = np.array([u*co[index]-v*si[index], u*si[index]+v*co[index]]) + return centre, float(angles[index]), float(widths[index]), float(heights[index]) + + +def _cube_rays(shape, intrinsics, transform, centre, yaw, side): + """Expected pixel-centre ray hits on the declared opaque geometric cube.""" + height, width = shape; fx, fy, cx, cy = intrinsics + rows, columns = np.indices(shape) + optical = np.stack(((columns+.5-cx)/fx, (rows+.5-cy)/fy, np.ones(shape)), axis=-1) + co, si = math.cos(yaw), math.sin(yaw) + rotation = np.array([[co, -si, 0], [si, co, 0], [0, 0, 1.]]) + origin = (transform[:3, 3]-centre) @ rotation + rays = optical @ transform[:3, :3].T @ rotation + half = side/2 + parallel = np.abs(rays) < 1e-12 + safe = np.where(parallel, 1., rays) + a, b = (-half-origin)/safe, (half-origin)/safe + near, far = np.minimum(a, b), np.maximum(a, b) + inside = np.abs(origin) <= half + near = np.where(parallel, np.where(inside, -np.inf, np.inf), near) + far = np.where(parallel, np.where(inside, np.inf, -np.inf), far) + enter, leave = near.max(axis=-1), far.min(axis=-1) + valid = (enter > 0) & (leave >= enter) + return valid, np.where(valid, enter, 0.) + + +def estimate_pose(rgb, depth, calibration, *, frame_index, captured_at_seconds, + available_at_seconds, decision_at_seconds, config=TrackerConfig()): + """Return current-frame evidence or explicit abstention, never native truth. + + Inputs are aligned RGB uint8 H×W×3 and floating camera-Z metres H×W; + the camera is right/down/forward, samples at column+.5,row+.5. The caller + supplies actual availability, not this function's local processing timer. + Uncertainty scalars are top-level scalars and size-one vectors in signals. + The 100 ms age boundary permits only 1e-12 s of floating-point tolerance; + timestamp ordering remains strict. This is not a delivery-latency allowance. + """ + began = time.perf_counter() + result = _empty(frame_index, captured_at_seconds, available_at_seconds, decision_at_seconds) + + def finish(status, reason): + result['status'] = status; result['reason'] = reason + result['diagnostics']['processing_seconds'] = time.perf_counter()-began + return result + + if not isinstance(config, TrackerConfig): + return finish('refused', 'invalid_configuration') + if type(frame_index) is not int or frame_index < 0: + return finish('refused', 'invalid_frame_index') + times = (captured_at_seconds, available_at_seconds, decision_at_seconds) + if not all(_number(t) and t >= 0 for t in times): + return finish('refused', 'invalid_timestamps') + captured, available, decision = map(float, times) + if not captured <= available <= decision: + return finish('refused', 'noncausal_timestamps') + result['age_seconds'] = decision-captured + if decision-captured > config.maximum_age_seconds+CLOCK_TOLERANCE_SECONDS: + return finish('stale', 'frame_older_than_100ms') + if (not isinstance(rgb, np.ndarray) or rgb.dtype != np.uint8 or rgb.ndim != 3 + or rgb.shape[-1] != 3 or not 16 <= rgb.shape[0] <= 480 + or not 16 <= rgb.shape[1] <= 640): + return finish('refused', 'invalid_rgb_array') + shape = rgb.shape[:2] + if (not isinstance(depth, np.ndarray) or depth.dtype.kind != 'f' + or depth.dtype.itemsize not in (4, 8) or depth.shape != shape): + return finish('refused', 'invalid_aligned_depth_array') + if not np.isfinite(depth).all() or np.any(depth <= 0): + return finish('refused', 'nonfinite_or_nonpositive_depth') + try: + intrinsics, transform = _calibration(calibration, shape, captured, decision) + except (ValueError, TypeError, OverflowError) as error: + return finish('refused', str(error) if type(error) is ValueError else 'invalid_calibration') + # Declared orange profile: ratios tolerate uniform shading, not arbitrary + # lighting/material changes. Convert before multiplication to avoid overflow. + colour = rgb.astype(float); red, green, blue = np.moveaxis(colour, -1, 0) + # Keep the chromatic ratios fixed under shading. The positive brightness + # floors exclude near-black quantization noise; a shadow is not occlusion. + mask = ((red >= config.minimum_red_intensity) & (green >= config.minimum_green_intensity) & (red >= green*1.35) + & (red >= blue*1.8) & (green >= blue*1.15)) + result['diagnostics']['orange_pixels'] = int(mask.sum()) + if result['diagnostics']['orange_pixels'] > config.maximum_colour_pixels: + return finish('missing', 'colour_support_exceeds_bounded_profile') + components = _components(mask) + usable = [component for component in components if len(component) >= config.minimum_component_pixels] + result['diagnostics']['candidate_components'] = len(usable) + if not usable: + return finish('missing', 'object_absent_or_too_small') + if len(usable) != 1: + return finish('missing', 'ambiguous_colour_components') + pixels = usable[0]; rows, columns = pixels.T + if (np.any(rows == 0) or np.any(rows == shape[0]-1) + or np.any(columns == 0) or np.any(columns == shape[1]-1)): + return finish('missing', 'object_cropped_at_image_boundary') + z = depth[rows, columns].astype(float) + if np.any(z < config.minimum_depth_m) or np.any(z > config.maximum_depth_m): + return finish('missing', 'object_outside_usable_depth') + fx, fy, cx, cy = intrinsics + optical = np.stack(((columns+.5-cx)*z/fx, (rows+.5-cy)*z/fy, z), axis=-1) + points = optical @ transform[:3, :3].T + transform[:3, 3] + plane = _top_plane(points, config) + if plane is None: + return finish('missing', 'insufficient_top_plane') + face, normal, residual = plane + tilt = math.acos(float(np.clip(normal[2], -1, 1))) + result['diagnostics'].update(top_pixels=len(face), top_tilt_rad=tilt, plane_rms_m=residual) + if tilt > config.maximum_tilt_rad: + return finish('missing', 'outside_upright_profile') + xy, yaw, width, length = _rectangle(face, config.yaw_grid_steps) + result['diagnostics'].update(top_sample_spans_m=[width, length]) + if (min(width, length) < config.side_m*config.minimum_span_fraction + or max(width, length) > config.side_m*1.15): + return finish('missing', 'incomplete_or_wrong_size_top_surface') + centre = np.array([*xy, float(np.median(face[:, 2]))-config.side_m/2]) + expected, expected_z = _cube_rays(shape, intrinsics, transform, centre, yaw, config.side_m) + actual = np.zeros(shape, dtype=bool); actual[rows, columns] = True + intersection = expected & actual + predicted_count = int(expected.sum()); observed_count = len(pixels) + visibility = float(intersection.sum()/max(predicted_count, 1)) + iou = float(intersection.sum()/max(int((expected | actual).sum()), 1)) + depth_error = np.abs(depth[intersection]-expected_z[intersection]) + depth_p95 = float(np.quantile(depth_error, .95)) if depth_error.size else None + result['diagnostics'].update(predicted_pixels=predicted_count, component_pixels=observed_count, + visibility_fraction=visibility, mask_iou=iou, model_depth_p95_m=depth_p95) + if (predicted_count < config.minimum_component_pixels or visibility < config.minimum_visibility + or iou < config.minimum_mask_iou or depth_p95 is None + or depth_p95 > config.maximum_depth_residual_m): + return finish('missing', 'occluded_or_inconsistent_cube_geometry') + pixel_scale = float(np.median(z)/min(fx, fy)) + position_scale = math.sqrt(3)*pixel_scale/2 + residual + orientation_scale = min(math.pi/4, 2*pixel_scale/config.side_m + math.pi/2/config.yaw_grid_steps) + values = (centre.tolist(), [math.cos(yaw/2), 0., 0., math.sin(yaw/2)], + position_scale, orientation_scale) + for name, value in zip(SIGNALS, values): + result[name] = value + result['signals'][name] = dict(status='available', captured_at_seconds=captured, + available_at_seconds=available, age_seconds=decision-captured, + value=value if isinstance(value, list) else [value]) + return finish('available', 'current_frame_upright_cube_estimate') diff --git a/tests/test_piper_rgbd_benchmark.py b/tests/test_piper_rgbd_benchmark.py new file mode 100644 index 0000000..e3e1654 --- /dev/null +++ b/tests/test_piper_rgbd_benchmark.py @@ -0,0 +1,316 @@ +"""Runner contracts with fake renderer/estimator only; no native simulation.""" +from __future__ import annotations + +from contextlib import ExitStack, redirect_stdout +import io +from copy import deepcopy +import json +from pathlib import Path +from tempfile import TemporaryDirectory +from types import SimpleNamespace +import unittest +from unittest.mock import patch + +import numpy as np + +from dvidia_training import piper_rgbd_benchmark as bench + + +def with_signals(result): + result['age_seconds']=result['decision_at_seconds']-result['captured_at_seconds'] + result['signals']={} + for name in bench.SIGNALS: + if result['status']=='available': + value=result[name] + result['signals'][name]=dict(status='available',captured_at_seconds=result['captured_at_seconds'], + available_at_seconds=result['available_at_seconds'],age_seconds=result['age_seconds'], + value=value if isinstance(value,list) else [value]) + else: + result['signals'][name]=dict(status='missing',captured_at_seconds=None, + available_at_seconds=None,age_seconds=None,value=None) + return result + + +def missing_result(rgb, depth, calibration, **timing): + return with_signals(dict(status='missing',reason='fake_test_abstention',**timing, + **{name:None for name in bench.SIGNALS})) + + +class FakeSession: + failures = set() + closes = [] + + def __init__(self, assets, config): + self.config=config; self.renderer_warnings=[] + self.env=SimpleNamespace(data=SimpleNamespace(qpos=np.zeros(7)),block_geom=0, + model=SimpleNamespace(jnt_qposadr=np.array([0]),geom_rgba=np.array([[.75,.3,.16,1.]]))) + self.mj=SimpleNamespace(mjtObj=SimpleNamespace(mjOBJ_JOINT=0),mj_name2id=lambda *args:0) + + def open(self): + if self.config.width in self.failures: raise RuntimeError('fake context failure') + return {'binding':{}},None,{'assets':{},'runtime':{},'camera':{}} + + def close(self): self.closes.append(self.config.width) + + +def fake_binding(session, header, metadata): + return dict(resolution=session.config.width,calibration={'id':'fake-calibration'}, + object_half_size_m=[.02]*3,object_rgba=[.75,.3,.16,1.]) + + +def fake_render(session,item,group,folder,baseline,address,rgba,binding): + rgb=np.zeros((16,16,3),dtype=np.uint8);depth=np.ones((16,16),dtype=np.float32) + bench._record(folder/'rgb.png',bench._png(rgb));bench._npy(folder/'rgb.npy',rgb);bench._npy(folder/'depth.npy',depth) + timing=dict(frame_index=item['frame_index'],captured_at_seconds=.2,available_at_seconds=.2,decision_at_seconds=.2) + bench._json(folder/'input.json',dict(calibration=binding['calibration'],timing=timing)) + bench._json(folder/'truth.json',dict(position_m=[.3,0,.1],quaternion_wxyz=[1,0,0,0],yaw_rad=0.,tilt_rad=0.,present=item['variant']!='absent')) + + +class RunnerContracts(unittest.TestCase): + def setUp(self): + self.temp=TemporaryDirectory();self.addCleanup(self.temp.cleanup) + self.root=Path(self.temp.name)/'study';bench.prepare(self.root) + self.protocol=bench._load(self.root/'protocol.json') + # Two independent groups, ten variants, two resolutions: 20 IDs per split. + self.protocol['groups']=[self.protocol['groups'][0],self.protocol['groups'][8]] + self.write(self.root/'protocol.json',self.protocol) + self.stack=ExitStack();self.addCleanup(self.stack.close) + self.stack.enter_context(patch.object(bench,'_NativeSession',FakeSession)) + self.stack.enter_context(patch.object(bench,'_renderer_binding',fake_binding)) + self.stack.enter_context(patch.object(bench,'_render_fixture',fake_render)) + self.stack.enter_context(patch.object(bench,'runtime',lambda:{'runtime':'test'})) + self.stack.enter_context(patch.object(bench,'hashes',lambda:{'source':'test'})) + FakeSession.failures=set();FakeSession.closes=[] + + @staticmethod + def write(path,value): path.write_text(json.dumps(value,allow_nan=False)+'\n') + + def render(self,split='development'): return bench.render(self.root,'unused',split) + + def estimate(self,split='development',estimator=missing_result): + with patch('dvidia_training.piper_rgbd_tracker.estimate_pose',estimator): + return bench.estimate(self.root,split) + + def score(self,split='development'): return bench.score(self.root,split) + + def ready(self): self.render();self.estimate();self.score() + + def test_predeclared_schedule_preserves_fixture_failure(self): + calls=[] + def fail_one(session,item,*args): + calls.append(item['id']) + if item['frame_index']==1: raise ValueError('fixture failed') + return fake_render(session,item,*args) + with patch.object(bench,'_render_fixture',fail_one): result=self.render() + self.assertEqual(result,dict(scheduled=20,rendered=19,split='development',pipeline_complete=False)) + self.assertEqual(len(calls),20) + manifest=bench._load(self.root/'development/manifest.json') + self.assertEqual(len(manifest['records']),20) + self.assertEqual(manifest['records'][1]['render_status'],'failed') + self.assertTrue((self.root/'development'/manifest['records'][1]['id']/'started.json').exists()) + self.estimate();self.score() + report=bench._load(self.root/'development/report.json') + self.assertEqual(report['scheduled'],20) + self.assertEqual(sum(r['status']=='pipeline_failure' for r in report['rows']),1) + self.assertFalse(report['gates'][0]['checks']['all_scheduled_fixtures_complete']) + + def test_setup_failure_keeps_all_ids_and_other_resolution(self): + FakeSession.failures={160};self.assertEqual(self.render()['rendered'],10) + rows=bench._load(self.root/'development/manifest.json')['records'] + self.assertEqual([r['render_status'] for r in rows[:10]],['unattempted']*10) + self.assertEqual([r['render_status'] for r in rows[10:]],['rendered']*10) + self.assertEqual(FakeSession.closes,[160,320]) + + def test_interrupt_never_retries_remaining_fixtures(self): + with patch.object(bench,'_render_fixture',side_effect=KeyboardInterrupt): + result=self.render() + rows=bench._load(self.root/'development/manifest.json')['records'] + self.assertEqual(result['rendered'],0) + self.assertEqual(rows[0]['render_status'],'failed') + self.assertTrue(all(r['render_status']=='unattempted' for r in rows[1:])) + self.assertEqual(FakeSession.closes,[160]) + + def test_estimator_exception_preserved_and_reinvocation_refused(self): + self.render();calls=[] + def estimator(*args,**kwargs): + calls.append(kwargs['frame_index']) + if kwargs['frame_index']==2:raise RuntimeError('failure') + return missing_result(*args,**kwargs) + result=self.estimate(estimator=estimator) + self.assertEqual(result['estimated'],19);self.assertEqual(len(calls),20) + self.score() + self.assertEqual(sum(s['pipeline_failures'] for s in bench._load(self.root/'development/report.json')['summary']),1) + with self.assertRaises(FileExistsError):self.estimate() + + def test_truth_is_never_read_by_estimate(self): + self.render();original_load=bench._load;opened=[] + def spy(path): + opened.append(Path(path).name) + if Path(path).name=='truth.json':raise AssertionError('leaked truth') + return original_load(path) + with patch.object(bench,'_load',spy):self.estimate() + self.assertNotIn('truth.json',opened) + + def test_missing_duplicate_and_unknown_prediction_ids_refused(self): + self.render();self.estimate();path=self.root/'development/predictions.json';original=bench._load(path) + alternatives=[original['predictions'][:-1], original['predictions']+[original['predictions'][0]], + [dict(original['predictions'][0],id='unknown')]+original['predictions'][1:]] + for alternative in alternatives: + self.write(path,dict(original,predictions=alternative)) + with self.assertRaisesRegex(ValueError,'every scheduled ID'):self.score() + + def test_changed_truth_refuses_offline_score(self): + self.render();self.estimate();item=bench._load(self.root/'development/manifest.json')['records'][0] + path=self.root/'development'/item['id']/'truth.json';truth=bench._load(path);truth['position_m'][0]+=1 + self.write(path,truth) + with self.assertRaisesRegex(ValueError,'artifact changed'):self.score() + + def test_changed_image_is_not_reestimated(self): + self.render();item=bench._load(self.root/'development/manifest.json')['records'][0] + path=self.root/'development'/item['id']/'rgb.npy';path.write_bytes(b'altered') + result=self.estimate();self.assertEqual(result['estimated'],19) + with self.assertRaisesRegex(ValueError,'artifact changed'):self.score() + + def test_changed_prediction_receipt_refuses_score(self): + self.render();self.estimate();pred=bench._load(self.root/'development/predictions.json')['predictions'][0] + (self.root/'development/estimates'/(pred['id']+'.json')).write_text('{}') + with self.assertRaisesRegex(ValueError,'journal changed'):self.score() + + def test_freeze_checks_protocol_runtime_and_calibration(self): + self.ready();bench.freeze(self.root) + with patch.object(bench,'runtime',lambda:{'runtime':'different'}): + with self.assertRaisesRegex(ValueError,'differs from freeze'):self.render('confirmation') + protocol=deepcopy(self.protocol);protocol['seed']=999;self.write(self.root/'protocol.json',protocol) + with self.assertRaisesRegex(ValueError,'differs from freeze'):self.render('confirmation') + self.write(self.root/'protocol.json',self.protocol) + with patch.object(bench,'_renderer_binding',lambda *args:dict(fake_binding(*args),calibration={'id':'changed'})): + result=self.render('confirmation') + self.assertEqual(result['rendered'],0) + with self.assertRaisesRegex(ValueError,'rendering differs'):self.estimate('confirmation') + + def test_frozen_confirmation_and_descriptive_gate(self): + self.ready();bench.freeze(self.root);self.render('confirmation');self.estimate('confirmation');self.score('confirmation') + report=bench._load(self.root/'confirmation/report.json') + self.assertEqual(report['scheduled'],20) + self.assertTrue(all(g['checks']['no_negative_false_available'] for g in report['gates'])) + self.assertTrue(all(not g['passed'] for g in report['gates'])) + self.assertFalse(report['physical_robot_ready']) + with self.assertRaisesRegex(ValueError,'after confirmation'):bench.freeze(self.root) + + def test_false_available_prevents_gate_even_when_clear_passes(self): + self.render() + def available(*args,**timing): + return with_signals(dict(status='available',reason='fake',base_frame='robot_base',**timing, + object_position_estimate_m=[.3,0,.1],object_quaternion_estimate_wxyz=[1.,0,0,0], + object_position_uncertainty_m=.001,object_orientation_uncertainty_rad=.01)) + self.estimate(estimator=available);self.score() + report=bench._load(self.root/'development/report.json') + self.assertTrue(all(g['checks']['clear_position'] for g in report['gates'])) + self.assertTrue(all(g['checks']['clear_coverage'] for g in report['gates'])) + self.assertTrue(all(not g['passed'] for g in report['gates'])) + clear=next(s for s in report['summary'] if s['variant']=='clear') + self.assertEqual(clear['position_error_within_heuristic_scale'],1) + + def test_upright_native_roundoff_keeps_yaw_denominator(self): + def fixture(session,item,group,folder,*args): + fake_render(session,item,group,folder,*args) + truth=bench._load(folder/'truth.json');truth['tilt_rad']=1.49e-8 + self.write(folder/'truth.json',truth) + with patch.object(bench,'_render_fixture',fixture):self.render() + def available(*args,**timing): + if timing['frame_index']%10: return missing_result(*args,**timing) + return with_signals(dict(status='available',reason='fake',base_frame='robot_base',**timing, + object_position_estimate_m=[.3,0,.1],object_quaternion_estimate_wxyz=[1.,0,0,0], + object_position_uncertainty_m=.001,object_orientation_uncertainty_rad=.01)) + self.estimate(estimator=available);self.score() + report=bench._load(self.root/'development/report.json') + self.assertTrue(all(s['yaw_evaluated']==s['available'] for s in report['summary'] if s['variant']=='clear')) + self.assertTrue(all(g['checks']['clear_pose_metrics_complete'] for g in report['gates'])) + self.assertTrue(all(g['passed'] for g in report['gates'])) + + def test_unscorable_available_clear_pose_fails_gate(self): + def fixture(session,item,group,folder,*args): + fake_render(session,item,group,folder,*args) + truth=bench._load(folder/'truth.json');truth['tilt_rad']=2e-6 + self.write(folder/'truth.json',truth) + with patch.object(bench,'_render_fixture',fixture):self.render() + def available(*args,**timing): + if timing['frame_index']%10: return missing_result(*args,**timing) + return with_signals(dict(status='available',reason='fake',base_frame='robot_base',**timing, + object_position_estimate_m=[.3,0,.1],object_quaternion_estimate_wxyz=[1.,0,0,0], + object_position_uncertainty_m=.001,object_orientation_uncertainty_rad=.01)) + self.estimate(estimator=available);self.score() + report=bench._load(self.root/'development/report.json') + self.assertTrue(all(not g['checks']['clear_pose_metrics_complete'] for g in report['gates'])) + self.assertTrue(all(not g['passed'] for g in report['gates'])) + + def test_freeze_rejects_predictions_from_older_candidate(self): + self.ready() + with patch.object(bench,'hashes',lambda:{'source':'new-candidate'}): + with self.assertRaisesRegex(ValueError,'predictions do not bind the current'):bench.freeze(self.root) + self.assertFalse((self.root/'freeze.json').exists()) + + def test_freeze_rejects_report_from_other_predictions_or_scorer(self): + self.ready();path=self.root/'development/report.json';original=bench._load(path) + for bad in (dict(original,predictions_sha256='0'*64),dict(original,scorer_source_hashes={'source':'older'})): + self.write(path,bad) + with self.assertRaisesRegex(ValueError,'report does not bind'):bench.freeze(self.root) + self.assertFalse((self.root/'freeze.json').exists()) + + def test_confirmation_score_rejects_changed_current_code(self): + self.ready();bench.freeze(self.root);self.render('confirmation');self.estimate('confirmation') + with patch.object(bench,'hashes',lambda:{'source':'edited-after-freeze'}): + with self.assertRaisesRegex(ValueError,'differs from freeze'):self.score('confirmation') + self.assertFalse((self.root/'confirmation/report.json').exists()) + + def test_nested_observation_contract_rejects_replacement_estimator(self): + timing=dict(frame_index=0,captured_at_seconds=.2,available_at_seconds=.2,decision_at_seconds=.2) + valid=with_signals(dict(status='available',reason='fake',base_frame='robot_base',**timing, + object_position_estimate_m=[.3,0,.1],object_quaternion_estimate_wxyz=[1.,0,0,0], + object_position_uncertainty_m=.001,object_orientation_uncertainty_rad=.01)) + bench._valid_result(valid,timing) + for field,value in (('value',[99.,0.,0.]),('status','missing'),('captured_at_seconds',.1), + ('available_at_seconds',.3),('age_seconds',.1),('age_seconds',False)): + bad=deepcopy(valid);bad['signals'][bench.SIGNALS[0]][field]=value + with self.assertRaisesRegex(ValueError,'Nested observation'):bench._valid_result(bad,timing) + bad=deepcopy(valid);bad['age_seconds']=.1 + with self.assertRaisesRegex(ValueError,'Top-level estimator age'):bench._valid_result(bad,timing) + missing=missing_result(None,None,None,**timing);missing['signals'][bench.SIGNALS[0]]['value']=[0.,0.,0.] + with self.assertRaisesRegex(ValueError,'missing/null'):bench._valid_result(missing,timing) + self.render() + def replacement(*args,**kwargs): + result=missing_result(*args,**kwargs);result.pop('signals');return result + result=self.estimate(estimator=replacement) + self.assertFalse(result['pipeline_complete']);self.assertEqual(result['estimated'],0) + + def test_cli_pipeline_failures_exit_nonzero_with_receipts_retained(self): + FakeSession.failures={160} + with redirect_stdout(io.StringIO()) as output: + code=bench.main(['render','--output',str(self.root),'--assets','unused']) + self.assertEqual(code,1) + self.assertEqual(json.loads(output.getvalue())['scheduled'],20) + self.assertTrue((self.root/'development/manifest.json').exists()) + with patch('dvidia_training.piper_rgbd_tracker.estimate_pose',missing_result): + with redirect_stdout(io.StringIO()) as output: + code=bench.main(['estimate','--output',str(self.root)]) + self.assertEqual(code,1) + self.assertEqual(json.loads(output.getvalue())['scheduled'],20) + self.assertTrue((self.root/'development/predictions.json').exists()) + + def test_cli_valid_abstentions_exit_zero(self): + self.render() + with patch('dvidia_training.piper_rgbd_tracker.estimate_pose',missing_result): + with redirect_stdout(io.StringIO()): + code=bench.main(['estimate','--output',str(self.root)]) + self.assertEqual(code,0) + + def test_custom_geom_kept_in_depth_segmentation(self): + geom0=SimpleNamespace(segid=7);geom1=SimpleNamespace(segid=-1) + scene=SimpleNamespace(ngeom=1,maxgeom=2,geoms=[geom0,geom1]) + session=SimpleNamespace(renderer=SimpleNamespace(scene=scene), + mj=SimpleNamespace(mjtGeom=SimpleNamespace(mjGEOM_BOX=6),mjv_initGeom=lambda *args:None)) + self.assertEqual(bench._visual_box(session,[0,0,0],[.02]*3,np.eye(3),[1]*4),8) + self.assertEqual(scene.ngeom,2);self.assertEqual(geom1.segid,8) + + +if __name__=='__main__':unittest.main() diff --git a/tests/test_piper_rgbd_tracker.py b/tests/test_piper_rgbd_tracker.py new file mode 100644 index 0000000..3d64972 --- /dev/null +++ b/tests/test_piper_rgbd_tracker.py @@ -0,0 +1,213 @@ +"""Analytical camera fixtures, never native ground truth passed to tracker.""" +from copy import deepcopy +from dataclasses import replace +import inspect +import json +import math +import unittest + +import numpy as np + +from dvidia_training import piper_rgbd_tracker as tracker + + +def fixture(*, width=160, height=120, focal=400., centre=(.012, -.009, .12), yaw=.31): + """Independent top-plane raster: top-down camera, known square in base XY. + + A camera at z=.6 points down, with optical right along base Y and down + along base X. Pixel centres intersect the horizontal cube top at z=.14. + No robot state, native masks or estimator helper constructs these frames. + """ + camera = np.array([[0., 1., 0., .01], [1., 0., 0., -.02], [0., 0., -1., .6], [0., 0., 0., 1.]]) + calibration = dict(base_frame='robot_base', camera_frame='rgbd_optical', depth_units='m', + intrinsics=dict(width=width, height=height, fx=focal, fy=focal, cx=width/2, cy=height/2), + base_from_camera=camera.flatten().tolist(), valid_from_seconds=0., valid_until_seconds=2.) + top = centre[2]+.02; z = .6-top + row, column = np.indices((height, width)) + world_x = .01+(row+.5-height/2)*z/focal + world_y = -.02+(column+.5-width/2)*z/focal + x, y = world_x-centre[0], world_y-centre[1] + local_x = x*math.cos(yaw)+y*math.sin(yaw) + local_y = -x*math.sin(yaw)+y*math.cos(yaw) + mask = (np.abs(local_x) < .02) & (np.abs(local_y) < .02) + rgb = np.full((height, width, 3), 90, dtype=np.uint8); rgb[mask] = [191, 77, 41] + depth = np.full((height, width), 2., dtype=np.float32); depth[mask] = z + return rgb, depth, calibration, np.array(centre), yaw, mask + + +def run(rgb, depth, calibration, **kwargs): + return tracker.estimate_pose(rgb, depth, calibration, frame_index=4, + captured_at_seconds=.2, available_at_seconds=.205, decision_at_seconds=.21, **kwargs) + + +class RGBDTrackerTests(unittest.TestCase): + def test_known_square_pixel_centres_base_transform_and_yaw_symmetry(self): + for yaw in (0., .31, .74, 1.30, 1.88, 3.12): + with self.subTest(yaw=yaw): + rgb, depth, calibration, centre, expected_yaw, _ = fixture(yaw=yaw) + result = run(rgb, depth, calibration) + self.assertEqual(result['status'], 'available', result) + np.testing.assert_allclose(result['object_position_estimate_m'], centre, atol=.0015) + quaternion = result['object_quaternion_estimate_wxyz'] + self.assertAlmostEqual(float(np.linalg.norm(quaternion)), 1.) + estimate = 2*math.atan2(quaternion[3], quaternion[0]) + difference = abs((estimate-expected_yaw+math.pi/4) % (math.pi/2)-math.pi/4) + self.assertLess(difference, math.radians(3)) + self.assertEqual(result['frame_index'], 4) + self.assertAlmostEqual(result['age_seconds'], .01) + self.assertGreater(result['object_position_uncertainty_m'], 0) + self.assertGreater(result['object_orientation_uncertainty_rad'], 0) + self.assertFalse(result['depth_accuracy_verified']) + self.assertIn('not calibrated', result['uncertainty_meaning']) + for name in tracker.SIGNALS: + sample = result['signals'][name] + self.assertEqual(sample['captured_at_seconds'], .2) + self.assertEqual(sample['available_at_seconds'], .205) + self.assertEqual(sample['status'], 'available') + self.assertIsInstance(result['signals']['object_position_uncertainty_m']['value'], list) + json.dumps(result, allow_nan=False) + + def assert_abstains(self, result, reason=None): + self.assertNotEqual(result['status'], 'available', result) + if reason is not None: + self.assertEqual(result['reason'], reason, result) + for name in tracker.SIGNALS: + self.assertIsNone(result[name]) + self.assertEqual(result['signals'][name], dict(status='missing', value=None, + captured_at_seconds=None, available_at_seconds=None, age_seconds=None)) + json.dumps(result, allow_nan=False) + + def test_absence_and_full_occlusion_do_not_forward_fill(self): + rgb, depth, calibration, *_ = fixture() + self.assertEqual(run(rgb, depth, calibration)['status'], 'available') + rgb[:] = 90 + result = run(rgb, depth, calibration) + self.assert_abstains(result, 'object_absent_or_too_small') + self.assertEqual(result['frame_index'], 4) + self.assertEqual(result['captured_at_seconds'], .2) + + def test_shading_preserves_colour_ratios_but_grey_or_near_black_abstains(self): + rgb, depth, calibration, centre, _, mask = fixture() + for colour in ([41, 16, 9], [50, 20, 11], [31, 12, 7]): + with self.subTest(colour=colour): + rgb[mask] = colour + result = run(rgb, depth, calibration) + self.assertEqual(result['status'], 'available', result) + np.testing.assert_allclose(result['object_position_estimate_m'], centre, atol=.0015) + for colour in ([41, 41, 41], [50, 45, 40], [10, 4, 2]): + rgb[mask] = colour + self.assert_abstains(run(rgb, depth, calibration), 'object_absent_or_too_small') + + def test_half_occlusion_abstains_instead_of_inventing_centre(self): + rgb, depth, calibration, _, _, mask = fixture(yaw=0) + columns = np.where(mask)[1]; split = int(np.median(columns)) + rgb[:, :split] = 90 + depth[:, :split] = .40 + self.assert_abstains(run(rgb, depth, calibration)) + + def test_two_same_colour_objects_are_ambiguous(self): + rgb, depth, calibration, *_ = fixture(centre=(-.02, -.04, .12)) + rgb2, depth2, _, _, _, mask2 = fixture(centre=(.04, .02, .12)) + rgb[mask2] = rgb2[mask2]; depth[mask2] = depth2[mask2] + self.assert_abstains(run(rgb, depth, calibration), 'ambiguous_colour_components') + + def test_cropped_and_wrong_sized_colour_regions_abstain(self): + rgb, depth, calibration, *_ = fixture() + rgb[:] = 90; rgb[0:30, 20:50] = [191, 77, 41]; depth[0:30, 20:50] = .46 + self.assert_abstains(run(rgb, depth, calibration), 'object_cropped_at_image_boundary') + rgb[:] = 90; rgb[40:110, 30:100] = [191, 77, 41]; depth[40:110, 30:100] = .46 + self.assert_abstains(run(rgb, depth, calibration), 'incomplete_or_wrong_size_top_surface') + + def test_invalid_far_plane_and_non_metric_depth_never_estimates(self): + rgb, depth, calibration, _, _, mask = fixture() + for value, reason in ((float('nan'), 'nonfinite_or_nonpositive_depth'), + (float('inf'), 'nonfinite_or_nonpositive_depth'), + (0., 'nonfinite_or_nonpositive_depth'), + (-1., 'nonfinite_or_nonpositive_depth'), + (100., 'object_outside_usable_depth'), + (.001, 'object_outside_usable_depth')): + with self.subTest(value=value): + bad = depth.copy(); bad[mask] = value + self.assert_abstains(run(rgb, bad, calibration), reason) + + def test_types_dimensions_alignment_and_calibration_are_fail_closed(self): + rgb, depth, calibration, *_ = fixture() + for bad_rgb, bad_depth in ((rgb.astype(float), depth), (rgb[..., 0], depth), + (rgb[..., :2], depth), (rgb, depth.astype(np.uint16)), + (rgb, depth[:-1]), (rgb.tolist(), depth)): + with self.subTest(rgb_type=type(bad_rgb), depth_shape=getattr(bad_depth, 'shape', None)): + self.assert_abstains(run(bad_rgb, bad_depth, calibration)) + for mutate in (lambda c: c.update(depth_units='mm'), + lambda c: c.update(base_frame='world'), + lambda c: c['intrinsics'].update(fx=0), + lambda c: c['intrinsics'].update(fy=float('nan')), + lambda c: c['intrinsics'].update(width=True), + lambda c: c['intrinsics'].update(width=161), + lambda c: c['base_from_camera'].__setitem__(0, 1), + lambda c: c['base_from_camera'].__setitem__(15, 0), + lambda c: c.update(valid_until_seconds=.205), + lambda c: c.update(valid_from_seconds=.201)): + bad = deepcopy(calibration); mutate(bad) + self.assert_abstains(run(rgb, depth, bad)) + + def test_time_order_boundary_source_id_and_json_safe_rejections(self): + rgb, depth, calibration, *_ = fixture() + def timed(**changes): + values = dict(frame_index=4, captured_at_seconds=.2, + available_at_seconds=.205, decision_at_seconds=.21) + values.update(changes) + return tracker.estimate_pose(rgb, depth, calibration, **values) + self.assertEqual(timed(decision_at_seconds=.3)['status'], 'available') + self.assertEqual(timed(decision_at_seconds=.3+.5e-12)['status'], 'available') + self.assert_abstains(timed(decision_at_seconds=.3+2e-12), 'frame_older_than_100ms') + self.assert_abstains(timed(decision_at_seconds=.300001), 'frame_older_than_100ms') + for changes in ({'captured_at_seconds':.22}, {'available_at_seconds':.19}, + {'available_at_seconds':.211}, {'captured_at_seconds':float('nan')}, + {'captured_at_seconds':True}, {'captured_at_seconds':10**1000}, {'captured_at_seconds':-.1}, + {'frame_index':True}, {'frame_index':-1}, {'frame_index':4.0}): + self.assert_abstains(timed(**changes)) + + def test_tilted_top_plane_refuses_upright_assumption(self): + rgb, depth, calibration, centre, _, mask = fixture(yaw=0) + # Intersect the tilted plane z = top + tan(20deg)*(base_x-centre_x) + # with the declared top-down perspective rays; the colour footprint + # stays intentionally generous, so plane orientation must reject it. + rows, _ = np.indices(depth.shape) + dx_direction = (rows+.5-60)/400 + slope = math.tan(math.radians(20)) + depth[mask] = ((.6-(centre[2]+.02)-slope*(.01-centre[0])) / + (1+slope*dx_direction))[mask] + self.assert_abstains(run(rgb, depth, calibration), 'outside_upright_profile') + + def test_no_native_truth_inputs_imports_or_stateful_estimate_history(self): + parameters = inspect.signature(tracker.estimate_pose).parameters + self.assertEqual(set(parameters), {'rgb', 'depth', 'calibration', 'frame_index', + 'captured_at_seconds', 'available_at_seconds', 'decision_at_seconds', 'config'}) + source = inspect.getsource(tracker) + self.assertNotIn('import mujoco', source) + self.assertNotIn('piper_task', source) + self.assertNotIn('native_info', source) + rgb, depth, calibration, *_ = fixture() + with self.assertRaises(TypeError): + run(rgb, depth, calibration, native_object_position=[0., 0., 0.]) + before_rgb, before_depth, before_calibration = rgb.copy(), depth.copy(), deepcopy(calibration) + first = run(rgb, depth, calibration); second = run(rgb, depth, calibration) + first['diagnostics'].pop('processing_seconds'); second['diagnostics'].pop('processing_seconds') + self.assertEqual(first, second) + np.testing.assert_array_equal(rgb, before_rgb); np.testing.assert_array_equal(depth, before_depth) + self.assertEqual(calibration, before_calibration) + + def test_configuration_is_frozen_and_bounded(self): + for change in ({'side_m':.08}, {'maximum_age_seconds':1.}, {'minimum_top_pixels':True}, + {'maximum_tilt_rad':float('nan')}, {'minimum_visibility':1.1}): + with self.assertRaises(ValueError): + replace(tracker.TrackerConfig(), **change) + + def test_large_colour_field_is_bounded_before_component_walk(self): + rgb, depth, calibration, *_ = fixture(width=320, height=240) + rgb[:] = [191, 77, 41] + self.assert_abstains(run(rgb, depth, calibration), 'colour_support_exceeds_bounded_profile') + + +if __name__ == '__main__': + unittest.main() diff --git a/tests/test_rgbd_independent_audit.py b/tests/test_rgbd_independent_audit.py new file mode 100644 index 0000000..3dc082a --- /dev/null +++ b/tests/test_rgbd_independent_audit.py @@ -0,0 +1,182 @@ +"""Saved-data audit tests with a fabricated 320-row study, no native imports.""" +from contextlib import redirect_stdout +from hashlib import sha256 +import importlib.util +import io +import json +from pathlib import Path +from tempfile import TemporaryDirectory +import unittest + +import numpy as np + + +MODULE = Path(__file__).resolve().parents[1]/'benchmarks/piper-rgbd-tracker-v01/independent_audit.py' +spec = importlib.util.spec_from_file_location('rgbd_independent_audit', MODULE) +audit_module = importlib.util.module_from_spec(spec); spec.loader.exec_module(audit_module) + + +def write(path, document): + path.parent.mkdir(parents=True, exist_ok=True) + path.write_bytes(audit_module.canonical(document)+b'\n') + + +def identity(path): + body = path.read_bytes(); return dict(bytes=len(body), sha256=sha256(body).hexdigest()) + + +def fixture(root): + """Entirely authored data double; it does not measure a physical estimator.""" + root.mkdir(); folder = root/'confirmation'; folder.mkdir(); source = root/'source'; source.mkdir() + (source/'tracker.py').write_text('# frozen audit test double\n') + sources = {'tracker.py': identity(source/'tracker.py')['sha256']}; runtime = {'test_double': True} + bindings = {'160': {'test_double': True}, '320': {'test_double': True}} + groups = [dict(id=f'pose{i:02}', split='development' if i < 8 else 'confirmation', + position_m=[0.,0.,.1], yaw_rad=0.) for i in range(24)] + variants = list(audit_module.VARIANTS); negatives = variants[2:] + limits = dict(clear_coverage_min=.9, clear_position_p95_m_max=.01, + clear_yaw_p95_rad_max=.17453292519943295, negative_false_available_max=0) + protocol = dict(groups=groups, variants=variants, resolutions=[[160,120],[320,240]], + expected_available=['clear'], challenge_positive=['partial_occlusion'], expected_abstain=negatives, + provisional_gate=limits) + write(root/'protocol.json', protocol); protocol_hash = identity(root/'protocol.json')['sha256'] + dev = {} + for name in ('manifest.json','predictions.json','report.json'): + write(root/'development'/name, {'test_double': True}); dev[name] = identity(root/'development'/name) + freeze = dict(protocol_sha256=protocol_hash, development_evidence=dev, sources=sources, + runtime=runtime, renderer_bindings=bindings) + write(root/'freeze.json',freeze) + schedule=[]; items=[]; predictions=[]; rows=[] + for width,height in ((160,120),(320,240)): + calibration = dict(base_frame='robot_base',base_from_camera=[1.,0.,0.,0., 0.,-1.,0.,0., 0.,0.,-1.,1., 0.,0.,0.,1.], + intrinsics=dict(width=width,height=height,fx=100.,fy=100.,cx=width/2,cy=height/2)) + for group in groups[8:]: + for variant in variants: + entry=dict(id=f'{group["id"]}-{width}-{variant}', group_id=group['id'],variant=variant, + width=width,height=height,frame_index=len(schedule)); schedule.append(entry) + path=folder/entry['id'];path.mkdir() + timing=dict(frame_index=entry['frame_index'],captured_at_seconds=.2, + available_at_seconds=.201 if variant=='future' else .2, + decision_at_seconds=.301 if variant=='stale' else .2) + write(path/'input.json',dict(calibration=calibration,timing=timing)) + write(path/'truth.json',dict(position_m=[0.,0.,.1],quaternion_wxyz=[1.,0.,0.,0.],yaw_rad=0., + tilt_rad=.3 if variant in ('tilt15','tilt30') else 0.,present=variant!='absent')) + (path/'rgb.npy').write_bytes(b'authored RGB hash fixture; not consumed by auditor') + (path/'rgb.png').write_bytes(b'authored PNG hash fixture') + if variant in ('clear','occluded'): + np.save(path/'depth.npy',np.full((height,width),.838 if variant=='occluded' else .88,dtype=np.float32),allow_pickle=False) + else: np.save(path/'depth.npy',np.ones((1,1),dtype=np.float32),allow_pickle=False) + write(path/'started.json',dict(**entry,status='started')) + item=dict(**entry,render_status='rendered',error=None,files={p.name:identity(p) for p in path.iterdir()}, + render_wall_seconds=.01,renderer_warnings=[]) + write(path/'receipt.json',item);items.append(dict(**item,receipt_sha256=identity(path/'receipt.json')['sha256'])) + is_available=variant=='clear';age=None if variant=='future' else timing['decision_at_seconds']-.2 + values=([.001,0.,.1],[1.,0.,0.,0.],.002,.01) if is_available else (None,)*4 + result=dict(status='available' if is_available else 'missing',reason='authored_test_double', + **timing,age_seconds=age,base_frame='robot_base',depth_accuracy_verified=False,signals={}) + for name,value in zip(audit_module.SIGNALS,values): + result[name]=value + result['signals'][name]=(dict(status='available',value=value if isinstance(value,list) else [value], + captured_at_seconds=.2,available_at_seconds=.2,age_seconds=0.) if is_available else + dict(status='missing',value=None,captured_at_seconds=None,available_at_seconds=None,age_seconds=None)) + pred=dict(id=entry['id'],estimate_status='complete',result=result,error=None,wall_seconds=.001,cpu_seconds=.0008, + pipeline_wall_seconds=.002,inputs={name:item['files'][name] for name in audit_module.INPUTS}) + estimate_path=folder/'estimates'/(entry['id']+'.json');write(estimate_path,pred) + predictions.append(dict(**pred,receipt_sha256=identity(estimate_path)['sha256'])) + rows.append(dict(**items[-1],estimate_status='complete',available=is_available, + position_error_m=.001 if is_available else None,yaw_error_rad=0. if is_available else None, + position_heuristic_scale_m=.002 if is_available else None,orientation_heuristic_scale_rad=.01 if is_available else None, + position_error_within_heuristic_scale=True if is_available else None, + yaw_error_within_heuristic_scale=True if is_available else None,wall_seconds=.001,cpu_seconds=.0008)) + write(folder/'scheduled.json',dict(records=schedule,protocol_sha256=protocol_hash,split='confirmation')) + manifest=dict(records=items,source_hashes=sources,source_stable=True,runtime=runtime,renderer_bindings=bindings, + protocol_sha256=protocol_hash,scheduled_sha256=identity(folder/'scheduled.json')['sha256'],split='confirmation',session_errors=[]) + write(folder/'manifest.json',manifest) + write(folder/'predictions.json',dict(predictions=predictions,source_hashes=sources,runtime=runtime, + manifest_sha256=identity(folder/'manifest.json')['sha256'],protocol_sha256=protocol_hash,split='confirmation', + freeze_sha256=identity(root/'freeze.json')['sha256'])) + summaries=[];gates=[] + for width in (160,320): + for variant in variants: + available=variant=='clear' + summaries.append(dict(width=width,variant=variant,scheduled=16,available=16 if available else 0,pipeline_failures=0, + position_evaluated=16 if available else 0,yaw_evaluated=16 if available else 0, + position_p50_m=.001 if available else None,position_p95_m=.001 if available else None, + yaw_p95_rad=0. if available else None,wall_p50_s=.001,wall_p95_s=.001,cpu_p95_s=.0008, + heuristic_comparisons=16 if available else 0,yaw_heuristic_comparisons=16 if available else 0, + position_error_within_heuristic_scale=16 if available else 0,yaw_error_within_heuristic_scale=16 if available else 0)) + checks={name:True for name in ('all_scheduled_fixtures_complete','clear_pose_metrics_complete','clear_coverage', + 'clear_position','clear_yaw','no_negative_false_available')} + gates.append(dict(width=width,passed=True,checks=checks,clear_scheduled=16,clear_coverage=1.,negative_scheduled=128,negative_false_available=0)) + write(folder/'report.json',dict(rows=rows,summary=summaries,gates=gates,scheduled=320, + predictions_sha256=identity(folder/'predictions.json')['sha256'],manifest_sha256=identity(folder/'manifest.json')['sha256'], + physical_robot_ready=False,depth_accuracy_verified=False)) + return source + + +class IndependentAuditTests(unittest.TestCase): + def setUp(self): + self.temp=TemporaryDirectory();self.addCleanup(self.temp.cleanup) + self.root=Path(self.temp.name)/'study';self.source=fixture(self.root) + + def audit(self): return audit_module.audit_study(self.root,self.source) + + def test_valid_full_denominator_depth_and_metrics(self): + result=self.audit() + self.assertTrue(result['passed'],result['mismatches']) + self.assertEqual(result['processed_confirmation_ids'],320) + self.assertEqual(len(result['occluder_depth_checks']),32) + self.assertTrue(all(row['passed'] for row in result['occluder_depth_checks'])) + self.assertFalse(result['simulator_executed']);self.assertFalse(result['estimator_executed']) + self.assertTrue(result['source_bytes_verified']) + + def test_missing_artifact_fails_closed(self): + (self.root/'confirmation/pose08-160-clear/rgb.npy').unlink() + result=self.audit();self.assertFalse(result['passed']) + self.assertTrue(any('Missing/nonregular' in x for x in result['mismatches'])) + + def test_changed_raw_or_frozen_source_bytes_are_detected(self): + (self.source/'tracker.py').write_text('# altered\n') + result=self.audit();self.assertFalse(result['passed']) + self.assertIn('source.tracker.py',result['mismatches']) + + def test_wrong_summary_metric_fails_even_with_untouched_raw_data(self): + path=self.root/'confirmation/report.json';data=json.loads(path.read_text()) + data['summary'][0]['position_p95_m']=0.;write(path,data) + result=self.audit();self.assertFalse(result['passed']) + self.assertTrue(any('position_p95_m' in x for x in result['mismatches'])) + + def test_missing_prediction_denominator_is_refused(self): + path=self.root/'confirmation/predictions.json';data=json.loads(path.read_text()) + data['predictions'].pop();write(path,data) + result=self.audit();self.assertFalse(result['passed']) + self.assertTrue(any('missing/duplicate/unknown' in x for x in result['mismatches'])) + + def test_depth_card_check_uses_saved_depth_not_presence_label(self): + path=self.root/'confirmation/pose08-160-occluded/depth.npy' + np.save(path,np.full((120,160),.88,dtype=np.float32),allow_pickle=False) + # Call the independent geometric check directly: a forged receipt does + # not make a card that is RGB-only appear in the saved depth image. + audit=audit_module.Audit(self.root) + row=audit_module._occluder_check(audit,self.root/'confirmation','pose08',160) + self.assertFalse(row['passed']);self.assertEqual(len(audit.errors),1) + + def test_nested_signal_inconsistency_is_caught(self): + path=self.root/'confirmation/estimates/pose08-160-clear.json' + pred=json.loads(path.read_text());result=pred['result'];result['signals'][audit_module.SIGNALS[0]]['value'][0]=1. + audit=audit_module.Audit(self.root) + timing={k:result[k] for k in ('frame_index','captured_at_seconds','available_at_seconds','decision_at_seconds')} + audit_module._signal_contract(audit,result,timing,'test') + self.assertTrue(any('signals.object_position' in x for x in audit.errors)) + + def test_cli_writes_fresh_result_hash_and_refuses_overwrite(self): + path=Path(self.temp.name)/'audit.json' + with redirect_stdout(io.StringIO()) as output: + code=audit_module.main(['--study',str(self.root),'--source-root',str(self.source),'--output',str(path)]) + self.assertEqual(code,0);response=json.loads(output.getvalue()) + self.assertEqual(response['audit_sha256'],identity(path)['sha256']) + with self.assertRaises(FileExistsError): + audit_module.main(['--study',str(self.root),'--output',str(path)]) + + +if __name__=='__main__':unittest.main() diff --git a/tests/test_rgbd_package.py b/tests/test_rgbd_package.py new file mode 100644 index 0000000..db27fec --- /dev/null +++ b/tests/test_rgbd_package.py @@ -0,0 +1,140 @@ +"""Miniature packaging contracts; never run or repack the native study.""" +import importlib.util +import json +from pathlib import Path +from tempfile import TemporaryDirectory +import unittest +from unittest.mock import patch +import zipfile + +MODULE_PATH=Path(__file__).resolve().parents[1]/'benchmarks/piper-rgbd-tracker-v01/package_evidence.py' +spec=importlib.util.spec_from_file_location('rgbd_package_test_module',MODULE_PATH) +pkg=importlib.util.module_from_spec(spec);spec.loader.exec_module(pkg) + + +class PackageTests(unittest.TestCase): + def setUp(self): + self.temp=TemporaryDirectory();self.addCleanup(self.temp.cleanup) + self.root=Path(self.temp.name).resolve();self.study=self.root/'study';self.study.mkdir() + self.counts=patch.object(pkg,'COUNTS',{'development':1,'confirmation':1});self.counts.start();self.addCleanup(self.counts.stop) + self.write(self.study/'protocol.json',{'test':'miniature immutable protocol'}) + sources={} + for name in ('piper_rgbd_tracker.py','piper_rgbd_benchmark.py'): + source=self.study/'frozen-source'/name;self.put(source,b'# frozen miniature source\n') + sources[name]=pkg.identity(source.read_bytes())['sha256'] + self.write(self.study/'freeze.json',{'sources':sources,'protocol_sha256':pkg.identity((self.study/'protocol.json').read_bytes())['sha256']}) + self.write(self.study/'development-lineage.json',{'source_attempt':'test','identical_files':[]}) + self.put(self.study/'index.html',b'

Miniature evidence fixture

') + for split,pose in (('development','pose00'),('confirmation','pose08')): + folder=self.study/split;ident=pose+'-160-clear';files={} + for name in pkg.FIXTURE_FILES: + path=folder/ident/name;self.put(path,b'{}\n' if name.endswith('.json') else b'fixture bytes\n') + files[name]=pkg.identity(path.read_bytes()) + receipt=folder/ident/'receipt.json';self.write(receipt,{'fixture':ident}) + row={'id':ident,'files':files,'receipt_sha256':pkg.identity(receipt.read_bytes())['sha256']} + self.write(folder/'manifest.json',{'records':[row]}) + estimate=folder/'estimates'/(ident+'.json');self.write(estimate,{'estimate':ident}) + self.write(folder/'estimates'/(ident+'-started.json'),{'id':ident}) + self.write(folder/'estimates/started.json',{'started':True}) + self.write(folder/'predictions.json',{'predictions':[{'id':ident,'receipt_sha256':pkg.identity(estimate.read_bytes())['sha256']}]}) + self.write(folder/'report.json',{'scheduled':1,'rows':[{'id':ident}], + 'manifest_sha256':pkg.identity((folder/'manifest.json').read_bytes())['sha256'], + 'predictions_sha256':pkg.identity((folder/'predictions.json').read_bytes())['sha256']}) + for name in ('scheduled.json','renderer-160.json','renderer-320.json'):self.write(folder/name,{'fixture':True}) + inventory={p.relative_to(self.study).as_posix():pkg.identity(p.read_bytes()) for p in self.study.rglob('*') if p.is_file() + and not (p.relative_to(self.study).parts[0]=='development' and len(p.relative_to(self.study).parts)>2)} + self.audit=self.root/'audit.json';self.proof=dict(schema='dvidia.rgbd-independent-audit/v1',passed=True,mismatch_count=0, + mismatches=[],source_bytes_verified=True,expected_confirmation_ids=1,processed_confirmation_ids=1,evidence_inventory=inventory, + evidence_inventory_sha256=pkg.sha256(pkg.canonical(inventory)).hexdigest()) + self.write(self.audit,self.proof) + self.history=[] + for label in ('baseline','earlier'): + root=self.root/label;self.history.append(root) + for name in ('predictions.json','report.json'):self.write(root/'development'/name,{'historical':True}) + self.put(root/'development-source-v1/piper_rgbd_tracker.py',b'# historical source\n') + self.failed=self.root/'failed';self.write(self.failed/'development/manifest.json',{'graphics_failed':True}) + self.write(self.failed/'protocol.json',{'historical':True}) + self.card=self.root/'README.md';self.put(self.card,b'# Miniature fixture\n') + self.output=self.root/'release' + + @staticmethod + def put(path,raw):path.parent.mkdir(parents=True,exist_ok=True);path.write_bytes(raw) + + def write(self,path,value):self.put(path,pkg.canonical(value)+b'\n') + + def run_package(self): + return pkg.package(self.study,self.audit,self.output,baseline=self.history[0],earlier=self.history[1],failed=self.failed,card=self.card) + + def test_package_contains_full_frames_sources_history_and_verified_payload(self): + result=self.run_package();self.assertTrue(result['zip_readback_verified']) + archive=self.output/result['archive'];self.assertEqual(pkg.archive_identity(archive)['sha256'],result['sha256']) + with zipfile.ZipFile(archive) as zipped: + self.assertIn('development/pose00-160-clear/depth.npy',zipped.namelist()) + self.assertIn('confirmation/pose08-160-clear/rgb.png',zipped.namelist()) + self.assertIn('frozen-source/piper_rgbd_tracker.py',zipped.namelist()) + self.assertIn('history/initial-development/source/piper_rgbd_tracker.py',zipped.namelist()) + self.assertIn('history/sandbox-graphics-failure/manifest.json',zipped.namelist()) + self.assertEqual(zipped.read('confirmation/report.json'),(self.output/'huggingface/report.json').read_bytes()) + self.assertEqual({p.name for p in (self.output/'huggingface').iterdir()}, + {result['archive'],'README.md','LICENSE','report.json','protocol.json','inspection.json'}) + + def test_missing_or_failed_audit_refused_before_output(self): + for proof in (dict(self.proof,passed=False),{k:v for k,v in self.proof.items() if k!='evidence_inventory'}): + self.write(self.audit,proof) + with self.assertRaises(ValueError):self.run_package() + self.assertFalse(self.output.exists()) + + def test_wrong_audit_inventory_digest_refused(self): + self.write(self.audit,dict(self.proof,evidence_inventory_sha256='0'*64)) + with self.assertRaisesRegex(ValueError,'inventory digest'):self.run_package() + + def test_post_audit_frozen_source_tamper_refused(self): + (self.study/'frozen-source/piper_rgbd_tracker.py').write_text('# changed\n') + with self.assertRaisesRegex(ValueError,'differs from independent audit'):self.run_package() + self.assertFalse(self.output.exists()) + + def test_post_audit_report_tamper_refused(self): + (self.study/'confirmation/report.json').write_text('{}\n') + with self.assertRaisesRegex(ValueError,'sealed identity'):self.run_package() + + def test_development_frame_tamper_refused_by_sealed_manifest(self): + (self.study/'development/pose00-160-clear/depth.npy').write_bytes(b'changed depth') + with self.assertRaisesRegex(ValueError,'differs from independent audit or sealed manifest'):self.run_package() + + def test_symlink_file_and_directory_refused(self): + path=self.study/'frozen-source/piper_rgbd_tracker.py';raw=path.read_bytes();path.unlink() + elsewhere=self.root/'elsewhere.py';elsewhere.write_bytes(raw);path.symlink_to(elsewhere) + with self.assertRaisesRegex(ValueError,'Symlinks'):self.run_package() + path.unlink();path.write_bytes(raw) + (self.study/'extra').symlink_to(self.history[0],target_is_directory=True) + with self.assertRaisesRegex(ValueError,'Symlinks'):self.run_package() + + def test_unlisted_secret_or_mesh_file_refused(self): + path=self.study/'credentials.json';self.write(path,{'not_a_real_credential':'test'}) + with self.assertRaisesRegex(ValueError,'Unrecognized study file'):self.run_package() + path.unlink();(self.study/'arm.stl').write_bytes(b'not a mesh') + with self.assertRaisesRegex(ValueError,'Unrecognized study file'):self.run_package() + + def test_change_between_preflight_and_actual_copy_refused(self): + original=pkg.bound_read;calls=0 + def mutate_on_copy(path,expected): + nonlocal calls + if Path(path)==self.study/'protocol.json': + calls+=1 + if calls==2:Path(path).write_text('{"changed":true}\n') + return original(path,expected) + with patch.object(pkg,'bound_read',mutate_on_copy): + with self.assertRaisesRegex(ValueError,'sealed identity'):self.run_package() + self.assertFalse((self.output/'huggingface').exists()) + + def test_zip_readback_detects_changed_compressed_member(self): + original=zipfile.ZipFile.writestr + def corrupt(zipped,info,raw,*args,**kwargs): + if info.filename=='protocol.json':raw=b'changed archive member' + return original(zipped,info,raw,*args,**kwargs) + with patch.object(zipfile.ZipFile,'writestr',corrupt): + with self.assertRaisesRegex(ValueError,'Archived bytes differ'):self.run_package() + self.assertFalse((self.output/'huggingface').exists()) + + +if __name__=='__main__':unittest.main()