Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -83,7 +83,7 @@ jobs:
- name: Verify pinned learning caches
run: python -m unittest tests.test_camera_action_cache tests.test_camera_cache_cli tests.test_camera_action_export tests.test_camera_action_model -v
- name: Verify operator capture, recovery and reviewed exports without physics
run: python -m unittest tests.test_operator_capture tests.test_operator_export tests.test_godot_demonstration_bridge -v
run: python -m unittest tests.test_operator_capture tests.test_operator_export tests.test_godot_demonstration_bridge tests.test_tracker_ray_cache tests.test_piper_rgbd_tracker -v
- name: Verify exact-reference and initial-head contracts without physics
run: |
python -B -m unittest tests.test_camera_action_reference_export -v
Expand Down
3 changes: 3 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,9 @@ check reproduced all **10,200 native scalar values exactly** in independent repl
This is a capture/export prototype; human demonstrations, new policy training and
real-time teleoperation qualification remain ahead.
[Evidence and timing limits](benchmarks/godot-demonstration-v02/README.md).
[Capture performance study](benchmarks/godot-capture-performance-v01/README.md):
bounded camera-geometry reuse preserves exact observations/actions; its measured
speed benefit is preliminary, with mixed paired results.

**[Verified local learning cache](docs/camera-action-export-cache-v01.md).**
The **0.1.3.dev5 preview** adds `dvidia-train cache` to reuse verified camera/action
Expand Down
126 changes: 126 additions & 0 deletions benchmarks/godot-capture-performance-v01/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,126 @@
# Capture performance: fixed camera geometry

We added a small, bounded cache for the RGB-D tracker's fixed camera geometry.
Across three paired runs, average control-loop time fell **4.2%**, with one pair
slightly slower. This is a preliminary optimization result, not a reliable
speedup or real-time qualification. All compared observations, commands,
physics traces and learning arrays were preserved exactly.

The task is a small scripted movement of the simulated PiPER above a rigid
block. It does not demonstrate grasping, human teleoperation or a learned skill.
Every attempt is retained; all task scorers correctly report no placement.

## What changed

The tracker used to rebuild every pixel's optical ray and transform it into the
robot base frame on every image. Those directions depend only on image size,
camera intrinsics and camera rotation. We now reuse them with exact calibration
keys. The cache contains no images, depth, object position, task state or previous
estimates. Current observations still determine visibility, occlusion, missingness
and the estimated object pose.

Two immutable arrays are retained at most: **3.516 MiB** at 320×240 if both entries
are occupied, or **14.063 MiB** at the supported 640×480 maximum. Temporary
allocations and total process memory are separate and were not measured here.
No camera cadence, native tick, command rate limit, durable write or journal
commit barrier was removed.

## Measurements

Each confirmation run contains 100 control intervals, 51 RGB-D pairs and 2,000
native ticks: two simulated seconds at 50 Hz control, 25 Hz 320×240 cameras and
1 ms physics. The order was frozen as baseline, candidate, candidate, baseline,
baseline, candidate. Each run starts a fresh process and cache.

| Pair | Baseline control time | Cached control time | Time reduction |
|---|---:|---:|---:|
| 1 | 6.287 s | 5.920 s | 5.83% |
| 2 | 6.012 s | 6.093 s | −1.34% |
| 3 | 6.159 s | 5.668 s | 7.96% |
| Mean | 6.152 s | 5.894 s | 4.20% |

Control time includes context retrieval, encoder-only IK, native advancement,
camera/perception and durable recording. It excludes startup, finalization,
inspection, sockets, Godot and deliberate real-time pacing. Full per-run
percentiles, startup and finalization times are retained in the JSON sidecars.
`total_seconds` includes kinematics construction through capture finalization;
it excludes subsequent inspection and writing the profiling receipt.

A single actual windowed check per variant completed without client errors.
Two simulated seconds took **10.700 s baseline / 10.469 s cached** after handshake;
RPC p95 was **111.520 / 113.571 ms**. Tail latency did not improve in that check.
These runs demonstrate integration, not sustained throughput or a physical robot
reaction bound. The software does not yet meet a 20 ms wall-clock control deadline.

The paired and windowed runs use one worker process with
`OPENBLAS_NUM_THREADS=1` and `OMP_NUM_THREADS=1`. Earlier exploratory profiles
inherited the host thread settings, so their absolute timings are not directly
comparable. In that diagnostic profile, rendering/encoding consumed about
0.700 s and tracking 0.534 s of a 2.177 s profiled control loop; ray checks
accounted for 0.262 s within tracking. These cumulative scopes overlap.
The unprofiled diagnostic took 1.839 s; one comparison cannot isolate profiler
overhead from host variation. A separate-process warmup warms host/driver caches,
not the next process's Python or renderer instance.

Host: Apple M5, 10 logical processors, macOS 26.5.2 arm64, Python 3.12.9,
NumPy 2.5.3, MuJoCo 3.15.0 and Godot 4.7.2 compatibility OpenGL. This shared
host was not isolated from other work. Three pairs support a directional
observation, not statistical confidence or a hardware minimum recommendation.

## Correctness and installation

- **114 local test executions passed**, including randomized exact ray-depth
parity, cache eviction and immutability, calibration changes, missing/occluded
frames, causal export and interrupted-journal recovery.
- The three confirmation pairs matched **306,000 native scalar values**,
**306 RGB/depth files**, and **303 perception records** exactly, excluding only
explicitly identified source/run identities and wall-time diagnostics.
Calibration, timestamps, masks, issued/applied actions, lifecycle, source groups,
data types, 100×269 feature arrays and 100×7 target arrays also matched per pair.
- The windowed pair independently matched another **102,000 native values**,
102 frame files and 101 perception records, with the same full export checks.
- A separate native replay of one candidate recording matched all **102,000
values** and independently reconstructed the task score.
- A fresh base-package installation outside the checkout inspected the recording
and exported/reloaded 100 rows. This tests CLI/export installation; native
graphics used the existing simulation environment and remain tested only on Mac.

All eleven planned captures completed. The raw local evidence, including scene,
receipts and windowed screenshot, occupies about 201 MiB. No raw capture is bundled
in this small source publication. Integrity hashes bind the retained recordings.
There were no fits, human demonstrations, hardware commands or new skill claims.
The previous learned placement result remains 5/64 and unqualified.

## Reproduce

Start with the [local capture guide](../../docs/godot-demonstration-v02.md).
Use baseline commit `4ca9804c79e330221106217eac4faca582b4efa2` in a separate checkout.
The candidate source and script hashes are fixed in `candidate.json`; the
registered attempt budget and order are in `plan.json`.

From the candidate checkout, supply the baseline or candidate source directory:

```sh
PYTHONPATH=/absolute/baseline/src OPENBLAS_NUM_THREADS=1 OMP_NUM_THREADS=1 \
python scripts/profile-operator-capture.py --assets /absolute/robot-assets \
--output /absolute/fresh/baseline-run --duration 2

PYTHONPATH=/absolute/candidate/src OPENBLAS_NUM_THREADS=1 OMP_NUM_THREADS=1 \
python scripts/profile-operator-capture.py --assets /absolute/robot-assets \
--output /absolute/fresh/candidate-run --duration 2

python scripts/compare-operator-captures.py \
--baseline /absolute/fresh/baseline-run/capture \
--candidate /absolute/fresh/candidate-run/capture --output /absolute/fresh/parity.json
```

Use `--profile` only for diagnostic attribution; never combine profiled and
unprofiled samples into a speed claim. Keep all six confirmation attempts in
the declared order. The capture guard reserves 10 GiB plus each episode budget;
the original full-learning study's larger guard remains separate.

Next: isolate rendering, thread settings and file-finalization costs under a
longer, separately registered workload before claiming useful live teleoperation
latency. Human demonstration collection and held-out skill evaluation follow.

Added DVIDIA source is MIT; existing upstream robot notices remain unchanged.

Large diffs are not rendered by default.

Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{"camera_control":{"count":50,"max_ms":43.53804199490696,"p50_ms":30.42877052212134,"p95_ms":34.184999694116414,"total_seconds":1.5595909123076126},"camera_dt_seconds":0.04,"camera_pairs":51,"capture_bytes":19971489,"capture_integrity_sha256":"9eeb765704cb0665fee3cb469dd118e746d768e0a8355d7ac42412a38e6b08cb","case":{"goal_position_m":[0.35,0.08,0.08],"group_id":"operator-start-001","id":"operator-block-001","object_position_m":[0.3,-0.03,0.082],"seed":0},"complete":true,"control":{"count":100,"max_ms":43.53804199490696,"p50_ms":18.378999986452982,"p95_ms":33.44109557219781,"total_seconds":1.8394407449522987},"control_dt_seconds":0.02,"duration_s":2.0,"encoder_control":{"count":50,"max_ms":7.342415978200734,"p50_ms":5.4617709829472005,"p95_ms":6.623258112813345,"total_seconds":0.2798498326446861},"finish_seconds":0.33264308294747025,"ik":{"count":100,"max_ms":0.23612502263858914,"p50_ms":0.13704149750992656,"p95_ms":0.18392111815046516,"total_seconds":0.013867082772776484},"interval_seconds":[0.006021083972882479,0.04353804199490696,0.0056541249505244195,0.03531012503663078,0.005694540974218398,0.03177833301015198,0.0067646250245161355,0.032393583038356155,0.005654249980580062,0.030587624991312623,0.005554792005568743,0.0303683330421336,0.005400415975600481,0.031393583049066365,0.005733917001634836,0.0334840840077959,0.005887916020583361,0.03148225002223626,0.006534249987453222,0.03396454098401591,0.006382291961926967,0.03055520897032693,0.005305790982674807,0.029762874997686595,0.005266459018457681,0.03003495797747746,0.005211916984990239,0.029980125022120774,0.005443999951239675,0.030132291023619473,0.005422957998234779,0.029997041972819716,0.007342415978200734,0.033438833022955805,0.005256625008769333,0.03139275003923103,0.005215124983806163,0.030026708031073213,0.005289916996844113,0.03022508299909532,0.005468167015351355,0.029828791972249746,0.005895459034945816,0.03245987498667091,0.005328083003405482,0.030138041998725384,0.005228249996434897,0.029561957984697074,0.005348792008589953,0.02941558399470523,0.005403040966484696,0.030749833036679775,0.006171291985083371,0.030768041964620352,0.0055210410500876606,0.03436537500238046,0.005323499965015799,0.0324609589879401,0.005558834003750235,0.030730250000488013,0.005515499971807003,0.031263041018974036,0.005856207979377359,0.03168762504355982,0.005512707983143628,0.03141812502872199,0.005288499989546835,0.03027404099702835,0.005116415966767818,0.029874916013795882,0.005134709004778415,0.029905625036917627,0.005278750031720847,0.029934332997072488,0.005299625045154244,0.029545707977376878,0.005605582962743938,0.03048920800210908,0.006696082942653447,0.032743874995503575,0.005241583043243736,0.030028207984287292,0.005348750040866435,0.02992787503171712,0.005455374950543046,0.030114667024463415,0.0053635420044884086,0.030319540994241834,0.00557412498164922,0.031092084012925625,0.005568624997977167,0.030118249997030944,0.005263875005766749,0.02996841596905142,0.005214667005930096,0.029878125002142042,0.005653083964716643,0.030157333007082343,0.005578249983955175,0.030524833011440933],"numpy":"2.5.3","open_seconds":0.5098034159746021,"physical_robot_ready":false,"physics_dt_seconds":0.001,"profile_enabled":false,"profiles":{"control":[],"finish":[],"open":[]},"python":"3.12.9","recorded_intervals":100,"resolution":[320,240],"schema":"dvidia.operator-profile/v1","scope":"Single-process local scripted capture including encoder-only IK; excludes sockets, Godot, real-time pacing, model fitting and hardware. cProfile cumulative scopes overlap.","source_kind":"simulation_scripted_input","source_sha256":{"camera_action_capture.py":"399f512a08087c72bfa37e382157a2250f413e423974b77b1bd3c0a911ff0a1e","operator_capture.py":"f1b5b5c5abbdf090342bb18df76e95f1fd518bd05fc5705aedb956f0ce081715","piper_action_capture.py":"da037b2198cf7d2f3a9aa40fc19a4c0f5cc375a22c5dd75606e2c1856f6efc7f","piper_encoder_kinematics.py":"6606bb8aad04e5abb1e9b6d1a8ec628a4114acd167862167184fd4eef1d21c34","piper_rgbd_tracker.py":"43202cfcea404061a3a65124d6ad24aeeec5fae8ffaf863caa245f2f54c46c36","profile-operator-capture.py":"daf3aec5ad1ace998c0368df5658064b7300217657da81ff1fa4b318dab299f1"},"strict_success":false,"system":"macOS-26.5.2-arm64-arm-64bit","teacher_queries":0,"total_seconds":3.1507570419926196}
23 changes: 23 additions & 0 deletions benchmarks/godot-capture-performance-v01/candidate.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
{
"schema": "dvidia.operator-performance-candidate/v1",
"frozen_at": "2026-11-11T09:02:48.786301+00:00",
"optimization": "Cache immutable camera-to-base ray geometry with exact calibration keys, max2 entries; preserve current object/depth processing.",
"source_sha256": {
"src/dvidia_training/piper_rgbd_tracker.py": "e89cba0414832fcfc09127eb05a326557d3aa32afd458a9a8f59d6c4f850f2d3",
"tests/test_tracker_ray_cache.py": "def2baa4b73a93095f5005574aed3f7f969184e3ca17e4b1c320bdc3084ca15f",
"scripts/profile-operator-capture.py": "daf3aec5ad1ace998c0368df5658064b7300217657da81ff1fa4b318dab299f1",
"scripts/compare-operator-captures.py": "c6a45b36a84d66d71a830d9c620ef814bd060f34a0a261d54167de5c417c24c0"
},
"plan_sha256": "47e19a0d45a66bd7dcddbe2ab354776c676bae9708d3b6e2baaf7bb99d2a5fe4",
"retained_cache_max_bytes": 14745600,
"retained_cache_current_resolution_max_bytes": 3686400,
"transient_memory_measured": false,
"comparison_order": [
"baseline",
"candidate",
"candidate",
"baseline",
"baseline",
"candidate"
]
}
19 changes: 19 additions & 0 deletions benchmarks/godot-capture-performance-v01/fresh-install.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
{
"schema": "dvidia.operator-performance-fresh-install/v1",
"source_sha256": "e89cba0414832fcfc09127eb05a326557d3aa32afd458a9a8f59d6c4f850f2d3",
"installed_source_matches": true,
"cwd_outside_checkout": true,
"features_shape": [
100,
269
],
"targets_shape": [
100,
7
],
"source_kind": "simulation_scripted_input",
"base_cli_export_only": true,
"native_graphics_qualified": false,
"physical_robot_ready": false,
"training_started": false
}
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{"camera_control":{"count":50,"max_ms":202.28437497280538,"p50_ms":98.5261254827492,"p95_ms":124.12394999118985,"total_seconds":5.1247595408931375},"camera_dt_seconds":0.04,"camera_pairs":51,"capture_bytes":19971365,"capture_integrity_sha256":"8dc17e27637ca6eb0dea4810e341633380066d1c7e92bdd3baa140387dd6514f","case":{"goal_position_m":[0.35,0.08,0.08],"group_id":"operator-start-001","id":"operator-block-001","object_position_m":[0.3,-0.03,0.082],"seed":0},"complete":true,"control":{"count":100,"max_ms":202.28437497280538,"p50_ms":81.58202102640644,"p95_ms":111.04468926205298,"total_seconds":6.286720753996633},"control_dt_seconds":0.02,"duration_s":2.0,"encoder_control":{"count":50,"max_ms":78.26379203470424,"p50_ms":21.5492709830869,"p95_ms":29.8785145569127,"total_seconds":1.1619612131034955},"finish_seconds":1.184006416995544,"ik":{"count":100,"max_ms":1.74866704037413,"p50_ms":0.5229164962656796,"p95_ms":0.9545312525006009,"total_seconds":0.05820921575650573},"interval_seconds":[0.02358779194764793,0.16757237497949973,0.0252486250246875,0.10419220803305507,0.019289083022158593,0.095367832982447,0.018274333036970347,0.09857379196910188,0.0183087500045076,0.10568962496472523,0.019486667006276548,0.10061804199358448,0.019632457988336682,0.0893711670069024,0.025493375025689602,0.09613550000358373,0.023416875046677887,0.10071083303773776,0.023357084020972252,0.09524220897583291,0.020503124978858978,0.09577945899218321,0.021243499999400228,0.09879437502240762,0.029142041981685907,0.09132862504338846,0.021570916986092925,0.0944916249718517,0.02052266604732722,0.09895520802820101,0.030481083027552813,0.11820020800223574,0.027128459012601525,0.10954691597726196,0.022062291973270476,0.08752099995035678,0.02020299999276176,0.09391962498193607,0.019280874985270202,0.08664562494959682,0.01813979202415794,0.20228437497280538,0.07826379203470424,0.1106680830125697,0.02538241702131927,0.12767062499187887,0.0334250419982709,0.09847845899639651,0.024009000044316053,0.0857862500124611,0.025091584015171975,0.0927669579978101,0.01968974998453632,0.09608695801580325,0.020766332978382707,0.10098487499635667,0.01982699998188764,0.09576599998399615,0.01996233401587233,0.10765658301534131,0.0175874590058811,0.09694591700099409,0.02444329101126641,0.11978912499034777,0.024498209008015692,0.10868158302037045,0.02373262500623241,0.09831087494967505,0.024132042017299682,0.09944100002758205,0.01820541697088629,0.10604566702386364,0.022590375039726496,0.10654266702476889,0.018256540992297232,0.08886925003025681,0.021527624980080873,0.10009762499248609,0.02096049999818206,0.1016846249694936,0.018795792013406754,0.08490025001810864,0.017873667005915195,0.0970781670184806,0.019803207949735224,0.0958981669973582,0.021679249999579042,0.10187750001205131,0.02368304197443649,0.09816074999980628,0.02053104096557945,0.09019804198760539,0.02293487498536706,0.10006620798958465,0.018435459001921117,0.10077533300500363,0.02566704200580716,0.09232158301165327,0.02383370796451345,0.09026979096233845],"numpy":"2.5.3","open_seconds":1.4127261670073494,"physical_robot_ready":false,"physics_dt_seconds":0.001,"profile_enabled":false,"profiles":{"control":[],"finish":[],"open":[]},"python":"3.12.9","recorded_intervals":100,"resolution":[320,240],"schema":"dvidia.operator-profile/v1","scope":"Single-process local scripted capture including encoder-only IK; excludes sockets, Godot, real-time pacing, model fitting and hardware. cProfile cumulative scopes overlap.","source_kind":"simulation_scripted_input","source_sha256":{"camera_action_capture.py":"399f512a08087c72bfa37e382157a2250f413e423974b77b1bd3c0a911ff0a1e","operator_capture.py":"f1b5b5c5abbdf090342bb18df76e95f1fd518bd05fc5705aedb956f0ce081715","piper_action_capture.py":"da037b2198cf7d2f3a9aa40fc19a4c0f5cc375a22c5dd75606e2c1856f6efc7f","piper_encoder_kinematics.py":"6606bb8aad04e5abb1e9b6d1a8ec628a4114acd167862167184fd4eef1d21c34","piper_rgbd_tracker.py":"43202cfcea404061a3a65124d6ad24aeeec5fae8ffaf863caa245f2f54c46c36","profile-operator-capture.py":"daf3aec5ad1ace998c0368df5658064b7300217657da81ff1fa4b318dab299f1"},"strict_success":false,"system":"macOS-26.5.2-arm64-arm-64bit","teacher_queries":0,"total_seconds":10.213273708010092}
Loading
Loading