Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
44 changes: 43 additions & 1 deletion .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -68,6 +68,9 @@ jobs:
dvidia-motion --help
dvidia-transition --help
dvidia-embodiment --help
dvidia-reference-task --help
dvidia-robot-sim --help
dvidia-robot-sim list
dvidia-training-doctor --json
- name: Test footage, visual training and loopback studio
run: python -m unittest tests.test_footage_data tests.test_footage_train tests.test_footage_pipeline tests.test_footage_benchmark tests.test_training_studio tests.test_doctor tests.test_coverage tests.test_episode_tape tests.test_reaction tests.test_reaction_metrics tests.test_reaction_convergence tests.test_reaction_benchmark tests.test_motor_memory tests.test_motor_metrics tests.test_motor_benchmark tests.test_observations tests.test_observation_models tests.test_motion tests.test_motion_benchmark tests.test_transition_model tests.test_transition_pilot tests.test_transition_cli tests.test_embodiment_calibration tests.test_embodiment_cli tests.test_reference_sim -v
Expand All @@ -90,6 +93,7 @@ jobs:
run: |
npm ci --ignore-scripts --prefix viewer
npm run --prefix viewer build
node tests/browser/reference_replay_loader.cjs
git diff --exit-code -- src/dvidia_training/reference_replay.bundle.js src/dvidia_training/reference_replay.third-party.txt
- name: Provision pinned reference models
run: |
Expand All @@ -99,13 +103,18 @@ jobs:
- name: Test compiled mechanics, collisions and replay scoring
env:
DVIDIA_REFERENCE_ASSETS: runs/reference-assets-ci
run: python -m unittest tests.test_reference_sim tests.test_reference_replay tests.test_embodiment_cli tests.test_embodiment_calibration -v
run: python -m unittest tests.test_reference_sim tests.test_reference_replay tests.test_embodiment_cli tests.test_embodiment_calibration tests.test_reference_task tests.test_reference_task_benchmark tests.test_reference_task_replay -v
- name: Retain all numerical reference attempts
run: |
dvidia-embodiment benchmark runs/reference-assets-ci --hands --output runs/reference-bench-ci --repeats 1 --seconds 2
dvidia-embodiment inspect runs/reference-bench-ci
- name: Export native mesh and recorded 3D poses
run: dvidia-embodiment export-replay runs/reference-assets-ci runs/reference-bench-ci --output runs/reference-replay-ci
- name: Retain native task and matched control attempts
run: |
dvidia-reference-task benchmark runs/reference-assets-ci --output runs/reference-task-ci --nominal 1 --stress 0
dvidia-reference-task inspect runs/reference-task-ci
dvidia-reference-task export-replay runs/reference-assets-ci runs/reference-task-ci --output runs/reference-task-replay-ci
- name: Preserve reference evidence
if: always()
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
Expand All @@ -114,6 +123,39 @@ jobs:
path: |
runs/reference-bench-ci
runs/reference-replay-ci
runs/reference-task-ci
runs/reference-task-replay-ci
if-no-files-found: ignore
- name: Verify additional CPU arm profiles
run: |
dvidia-robot-sim fetch piper --output runs/robot-profile-assets-ci
dvidia-robot-sim fetch so101 --output runs/robot-profile-assets-ci
dvidia-robot-sim audit piper runs/robot-profile-assets-ci
dvidia-robot-sim audit so101 runs/robot-profile-assets-ci
dvidia-robot-sim benchmark piper runs/robot-profile-assets-ci --output runs/piper-smoke-ci
dvidia-robot-sim benchmark so101 runs/robot-profile-assets-ci --output runs/so101-smoke-ci
- name: Test profile contracts against pinned native assets
env:
DVIDIA_ROBOT_PROFILE_ASSETS: runs/robot-profile-assets-ci
run: python -m unittest tests.test_robot_profiles tests.test_piper_task tests.test_piper_task_benchmark -v
- name: Retain PiPER contact task and matched controls
run: |
dvidia-piper-task benchmark runs/robot-profile-assets-ci --output runs/piper-task-ci --nominal 1 --stress 0
dvidia-piper-task inspect runs/piper-task-ci
- name: Test bounded static simulation publication without credentials
run: |
python -m pip install -r publishing/huggingface/requirements.txt
python -m unittest discover -s publishing/huggingface -p 'test_publish_robot_sim.py' -v
python -m unittest discover -s publishing/huggingface/robot-simulation-lab -p 'test_*.py' -v
- name: Preserve additional profile evidence
if: always()
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
with:
name: additional-robot-profiles
path: |
runs/piper-task-ci
runs/piper-smoke-ci
runs/so101-smoke-ci
if-no-files-found: ignore

motion:
Expand Down
23 changes: 19 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,12 +2,27 @@

# DVIDIA Training

[Install locally](docs/installation.md) · [Hardware guide](docs/hardware.md) · [Execution map](docs/roadmap.md) · [Upcoming arms](docs/hardware-roadmap.md) · [Official site](https://dvidia.org/) · [Research](https://research.dvidia.org/papers/skillspace-footage-training/) · [Frozen benchmark evidence](benchmarks/sample-count-19047/README.md)
[Install locally](docs/installation.md) · [Hardware guide](docs/hardware.md) · [Execution map](docs/roadmap.md) · [Robot simulations](docs/robot-simulation-library.md) · [Upcoming arms](docs/hardware-roadmap.md) · [Official site](https://dvidia.org/) · [Research](https://research.dvidia.org/papers/skillspace-footage-training/) · [Frozen benchmark evidence](benchmarks/sample-count-19047/README.md)

[Recorded demo](https://huggingface.co/spaces/Dvidia/dvidia-training) · [Synthetic datasets](https://huggingface.co/datasets/Dvidia/dvidia-training-examples) · [Pilot models](https://huggingface.co/Dvidia/dvidia-training-pilot)

**Research alpha.** The recorded demo covers the original one-scene simulation pilot; the frozen benchmark is a separate experiment. Training runs locally. No human-video action bridge or hardware skill is established.

The new [native grasp task benchmark](docs/native-task-benchmark.md) connects the
pinned six-axis arm and coupled gripper to free-object grasp, carry and release
attempts. It retains matched open-jaw/hold controls, frozen protocols and local
3D replay. This candidate source uses an authored controller; it is a test
environment for the next learned skill, separate from the older hosted pilot.

The [robot simulation library](docs/robot-simulation-library.md) adds pinned
PiPER (six arm axes) and SO-101 (five arm axes) profiles with separate grippers,
explicit CPU asset provisioning, native audits and unloaded motion benchmarks.
PiPER also has a [native contact pilot](docs/piper-contact-pilot.md) with an
authored pickup-and-place controller and matched controls. SO-101 remains at
unloaded mechanics; learned policies and physical qualification remain open. The
[Robot Simulation Lab](https://huggingface.co/spaces/Dvidia/robot-simulation-lab)
publishes recorded evidence and local run instructions.

Train a small visual model from Skillspace footage on your own computer or server. The pipeline checks actual videos and source groups, fixes training/development/test splits, fits weights on a CPU, and exports a result with held-out measurements and integrity hashes.

The base install needs **Python 3.12+, NumPy and FFmpeg**. It needs no GPU, CUDA, account or API key. Training is offline by default; downloading software or public footage is a separate, explicit step.
Expand Down Expand Up @@ -126,9 +141,9 @@ five-finger references have separate contracts. A paired replay scorer measures
joint, tool and jaw errors once aligned physical recordings are available.
Export actual mesh geometry and recorded native poses to an offline interactive
3D replay with orbit controls, six-joint readouts and collision-proxy inspection.
The [upcoming hardware list](docs/hardware-roadmap.md) tracks YAM, OpenArm, PiPER,
ARX X5 and PAROL6 as planned additions requiring separate profiles and benchmarks.
These tools use the `feat/reference-3d-replay` candidate branch; the new
The [hardware list](docs/hardware-roadmap.md) tracks implemented PiPER/SO-101
profiles and upcoming YAM, OpenArm, ARX X5 and PAROL6 integrations.
These tools use the `feat/reference-task-benchmark` candidate branch; the new
reference has no learned task policy or measured physical fidelity.

Generate an explicitly synthetic example, then train and verify its output:
Expand Down
56 changes: 56 additions & 0 deletions benchmarks/native-grasp-20261011/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,56 @@
# Native grasp task: frozen confirmation

11 October 2026 · source `ac920cbf70147a7edeb0c706a9b71047f318e12f`.

The authored native UR5e/Robotiq controller completes **4/4 nominal and 2/2
low-friction placements**. Recorded-arm/open-jaw controls complete **0/6** and
hold controls **0/6**. All 18 attempts are numerically valid, with no recorded
unexpected task contacts, carry-retention loss or native warnings in the six
placement attempts. They run the full 20 seconds, including after first success.

Final simulated placement error ranges from **0.489 to 0.551 mm**. These are
native simulator measurements against its authored target, not physical accuracy.
The six synthetic layouts vary initial/target coordinates within ±15 mm of a
single configuration; the lower-friction stratum uses 0.15 rather than 0.8. This
does not establish broad manipulation, shoe competence or sufficient video data.

The confirmation process ran with network access denied. Headless stepping,
native state/contact scoring, authored pose control and trace construction used
CPU. No learned policy, rendered observations or hardware control was involved.

## Timing includes the slow attempt

Placement attempt wall time ranges from 2.97 to **29.51 seconds** for 20 simulated
seconds. The slow nominal-03 attempt has a 78.14 ms control-boundary p95, a
1.479 s worst boundary and 251/1,000 boundaries above the 20 ms computation
budget. Other placement attempts also retain their deadline misses. The host was
not controlled for unrelated activity, and the cause of this variation was not
measured. These clocks do not establish real-time operation or a minimum laptop.

The recorded process high-water resident memory is 531,054,592 bytes, before
final bundle inspection and replay export. It includes imports, native models
and traces; it is neither per-episode memory nor a minimum RAM recommendation.

## Read and reproduce

- [Public task replay and complete receipt](https://research.dvidia.org/downloads/topics/physical-grounding/runs/2026-10-11-native-grasp/replay/)
- [Exact summary](report.json), [frozen protocol](protocol.json),
[original receipt manifest](receipt-integrity.json), [execution provenance](provenance.json).
- [Local instructions](../../docs/native-task-benchmark.md) and
[proposed next action-learning experiment](../../docs/native-grasp-learning.md).

This compact directory keeps summaries and the original manifest. Full raw
attempt JSON is in the public replay's `receipt/attempts/` directory. To inspect
the bundle, download the entire `receipt/` directory, retain its original
`integrity.json` name, and run `dvidia-reference-task inspect PATH`. This compact
directory alone is not a complete inspectable bundle.

Reproduction uses seed 521047, four nominal and two low-friction scenes, and
`--cohort confirmation` with the recorded source/runtime and pinned assets.
Those scenes are now known regression evidence. Use fresh scenarios after
changing a candidate; a directory or seed label alone cannot establish novelty.
The simulator's compiled-model hash may differ across platforms, so retain a
new platform-specific receipt instead of replacing this result.

The 537-test suite passed before this publication. The code remains an alpha
source candidate; earlier releases and Hugging Face Observation Lab are distinct.
1 change: 1 addition & 0 deletions benchmarks/native-grasp-20261011/protocol.json
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{"schema":"dvidia.reference-task-protocol/v1","profile":"ur5e-2f85-rigid-block-task-v0.1","contract":{"profile":"ur5e-2f85-rigid-block-task-v0.1","scope":"Authored native UR5e/Robotiq rigid-block manipulation, privileged state and synthetic commands. No shoe model, learned task policy, camera sensing, calibrated contact or physical qualification.","physical_robot_ready":false,"arm_joint_order":["shoulder_pan_joint","shoulder_lift_joint","elbow_joint","wrist_1_joint","wrist_2_joint","wrist_3_joint"],"gripper_linkage_order":["gripper/right_driver_joint","gripper/right_coupler_joint","gripper/right_spring_link_joint","gripper/right_follower_joint","gripper/left_driver_joint","gripper/left_coupler_joint","gripper/left_spring_link_joint","gripper/left_follower_joint"],"actions":{"joint_targets_rad":"six absolute joint targets, rad; ramped then gravity feedforward","gripper_0_255":"one native tendon servo target; 0 open / 255 close, not measured gap or force"},"observation_origin":"privileged_native_simulator_state","native_sensor_definitions":0,"observations":{"joint_position_rad":[6],"joint_velocity_rad_s":[6],"linkage_position_rad":[8],"tcp_position_m":[3],"tcp_quaternion_wxyz":[4],"object_position_m":[3],"object_quaternion_wxyz":[4],"grasp_contact_counts":"left/right native block contacts","pad_normal_forces_n":"native mj_contactForce, not deployable tactile sensing"},"contacts":"native collision proxies; object priority 1 overrides pad/table friction floors","object":"one free rigid box; no weld, grasp attachment or runtime object pose writes","success":"valid physics, retained bilateral table-free lift and supported destination carry completion, then released/settled destination dwell and tool retreat","carry_retention":{"bilateral_contact_required":"after sustained lift until supported at destination while held","contact_force_threshold_n":0.01,"consecutive_loss_tolerance_seconds":0.04,"loss_at_or_above_tolerance":"permanent failure, independent of teacher phase"},"fixed_base_collision_proxy":false},"cohort":"confirmation","seed":521047,"scenes":[{"id":"nominal-00","stratum":"nominal","seed":521047,"config":{"timestep":0.001,"control_dt":0.02,"timeout_s":20.0,"settle_s":0.5,"object_position_m":[-0.12102191784447544,0.48598939244410044,0.302],"target_position_m":[0.06731191885623511,0.485040893502719,0.3],"object_half_size_m":[0.02,0.02,0.02],"mass_kg":0.04,"friction":0.8,"table_top_m":0.28,"table_center_xy_m":[0.0,0.55],"table_half_size_xy_m":[0.32,0.26],"table_thickness_m":0.04,"goal_position_tolerance_m":0.025,"goal_dwell_s":0.25,"min_lift_m":0.06,"min_lift_dwell_s":0.1,"retreat_clearance_m":0.1,"joint_target_rate_rad_s":1.5,"jaw_target_rate_0_255_s":500.0,"tool_orientation_wxyz":[0.0,0.7071067811865476,0.7071067811865476,0.0]}},{"id":"nominal-01","stratum":"nominal","seed":521048,"config":{"timestep":0.001,"control_dt":0.02,"timeout_s":20.0,"settle_s":0.5,"object_position_m":[-0.13130390103702805,0.48052747200890933,0.302],"target_position_m":[0.06764916165706449,0.4879422849245993,0.3],"object_half_size_m":[0.02,0.02,0.02],"mass_kg":0.04,"friction":0.8,"table_top_m":0.28,"table_center_xy_m":[0.0,0.55],"table_half_size_xy_m":[0.32,0.26],"table_thickness_m":0.04,"goal_position_tolerance_m":0.025,"goal_dwell_s":0.25,"min_lift_m":0.06,"min_lift_dwell_s":0.1,"retreat_clearance_m":0.1,"joint_target_rate_rad_s":1.5,"jaw_target_rate_0_255_s":500.0,"tool_orientation_wxyz":[0.0,0.7071067811865476,0.7071067811865476,0.0]}},{"id":"nominal-02","stratum":"nominal","seed":521049,"config":{"timestep":0.001,"control_dt":0.02,"timeout_s":20.0,"settle_s":0.5,"object_position_m":[-0.13444185590820024,0.478946141966813,0.302],"target_position_m":[0.0681234861072569,0.487304618950296,0.3],"object_half_size_m":[0.02,0.02,0.02],"mass_kg":0.04,"friction":0.8,"table_top_m":0.28,"table_center_xy_m":[0.0,0.55],"table_half_size_xy_m":[0.32,0.26],"table_thickness_m":0.04,"goal_position_tolerance_m":0.025,"goal_dwell_s":0.25,"min_lift_m":0.06,"min_lift_dwell_s":0.1,"retreat_clearance_m":0.1,"joint_target_rate_rad_s":1.5,"jaw_target_rate_0_255_s":500.0,"tool_orientation_wxyz":[0.0,0.7071067811865476,0.7071067811865476,0.0]}},{"id":"nominal-03","stratum":"nominal","seed":521050,"config":{"timestep":0.001,"control_dt":0.02,"timeout_s":20.0,"settle_s":0.5,"object_position_m":[-0.12546270190646733,0.47731244649527643,0.302],"target_position_m":[0.08125537448652495,0.47723572262194736,0.3],"object_half_size_m":[0.02,0.02,0.02],"mass_kg":0.04,"friction":0.8,"table_top_m":0.28,"table_center_xy_m":[0.0,0.55],"table_half_size_xy_m":[0.32,0.26],"table_thickness_m":0.04,"goal_position_tolerance_m":0.025,"goal_dwell_s":0.25,"min_lift_m":0.06,"min_lift_dwell_s":0.1,"retreat_clearance_m":0.1,"joint_target_rate_rad_s":1.5,"jaw_target_rate_0_255_s":500.0,"tool_orientation_wxyz":[0.0,0.7071067811865476,0.7071067811865476,0.0]}},{"id":"low_friction-00","stratum":"low_friction","seed":521051,"config":{"timestep":0.001,"control_dt":0.02,"timeout_s":20.0,"settle_s":0.5,"object_position_m":[-0.127012804422937,0.4892900245959652,0.302],"target_position_m":[0.07742810415996912,0.4999092448715906,0.3],"object_half_size_m":[0.02,0.02,0.02],"mass_kg":0.04,"friction":0.15,"table_top_m":0.28,"table_center_xy_m":[0.0,0.55],"table_half_size_xy_m":[0.32,0.26],"table_thickness_m":0.04,"goal_position_tolerance_m":0.025,"goal_dwell_s":0.25,"min_lift_m":0.06,"min_lift_dwell_s":0.1,"retreat_clearance_m":0.1,"joint_target_rate_rad_s":1.5,"jaw_target_rate_0_255_s":500.0,"tool_orientation_wxyz":[0.0,0.7071067811865476,0.7071067811865476,0.0]}},{"id":"low_friction-01","stratum":"low_friction","seed":521052,"config":{"timestep":0.001,"control_dt":0.02,"timeout_s":20.0,"settle_s":0.5,"object_position_m":[-0.14707150885051776,0.49556896197880096,0.302],"target_position_m":[0.07433918934878232,0.5047373576545,0.3],"object_half_size_m":[0.02,0.02,0.02],"mass_kg":0.04,"friction":0.15,"table_top_m":0.28,"table_center_xy_m":[0.0,0.55],"table_half_size_xy_m":[0.32,0.26],"table_thickness_m":0.04,"goal_position_tolerance_m":0.025,"goal_dwell_s":0.25,"min_lift_m":0.06,"min_lift_dwell_s":0.1,"retreat_clearance_m":0.1,"joint_target_rate_rad_s":1.5,"jaw_target_rate_0_255_s":500.0,"tool_orientation_wxyz":[0.0,0.7071067811865476,0.7071067811865476,0.0]}}],"controls":["placement","replay_open_jaw","hold"],"planned_attempts":18,"implementation_sha256":{"reference_task.py":"bbc768ccc202b8dd41cbd235b206e6478806ce47c1f30ae15dc662c0c1edcc3e","reference_task_benchmark.py":"a04ab0918b1931601cc3861ab3b9b55ddac52a29c23d2e61ee8e467a87d461cf","reference_sim.py":"c1201d22eb13dd07a236ee63cc4ba68e129f6067e8db61ed41af2d820eb5e3b8","reference_assets.lock.json":"941eb6f9fd474635a590b82c66b59a9d5124d8e0d9ddc203006f5b921d8d2861"},"scope":"Authored, privileged-state rigid-block task in native reference physics. No learned policy, shoe physics, human-video action recovery or physical robot qualification.","physical_robot_ready":false,"learned_policy":false,"split_note":"Seeded synthetic layouts, not independent physical captures. Confirmation is a user declaration; this tool cannot establish whether a seed/layout was previously inspected.","comparison":"Open-jaw control replays the placement issued arm targets with jaws requested open; hold preserves initial arm targets. All attempts use the same scene timeout.","timing_boundary":"Attempt wall time includes asset checks, compilation, reset, planning, stepping and trace capture; separate inner timings identify narrower boundaries. No rendering, sensing, training, disk serialization, transport or physical actuation."}
Loading
Loading