Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 8 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,9 +35,10 @@ controller is authored; shoe physics and physical transfer remain unqualified.

[Hardware matched robot simulation](reports/Hardware%20matched%20robot%20simulation.md)
now includes an [interactive 3D arm/gripper replay](https://research.dvidia.org/downloads/topics/physical-grounding/runs/2026-10-11-hardware-reference/replay/)
and the [upcoming hardware support map](https://github.com/Dvidia-Inference/dvidia-training/blob/93fc549a370cdd448b49380c5beff3b2641c2795/docs/hardware-roadmap.md).
YAM, OpenArm, PiPER, ARX X5 and PAROL6 are planned targets, with separate simulator,
adapter and benchmark gates. The replay displays frozen native poses and geometry;
and the [hardware support map](https://github.com/Dvidia-Inference/dvidia-training/blob/928fed79eca09a1a5081aae7720ab78c2e5b817c/docs/hardware-roadmap.md).
PiPER and SO-101 have runnable CPU simulation profiles with unloaded numerical
receipts; their object-task and physical adapter qualification remain open.
YAM, OpenArm, ARX X5 and PAROL6 remain planned integration targets. The replay displays frozen native poses and geometry;
it adds no live physics, learned skill or physical-fidelity result.

[DVIDIA Observation Lab pilot](reports/DVIDIA%20Observation%20Lab%20pilot.md) publishes our local video detection and review workflow: sampled object/palm proposals, editable timeline revisions and JSON export. Each recorded engineering run completed six public excerpts (19 seconds and 38 sampled frames); independent annotation accuracy remains unmeasured. [Read the paper](https://research.dvidia.org/papers/dvidia-observation-lab/), [install the versioned open-source release](https://github.com/Dvidia-Inference/dvidia-training/releases/tag/observation-lab-v0.1.0), or [try the synthetic browser review demo](https://huggingface.co/spaces/Dvidia/observation-lab). The hosted demo uses authored clips and boxes; real detection runs locally with separately provisioned upstream models. No action captions, validated depth or robot skill are established.
Expand Down Expand Up @@ -90,3 +91,7 @@ The [agenda](reports/Physical%20skill%20research%20agenda.md) responds to Sam Pa
Citation keys such as `zhang2020hands` are local to each topic's `references.bib`; the same keys appear in `topic.json`. Topic prose links keys to primary URLs. Living documents use an explicitly labelled access year when a publication date is unavailable. Cite original work for its findings and [CITATION.cff](CITATION.cff) for these protocols. Record an immutable revision when actually running code or using a dataset.

Original writing and metadata use [CC BY 4.0](LICENSE); original code uses [MIT](LICENSE-CODE). Referenced code, checkpoints, media and datasets keep their own terms. Repository openness does not establish permission to redistribute a cited dataset.

## Robot simulation library

[Public Robot Simulation Lab](https://huggingface.co/spaces/Dvidia/robot-simulation-lab) combines the recorded UR5e contact task with separate PiPER and SO-101 CPU mechanics receipts. [Run the open-source profiles locally](https://github.com/Dvidia-Inference/dvidia-training/blob/928fed79eca09a1a5081aae7720ab78c2e5b817c/docs/robot-simulation-library.md). Additional profile support does not transfer task or hardware qualification.
27 changes: 22 additions & 5 deletions docs/downloads/reports/Hardware matched robot simulation.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,19 +28,19 @@ The exporter verifies the frozen receipt and pinned assets, reconstructs recorde

The ready-made browser replay can be viewed on another supported browser without MuJoCo. Reconstructing an original numerical receipt requires its compiled-model hash to match; native builds can differ between platforms. When they differ, generate and retain a separate benchmark and replay on the current machine alongside the original results.

## Upcoming hardware profiles
## Hardware integration roadmap

The following support list is a plan, not a purchase decision or implemented compatibility. **All five additions remain upcoming.** Each needs a pinned simulator, physical adapter and benchmark evidence tied to its exact hardware/controller revision. Arm-axis counts exclude the gripper; a future five-finger hand has a separate actuation and sensing contract.
The following list separates runnable numerical simulation from planned integration. **PiPER now has a pinned CPU simulation profile; YAM, OpenArm, ARX X5 and PAROL6 remain upcoming.** All five still need their own object-task qualification and physical adapter evidence tied to the exact hardware/controller revision. Arm-axis counts exclude the gripper; a future five-finger hand has a separate actuation and sensing contract.

| Priority | Target | Intended role and remaining work |
| --- | --- | --- |
| 1 | **YAM — 6 arm axes + gripper** | First candidate for a full-size shoe grasp-and-place benchmark. MIT control/models include MuJoCo and teleoperation; complete open manufacturing designs are not established. Pin arm/gripper revisions and measure their physical response. [Official SDK/models](https://github.com/i2rt-robotics/i2rt). |
| 2 | **OpenArm — 7 arm axes + gripper per side** | Later two-arm coordination. CERN-OHL-S-2.0 hardware and Apache-2.0 simulation; native LeRobot connection currently requires Linux. Keep v1/v2 profiles distinct and assess hand mounts, mass, power and remaining payload. [Hardware](https://github.com/enactic/openarm_hardware), [MuJoCo](https://github.com/enactic/openarm_mujoco), [LeRobot](https://huggingface.co/docs/lerobot/main/openarm). |
| 3 | **AgileX PiPER — 6 arm axes + gripper** | Alternative six-axis research/community target. Open SDK and MIT Menagerie model; full manufacturing openness is not established. Bind firmware, offsets, control modes and jaw calibration. [SDK](https://github.com/agilexrobotics/piper_sdk), [model](https://github.com/google-deepmind/mujoco_menagerie/tree/main/agilex_piper). |
| 3 | **AgileX PiPER — 6 arm axes + gripper** | Runnable pinned CPU profile with unloaded numerical receipts; object-task and physical qualification remain open. Open SDK and MIT Menagerie model; full manufacturing openness is not established. Bind firmware, offsets, control modes and jaw calibration for a physical adapter. [SDK](https://github.com/agilexrobotics/piper_sdk), [model](https://github.com/google-deepmind/mujoco_menagerie/tree/main/agilex_piper). |
| 4 | **ARX X5 — 6 arm axes + gripper** | Joint position/velocity/torque research adapter and later multiple-arm control. MIT research SDK and community LeRobot plugin; exact X5 simulation and hardware licensing need qualification. [SDK](https://github.com/real-stanford/arx5-sdk), [plugin](https://github.com/villekuosmanen/lerobot-arx5). |
| 5 | **PAROL6 — 6 arm axes + separate gripper** | Community-build desktop target. GPLv3 mechanics/software and ROS/Python tools; the documented open-loop version has limit-switch sensing, so actual joint feedback requires verification. Complete electronics openness is not established. [Project](https://github.com/Source-Robotics/PAROL6-Desktop-robot-arm), [specifications](https://source-robotics.github.io/PAROL-docs/page2_2/). |

SO-101 remains the accessible planned entry profile, with five arm axes plus a gripper. The current UR5e/Robotiq reference is separately characterized. Gripper opening, finger geometry, slip, workspace and usable payload drive the first shoe hardware choice. Device support advances separately from learned skill and physical qualification. ([DVIDIA hardware support map](https://github.com/Dvidia-Inference/dvidia-training/blob/93fc549a370cdd448b49380c5beff3b2641c2795/docs/hardware-roadmap.md))
SO-101 is the accessible entry profile, with five arm axes plus a gripper. Its pinned CPU simulation is now runnable; its object-task and physical adapter qualification remain open. The UR5e/Robotiq reference is separately characterized. Gripper opening, finger geometry, slip, workspace and usable payload drive the first shoe hardware choice. Device support advances separately from learned skill and physical qualification. ([DVIDIA hardware support map](https://github.com/Dvidia-Inference/dvidia-training/blob/928fed79eca09a1a5081aae7720ab78c2e5b817c/docs/hardware-roadmap.md))

## Match observable mechanics before calling it fidelity

Expand Down Expand Up @@ -76,7 +76,7 @@ Use fixed issued-command replays to expose timestep sensitivity, then measure co

LeRobot offers hardware/teleoperation interfaces, cameras and dataset tooling without requiring a supplied policy architecture. Its official custom-policy interface supports independently packaged models with training, reset and action-selection methods. DVIDIA can reuse that infrastructure while developing its own objectives, model and skill artifacts. **No DVIDIA/LeRobot integration or robot action policy is built by this reference release.** The earlier compact transition predictor forecasts motion conditional on a command; it does not choose the command. A proposed first seam is a validated local dataset reader and strict observation/action adapter, followed by an original DVIDIA trainer. ([Custom hardware guide](https://huggingface.co/docs/lerobot/main/integrate_hardware), [Custom policy guide](https://huggingface.co/docs/lerobot/main/bring_your_own_policies))

Hardware selection remains explicit. SO-101's six motor channels are shoulder pan, shoulder lift, elbow flex, wrist flex, wrist roll and gripper: **five arm axes plus a gripper, not six arm axes plus a gripper**. Its body positions can use degrees or normalized values, and the jaw uses a normalized command. A separate named profile must specify ordering, units, calibration, action representation and requested versus actually sent targets; it cannot accept DVIDIA's current six-axis/radian artifact unchanged. ([SO-101 guide](https://huggingface.co/docs/lerobot/main/so101), [Follower implementation](https://github.com/huggingface/lerobot/blob/main/src/lerobot/robots/so_follower/so_follower.py), [Action representations](https://huggingface.co/docs/lerobot/main/action_representations))
Hardware selection remains explicit. SO-101's six motor channels are shoulder pan, shoulder lift, elbow flex, wrist flex, wrist roll and gripper: **five arm axes plus a gripper, not six arm axes plus a gripper**. LeRobot's physical follower interface can use degrees or normalized body positions, and its jaw uses a normalized command. A physical adapter must specify ordering, units, calibration, action representation and requested versus actually sent targets; it cannot accept DVIDIA's six-axis reference artifact or the native SO-101 profile's radian targets unchanged. ([SO-101 guide](https://huggingface.co/docs/lerobot/main/so101), [Follower implementation](https://github.com/huggingface/lerobot/blob/main/src/lerobot/robots/so_follower/so_follower.py), [Action representations](https://huggingface.co/docs/lerobot/main/action_representations))

LeRobotDataset v3 stores state/action rows, camera video and episode/schema metadata, and recordings can stay local. However, the recording guide warns that frame-index/FPS timestamps can make motion appear faster when acquisition misses the requested cadence. DVIDIA needs a sidecar with **actual monotonic acquisition, arrival, command and feedback clocks**, camera calibration, missed samples, controller settings and physical source groups. Nominal timestamps alone cannot fit actuator delay or validate physical speed. Pin and test the chosen LeRobot version and review datasets/assets separately from its Apache-2.0 software license. ([Dataset format](https://huggingface.co/docs/lerobot/main/lerobot-dataset-v3), [Recording and cadence guide](https://huggingface.co/docs/lerobot/main/il_robots), [Software license](https://github.com/huggingface/lerobot/blob/main/LICENSE))

Expand Down Expand Up @@ -106,6 +106,23 @@ Adjacent-timestep comparisons align 201 elapsed boundaries per comparison, inclu

A separate authored sphere-overlap diagnostic produces one arm contact with active collision masks and zero with those masks disabled. It checks that the selected proxy participates in collision detection; it is not a moving collision-response or avoidance test. The software checkpoint also passed 488 local tests, including native contract checks. CPU-only headless characterization is demonstrated for this unloaded reference workload; shoe contact, complete training throughput, minimum hardware, physical accuracy and the composed two-arm system remain unmeasured. ([Collision receipt](https://research.dvidia.org/downloads/topics/physical-grounding/runs/2026-10-11-hardware-reference/report.json), [Reference instructions](https://github.com/Dvidia-Inference/dvidia-training/blob/c95a4385dae4b654b01ef75f56cbcc144fde2f5c/docs/hardware-reference.md))

## PiPER and SO-101 now have runnable CPU profiles

The library now includes pinned **PiPER (six arm axes plus one gripper command)** and **SO-101 (five arm axes plus one gripper command)** profiles. Both expose explicit model fetch, offline asset verification, native audit and bounded numerical benchmarking. The packaged models preserve upstream geometry and mechanics; DVIDIA supplies the profile tooling and evidence contracts. PiPER's slider command is single-finger travel in metres with a coupled opposite finger. SO-101's moving jaw uses a hinge target in radians. Neither is a calibrated physical gap or force command, and SO-101 must not be labeled as a six-axis arm. ([Local simulation guide](https://github.com/Dvidia-Inference/dvidia-training/blob/928fed79eca09a1a5081aae7720ab78c2e5b817c/docs/robot-simulation-library.md), [Pinned model contracts](https://github.com/Dvidia-Inference/dvidia-training/blob/928fed79eca09a1a5081aae7720ab78c2e5b817c/src/dvidia_training/robot_profiles.lock.json))

A serial run with process-level network denial retained six attempts per profile: hold, small arm sweep and gripper cycle, each repeated twice for two simulated seconds at a 1 ms physics step and 20 ms command boundary. These repeats check numerical mechanics; they are not independent task scenes or a learned-policy evaluation. Both profiles produced **6/6 numerically valid attempts**. Focused tests separately confirm that commanded arm and jaw cases change the native state compared with holding the starting commands.

| Profile | Simulated seconds / complete-attempt wall second | Per-attempt 20 ms computation p95 | Computation windows above 20 ms |
| --- | ---: | ---: | ---: |
| PiPER | 4.38–14.62× | 0.446–0.500 ms | 0/600 |
| SO-101 | 14.73–20.17× | 0.396–0.744 ms | 0/600 |

All attempts remain in the [Robot Simulation Lab](https://huggingface.co/spaces/Dvidia/robot-simulation-lab), including PiPER's slower first attempt. Wall time includes asset checks, compilation and recorded state construction; output serialization, browser rendering, physical sensing/transport and learned inference are outside that measurement. Runtime was macOS/Darwin arm64 with Python 3.12.9, MuJoCo 3.15.0 and NumPy 2.5.3. These CPU runs used no discrete-GPU workload; host load was not controlled, and universal minimum memory or real-time hardware behavior is not established. The exact bundles include source/runtime identities, native controls and effort units, gravity settings, every recorded trajectory and original licenses. ([PiPER receipt](https://huggingface.co/spaces/Dvidia/robot-simulation-lab/resolve/main/profiles/piper/report.json), [SO-101 receipt](https://huggingface.co/spaces/Dvidia/robot-simulation-lab/resolve/main/profiles/so101/report.json))

PiPER simulation assets are MIT; SO-101 assets are Apache-2.0. The latter has published build/design files and LeRobot guidance, while an open PiPER SDK does not by itself establish open manufacturing designs. The models' original notices are retained, and their attribution is separate from DVIDIA software. ([SO-101 design](https://github.com/TheRobotStudio/SO-ARM100), [LeRobot guide](https://huggingface.co/docs/lerobot/so101), [PiPER SDK](https://github.com/agilexrobotics/piper_sdk))

These additions are ready for local unloaded simulation. Each still needs its own object-task environment, grasp/release controls, calibrated observations and fresh closed-loop policy benchmark. YAM, OpenArm, ARX X5 and PAROL6 remain further integration targets; bimanual and dexterous manipulation retain separate contracts. The hosted lab displays the UR5e contact replay separately from these numerical receipts, so additional model support cannot be mistaken for inherited grasp competence.

## Release an evidence-bearing skill, not an appearance match

The installable unit should carry the policy, skill goal, observations/actions and units, named embodiment ordering, adapter, dependency hashes, calibration requirements, supported conditions, refusal/recovery behavior and benchmark receipts. Imported asset structure, characterized simulation and measured hardware agreement advance separately. A reference-only or simulation-qualified package remains useful for research while declaring physical execution unqualified. This follows the current DVIDIA roadmap's separation of collection, learning, simulation and physical qualification. ([Execution map](https://github.com/Dvidia-Inference/dvidia-training/blob/9172c37baeed5925e7bea3af2c92b0bf37a63c24/docs/roadmap.md))
Expand Down
2 changes: 2 additions & 0 deletions docs/downloads/reports/Native grasp task benchmark.md
Original file line number Diff line number Diff line change
Expand Up @@ -82,3 +82,5 @@ The product opportunity is the loop connecting reviewed skill-space evidence, bo
Run the benchmark using the [installation and evaluation guide](https://github.com/Dvidia-Inference/dvidia-training/blob/ac920cbf70147a7edeb0c706a9b71047f318e12f/docs/native-task-benchmark.md). The implementation and review are available in [GitHub PR #11](https://github.com/Dvidia-Inference/dvidia-training/pull/11).

Published by **DVIDIA** · [hello@dvidia.org](mailto:hello@dvidia.org) · [@iammrriver on X](https://x.com/iammrriver)

The [public Robot Simulation Lab](https://huggingface.co/spaces/Dvidia/robot-simulation-lab) also provides separate unloaded PiPER and SO-101 receipts and installation instructions. [Browser acceptance evidence](https://research.dvidia.org/downloads/topics/physical-grounding/runs/2026-10-11-native-grasp/replay-browser/acceptance.json) checks the recorded task replay on desktop and narrow screens.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
# Recorded task replay browser checks

These checks inspect the frozen native replay; they do not rerun physics. All nine displayed attempts and their source states are checked. The complete source denominator remains eighteen attempts. Screenshots show the initial block, elevated carry and released placement. The 320px layout retains readable controls. No browser errors or external requests were observed.
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
{
"scope": "Browser recorded-task replay acceptance; no learned-policy or physical qualification",
"demonstrated_attempt_id": "nominal-00--placement",
"recorded_success": true,
"carry_frame": 291,
"carry_seconds": 5.820000000000278,
"rendered_attempts": 9,
"full_benchmark_attempted": 18,
"checks": [
"Native block, table and target geometry; six joint readouts; authored-controller scope and full source denominator",
"Every selected outcome matches its sealed source; every block pose matches the recorded native state; all controls replay to their terminal row",
"Initial, observed carry and terminal target views captured from actual state; completed source labels require measured lifted transfer and final target tolerance",
"Recorded playback, collision mode and idle rendering; 390 px and 320 px layouts keep 44 px controls without horizontal overflow"
],
"errors": [],
"remote": []
}
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
{
"scope": "Browser replay acceptance, not fresh physics or hardware validation",
"tested_replay_integrity_sha256": "f4a334c177eab76942ee0ebb282138340040fc22be3f6377713d332fc572ba21",
"harness_source_sha256": "0d41eba08d7717d9b40e446ffa59ee017beeb1c20d9e40d4b73d62d82d61f729",
"checks": "All selected body positions, recorded outcomes, frame counts, paused rendering, desktop and mobile layout; no browser errors or external requests",
"screenshots": "Initial, measured carry, final placement and 320px mobile view; independently visually reviewed"
}
Loading
Loading