Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions .gitattributes
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
# Generated viewer bundles retain upstream GLSL text; review their source repository.
**/reference_replay.bundle.js linguist-generated=true -diff
# These byte-identical copies preserve the original receipt beside the replay.
**/2026-10-11-hardware-reference/replay/receipt/report.* linguist-generated=true -diff
7 changes: 7 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,13 @@ All four covers are **AI-generated editorial illustrations**, not footage, measu

## Latest studies

[Hardware matched robot simulation](reports/Hardware%20matched%20robot%20simulation.md)
now includes an [interactive 3D arm/gripper replay](https://research.dvidia.org/downloads/topics/physical-grounding/runs/2026-10-11-hardware-reference/replay/)
and the [upcoming hardware support map](https://github.com/Dvidia-Inference/dvidia-training/blob/93fc549a370cdd448b49380c5beff3b2641c2795/docs/hardware-roadmap.md).
YAM, OpenArm, PiPER, ARX X5 and PAROL6 are planned targets, with separate simulator,
adapter and benchmark gates. The replay displays frozen native poses and geometry;
it adds no live physics, learned skill or physical-fidelity result.

[DVIDIA Observation Lab pilot](reports/DVIDIA%20Observation%20Lab%20pilot.md) publishes our local video detection and review workflow: sampled object/palm proposals, editable timeline revisions and JSON export. Each recorded engineering run completed six public excerpts (19 seconds and 38 sampled frames); independent annotation accuracy remains unmeasured. [Read the paper](https://research.dvidia.org/papers/dvidia-observation-lab/), [install the versioned open-source release](https://github.com/Dvidia-Inference/dvidia-training/releases/tag/observation-lab-v0.1.0), or [try the synthetic browser review demo](https://huggingface.co/spaces/Dvidia/observation-lab). The hosted demo uses authored clips and boxes; real detection runs locally with separately provisioned upstream models. No action captions, validated depth or robot skill are established.

The [image-motion source alpha](https://github.com/Dvidia-Inference/dvidia-training/pull/7) adds manually seeded causal image tracking, pixel speed, short-horizon forecasts and local replay. Its six authored cases cover 720 frames and process in 1.454 seconds for 24 source seconds on one CPU run; occlusion/low-texture losses and an 89.975 px worst 500 ms reversal forecast remain explicit. [Exact source, benchmark and installation](https://github.com/Dvidia-Inference/dvidia-training/blob/ade2b541d8ca4a785bbd1effdfd33eb2a306f150/docs/motion-observations.md). The v0.1 release/Space remain separate; real-scene accuracy, metric speed, live robot latency and the full v0.2 are not established.
Expand Down
26 changes: 26 additions & 0 deletions docs/downloads/reports/Hardware matched robot simulation.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,32 @@ The assembly has 24 active collision geoms. Each of its six moving arm links has

Pin the asset revision `0059d4335f8156206f63a35662313385f7ad6d74`, model and mesh hashes, derivation notes and per-directory licenses. The arm's BSD-3-Clause and gripper's BSD-2-Clause notices accompany any redistributed assets; DVIDIA's original MIT software license does not replace them. A cached dependency bundle supports local operation after provisioning. Downloading the bundle is a separate setup step. ([Pinned revision](https://github.com/google-deepmind/mujoco_menagerie/commit/0059d4335f8156206f63a35662313385f7ad6d74), [Arm license](https://github.com/google-deepmind/mujoco_menagerie/blob/0059d4335f8156206f63a35662313385f7ad6d74/universal_robots_ur5e/LICENSE), [Gripper license](https://github.com/google-deepmind/mujoco_menagerie/blob/0059d4335f8156206f63a35662313385f7ad6d74/robotiq_2f85/LICENSE))

## Inspect the arm and gripper in three dimensions

**[Open the interactive 3D replay](https://research.dvidia.org/downloads/topics/physical-grounding/runs/2026-10-11-hardware-reference/replay/index.html).** The original numerical report draws a flat projection of recorded link centers. The native physics already simulates three-dimensional positions, orientations and contact geometry; six arm DOFs mean six rotating joints. The new replay renders the actual compiled arm/gripper meshes and collision proxies, with orbit/zoom controls, preset views, six-joint readouts and a recorded timeline.

![Native UR5e arm and Robotiq gripper displayed in the interactive 3D replay](https://research.dvidia.org/downloads/topics/physical-grounding/runs/2026-10-11-hardware-reference/replay-browser/arm-view.png)

*The native mesh replay at 1.10 seconds in the recorded arm sweep. Eight browser acceptance groups passed across desktop, phone and landscape layouts; this is display validation, not physical-fidelity or rendering-speed evidence. [Browser checks](https://research.dvidia.org/downloads/topics/physical-grounding/runs/2026-10-11-hardware-reference/replay-browser/acceptance.json).*

The exporter verifies the frozen receipt and pinned assets, reconstructs recorded arm/linkage poses with native forward kinematics and checks tool/pad/link-center agreement before writing display transforms. It does not step physics or rerun the benchmark. Mesh vertices and transforms use float32 display storage; exact repeated vertices are deduplicated without changing triangle coordinates. Original numerical files remain unchanged and downloadable under the replay's `receipt/` directory. The viewer is bundled locally with retained asset and Three.js notices; it requires browser WebGL2 and reports loading/graphics failures explicitly. No new rendering-speed, task-success or physical-fidelity claim follows from this display. ([Export and offline guide](https://github.com/Dvidia-Inference/dvidia-training/blob/93fc549a370cdd448b49380c5beff3b2641c2795/docs/hardware-reference.md), [Original frozen report](https://research.dvidia.org/downloads/topics/physical-grounding/runs/2026-10-11-hardware-reference/report.html))

The ready-made browser replay can be viewed on another supported browser without MuJoCo. Reconstructing an original numerical receipt requires its compiled-model hash to match; native builds can differ between platforms. When they differ, generate and retain a separate benchmark and replay on the current machine alongside the original results.

## Upcoming hardware profiles

The following support list is a plan, not a purchase decision or implemented compatibility. **All five additions remain upcoming.** Each needs a pinned simulator, physical adapter and benchmark evidence tied to its exact hardware/controller revision. Arm-axis counts exclude the gripper; a future five-finger hand has a separate actuation and sensing contract.

| Priority | Target | Intended role and remaining work |
| --- | --- | --- |
| 1 | **YAM — 6 arm axes + gripper** | First candidate for a full-size shoe grasp-and-place benchmark. MIT control/models include MuJoCo and teleoperation; complete open manufacturing designs are not established. Pin arm/gripper revisions and measure their physical response. [Official SDK/models](https://github.com/i2rt-robotics/i2rt). |
| 2 | **OpenArm — 7 arm axes + gripper per side** | Later two-arm coordination. CERN-OHL-S-2.0 hardware and Apache-2.0 simulation; native LeRobot connection currently requires Linux. Keep v1/v2 profiles distinct and assess hand mounts, mass, power and remaining payload. [Hardware](https://github.com/enactic/openarm_hardware), [MuJoCo](https://github.com/enactic/openarm_mujoco), [LeRobot](https://huggingface.co/docs/lerobot/main/openarm). |
| 3 | **AgileX PiPER — 6 arm axes + gripper** | Alternative six-axis research/community target. Open SDK and MIT Menagerie model; full manufacturing openness is not established. Bind firmware, offsets, control modes and jaw calibration. [SDK](https://github.com/agilexrobotics/piper_sdk), [model](https://github.com/google-deepmind/mujoco_menagerie/tree/main/agilex_piper). |
| 4 | **ARX X5 — 6 arm axes + gripper** | Joint position/velocity/torque research adapter and later multiple-arm control. MIT research SDK and community LeRobot plugin; exact X5 simulation and hardware licensing need qualification. [SDK](https://github.com/real-stanford/arx5-sdk), [plugin](https://github.com/villekuosmanen/lerobot-arx5). |
| 5 | **PAROL6 — 6 arm axes + separate gripper** | Community-build desktop target. GPLv3 mechanics/software and ROS/Python tools; the documented open-loop version has limit-switch sensing, so actual joint feedback requires verification. Complete electronics openness is not established. [Project](https://github.com/Source-Robotics/PAROL6-Desktop-robot-arm), [specifications](https://source-robotics.github.io/PAROL-docs/page2_2/). |

SO-101 remains the accessible planned entry profile, with five arm axes plus a gripper. The current UR5e/Robotiq reference is separately characterized. Gripper opening, finger geometry, slip, workspace and usable payload drive the first shoe hardware choice. Device support advances separately from learned skill and physical qualification. ([DVIDIA hardware support map](https://github.com/Dvidia-Inference/dvidia-training/blob/93fc549a370cdd448b49380c5beff3b2641c2795/docs/hardware-roadmap.md))

## Match observable mechanics before calling it fidelity

The first physical calibration should fit base and tool transforms and serial-specific kinematics on measured poses, then score untouched poses throughout the intended tabletop workspace. Universal Robots warns that using generic calibration can introduce end-effector discrepancies of centimeters. Report independently measured tool-position and orientation residuals, reference uncertainty, operating conditions and the calibration revision. Joint encoder agreement alone cannot reveal a wrong base or tool transform. ([UR calibration documentation](https://docs.universal-robots.com/Universal_Robots_ROS_Documentation/kilted/doc/ur_robot_driver/ur_calibration/doc/index.html))
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
# Native 3D replay browser checks

Eight browser acceptance groups passed for the v3 replay with zero page errors and zero external requests. The screenshot shows native geometry at a recorded arm-sweep frame, not a photograph or learned task. Desktop, phone and landscape layouts, camera controls, timeline, joint readouts, partial-case selection, visible error handling and uncompressed fallback were checked.

Rendering used Chromium headless with SwiftShader and Playwright 1.64.0. This is a UI acceptance check, not a GPU performance, physical-fidelity or robot-skill benchmark. Source and replay hashes are recorded in source.json.

DVIDIA UI/software: MIT. The rendered UR5e and Robotiq models retain their BSD-3-Clause and BSD-2-Clause notices under ../replay/licenses/. Viewer dependency notices are under ../replay/reference_replay.third-party.txt.
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
{
"scope": "Native recorded 3D replay workflow acceptance; not physical accuracy, task success or live reaction timing",
"checks": [
"Native visual groups and active collision proxies are separate; six joints and recorded-replay scope are visible",
"Joint and gripper readouts exactly follow the selected measured row",
"Keyboard orbit and camera presets alter the rendered 3D view",
"Playback, overlays and pause work; paused unchanged scenes stop requesting renders",
"Desktop, phone and landscape layouts have no horizontal overflow and preserve 44px controls",
"Missing motion/step combinations are withheld rather than causing a replay error",
"Malformed body layout and unavailable WebGL2 fail visibly without fabricated geometry",
"Browsers without DecompressionStream use exact uncompressed local binaries"
],
"errors": [],
"remote": []
}
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
{
"schema": "dvidia.reference-replay-browser/v1",
"source_commit": "2ebbeb7212d49b5a7080d2da794b90b64d04195e",
"scope": "Recorded native 3D replay browser acceptance, not physical accuracy or a rendering-speed benchmark",
"browser": "Chromium headless, SwiftShader",
"playwright": "1.64.0",
"read_only_replay": true,
"replay_integrity_sha256": "ed2ef2db273fd921a2bb936f3779658d6d8e4011cfd39331795463f54ce05f1e",
"viewer_html_sha256": "edaceba9a6f57ca1bfc0bd51ccd1056c19344844a6d533054cc0496c961fa42b",
"viewer_bundle_sha256": "243fad0f36c087015f67c22016c5bc24a199f6930b4f0f490a9f6cda9946765b",
"files": {
"arm-view.png": {
"bytes": 163597,
"sha256": "f9ec3ac405133195fb3c26309ecda4edbe76acb22c8c0644c1451f4c62c2f6c1"
},
"acceptance.json": {
"bytes": 889,
"sha256": "4ac462e52ab334efa30488ab4a402ae3b4392117b976149fce9c1f8230734027"
}
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
MIT License

Copyright (c) 2026 DVIDIA contributors

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
# DVIDIA native reference replay

Recorded authored simulation from the frozen reference receipt. Native mesh geometry and recorded joint/linkage positions, reconstructed with mj_forward only. No live physics, robot control, task policy, physical accuracy or new speed measurement.

Serve this directory with a local static HTTP server and open index.html. All geometry, motion, viewer code, receipts and license notices are local. Receipt hashes establish consistency, not authenticity. The viewer uses Float32 body poses; receipt reconstruction checks used native Float64.

DVIDIA exporter/viewer code: MIT. UR5e model: BSD-3-Clause. Robotiq 2F-85 model: BSD-2-Clause. Original model notices and READMEs are under licenses/. Viewer dependencies are listed in reference_replay.third-party.txt.
Binary file not shown.
Binary file not shown.
Loading
Loading