Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .gitattributes
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
# Review the viewer source; this reproducible bundle includes upstream GLSL text.
src/dvidia_training/reference_replay.bundle.js linguist-generated=true -diff
16 changes: 14 additions & 2 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -83,6 +83,14 @@ jobs:
- uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6.3.0
with:
python-version: "3.12"
- uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4
with:
node-version: "22"
- name: Verify reproducible local viewer bundle
run: |
npm ci --ignore-scripts --prefix viewer
npm run --prefix viewer build
git diff --exit-code -- src/dvidia_training/reference_replay.bundle.js src/dvidia_training/reference_replay.third-party.txt
- name: Provision pinned reference models
run: |
python -m pip install ".[arm]"
Expand All @@ -91,17 +99,21 @@ jobs:
- name: Test compiled mechanics, collisions and replay scoring
env:
DVIDIA_REFERENCE_ASSETS: runs/reference-assets-ci
run: python -m unittest tests.test_reference_sim tests.test_embodiment_cli tests.test_embodiment_calibration -v
run: python -m unittest tests.test_reference_sim tests.test_reference_replay tests.test_embodiment_cli tests.test_embodiment_calibration -v
- name: Retain all numerical reference attempts
run: |
dvidia-embodiment benchmark runs/reference-assets-ci --hands --output runs/reference-bench-ci --repeats 1 --seconds 2
dvidia-embodiment inspect runs/reference-bench-ci
- name: Export native mesh and recorded 3D poses
run: dvidia-embodiment export-replay runs/reference-assets-ci runs/reference-bench-ci --output runs/reference-replay-ci
- name: Preserve reference evidence
if: always()
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
with:
name: hardware-reference-bench
path: runs/reference-bench-ci
path: |
runs/reference-bench-ci
runs/reference-replay-ci
if-no-files-found: ignore

motion:
Expand Down
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ dist/
runs/
examples/
wheelhouse/
node_modules/
.DS_Store
.env
.env.*
8 changes: 6 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

# DVIDIA Training

[Install locally](docs/installation.md) · [Hardware guide](docs/hardware.md) · [Execution map](docs/roadmap.md) · [Official site](https://dvidia.org/) · [Research](https://research.dvidia.org/papers/skillspace-footage-training/) · [Frozen benchmark evidence](benchmarks/sample-count-19047/README.md)
[Install locally](docs/installation.md) · [Hardware guide](docs/hardware.md) · [Execution map](docs/roadmap.md) · [Upcoming arms](docs/hardware-roadmap.md) · [Official site](https://dvidia.org/) · [Research](https://research.dvidia.org/papers/skillspace-footage-training/) · [Frozen benchmark evidence](benchmarks/sample-count-19047/README.md)

[Recorded demo](https://huggingface.co/spaces/Dvidia/dvidia-training) · [Synthetic datasets](https://huggingface.co/datasets/Dvidia/dvidia-training-examples) · [Pilot models](https://huggingface.co/Dvidia/dvidia-training-pilot)

Expand Down Expand Up @@ -124,7 +124,11 @@ UR5e arm and coupled Robotiq gripper, audits actual joint/actuator/collision
structure, and compares CPU physics at three timesteps. Optional left/right
five-finger references have separate contracts. A paired replay scorer measures
joint, tool and jaw errors once aligned physical recordings are available.
These tools use the `feat/hardware-reference-v0.1` candidate branch; the new
Export actual mesh geometry and recorded native poses to an offline interactive
3D replay with orbit controls, six-joint readouts and collision-proxy inspection.
The [upcoming hardware list](docs/hardware-roadmap.md) tracks YAM, OpenArm, PiPER,
ARX X5 and PAROL6 as planned additions requiring separate profiles and benchmarks.
These tools use the `feat/reference-3d-replay` candidate branch; the new
reference has no learned task policy or measured physical fidelity.

Generate an explicitly synthetic example, then train and verify its output:
Expand Down
45 changes: 40 additions & 5 deletions docs/hardware-reference.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,13 +13,13 @@ hardware controller or an acquired shoe skill.

## Install, provision once, then run offline

Use the candidate `feat/hardware-reference-v0.1` source branch until this feature
Use the candidate `feat/reference-3d-replay` source branch until this feature
is released. Python 3.12+, NumPy 2.5.3 and the optional MuJoCo 3.15.0 runtime are
required for simulation. FFmpeg is unnecessary for this mechanics-only bench.
Use `.venv\Scripts\COMMAND.exe` in Windows PowerShell.

```sh
git clone --branch feat/hardware-reference-v0.1 https://github.com/Dvidia-Inference/dvidia-training.git
git clone --branch feat/reference-3d-replay https://github.com/Dvidia-Inference/dvidia-training.git
cd dvidia-training
python3.12 -m venv .venv
.venv/bin/python -m pip install '.[arm]'
Expand All @@ -43,14 +43,49 @@ After provisioning, these commands use no network:
.venv/bin/dvidia-embodiment inspect runs/reference-bench
```

Open `runs/reference-bench/report.html` for a local schematic replay and timing /
numerical comparisons. Its poses come from native physics; it draws link and pad
centers, not the full mesh or contact surfaces. Optional native mesh viewing is
The original `runs/reference-bench/report.html` is a flat projection of recorded
3D link/pad centers with timing and numerical comparisons. The native physics
already models three-dimensional positions and orientations; the arm's six DOFs
are six rotating joints, not six spatial dimensions.

For an interactive 3D replay of the actual mesh and collision geometry, export
the frozen poses without rerunning physics:

```sh
.venv/bin/dvidia-embodiment export-replay runs/reference-assets runs/reference-bench --output runs/reference-3d
.venv/bin/python -m http.server 8290 --bind 127.0.0.1 --directory runs/reference-3d
```

Open `http://127.0.0.1:8290/`. Rotate/zoom the view, inspect all six arm joints and
the coupled gripper, switch visual meshes/collision proxies, and scrub the
recorded timeline. This browser requires WebGL2; ordinary integrated graphics
may suffice, but frame rate is measured per device rather than guaranteed. The
viewer is locally bundled and requests no CDN, API or model service. Node is
needed only to rebuild its browser bundle from source, not to export or view it.

The exporter verifies the frozen receipt and pinned assets, reconstructs recorded
joint/linkage poses with native forward kinematics and checks TCP/pad/link-center
agreement before publishing geometry and transforms. It never advances physics
or changes the original benchmark. Binary meshes/transforms are float32 display
data; the original numerical receipt stays intact under `receipt/`. Playback is
recorded simulation, not live physics, a trained task or a hardware controller.
Original asset licenses and bundled Three.js notices accompany every export.

Reconstruction requires the recorded compiled-model hash to match. Native
compilation can differ across platforms; a mismatch is rejected rather than
silently substituting another model. In that case, run a new benchmark on the
current machine and export that receipt, retaining both sets of results. An
already exported browser replay is portable and needs no native simulator.

Optional native mesh viewing is
available with `dvidia-embodiment view runs/reference-assets`. On macOS launch
the module using `.venv/bin/mjpython -m dvidia_training.embodiment_cli view
runs/reference-assets`; MuJoCo's graphical viewer needs its supported desktop
graphics context. Headless benchmarking requires no GPU or graphical viewer.

The [upcoming hardware list](hardware-roadmap.md) records the next arm candidates
and the evidence required for each supported hardware profile.

`inspect` checks bounded regular files, hashes and attempt-count consistency.
That establishes artifact consistency, not authentic measurements or recomputed
metrics. The inspector, asset verifier and paired trace scorer do not import the
Expand Down
55 changes: 55 additions & 0 deletions docs/hardware-roadmap.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
# Upcoming hardware support

Updated 11 October 2026. This is DVIDIA's planned support list, not a purchase
decision, delivery schedule or declaration of compatibility. All five additions
below remain **upcoming: no DVIDIA simulator profile, physical adapter or skill
qualification has been implemented for them**.

## Planned targets

Arm-axis counts below exclude the gripper. Fingers, coupled joints and independent
motor commands need their own inventory; a model never becomes compatible by
padding or truncating its action vector.

| Priority | Target | Intended role | Source availability and integration work |
| --- | --- | --- | --- |
| 1 | **YAM — 6 arm axes + gripper** | First candidate for a full-size, single-arm shoe grasp-and-place benchmark. | MIT control/model repository supplies MuJoCo, gripper control and teleoperation. Complete open manufacturing designs are not established by that repository. Pin the exact arm/gripper revision, audit its dynamics and fit physical response. [Official SDK/models](https://github.com/i2rt-robotics/i2rt). |
| 2 | **OpenArm — 7 arm axes + gripper per side** | Later two-arm coordination, with dexterous hands evaluated as separate end effectors. | Published hardware uses CERN-OHL-S-2.0; MuJoCo/control packages use Apache-2.0. Native LeRobot support currently connects hardware on Linux. Keep v1/v2 profiles distinct and account for hand mass, mounts, power and remaining object payload. [Hardware](https://github.com/enactic/openarm_hardware), [MuJoCo](https://github.com/enactic/openarm_mujoco), [LeRobot](https://huggingface.co/docs/lerobot/main/openarm). |
| 3 | **AgileX PiPER — 6 arm axes + gripper** | Alternative six-axis community/research target. | Open SDK and an MIT Menagerie model are available; full manufacturing openness is not established. Bind firmware, kinematic offsets, control modes and gripper calibration to the profile. [SDK](https://github.com/agilexrobotics/piper_sdk), [model](https://github.com/google-deepmind/mujoco_menagerie/tree/main/agilex_piper), [manufacturer](https://global.agilex.ai/products/piper). |
| 4 | **ARX X5 — 6 arm axes + gripper** | Research adapter for joint position, velocity and torque evidence; later multiple-arm control. | MIT research SDK and community LeRobot integration. Hardware licensing and an exact X5 simulation still need qualification; a differently named ARX model cannot substitute for it. [Research SDK](https://github.com/real-stanford/arx5-sdk), [LeRobot plugin](https://github.com/villekuosmanen/lerobot-arx5). |
| 5 | **PAROL6 — 6 arm axes + separate gripper** | Community-build desktop simulation target. | GPLv3 mechanics/software, Python and ROS tooling. The documented open-loop variant has limit-switch position sensing; verify actual joint feedback before claiming contact or tracking accuracy. Complete electronics openness is not established. [Project](https://github.com/Source-Robotics/PAROL6-Desktop-robot-arm), [specifications](https://source-robotics.github.io/PAROL-docs/page2_2/). |

The accessible **SO-101** entry profile remains planned alongside these additions.
It has five arm axes plus its gripper and official LeRobot collection support;
it requires a different action/observation contract from a six-axis arm.
[Official build](https://github.com/TheRobotStudio/SO-ARM100),
[LeRobot guide](https://huggingface.co/docs/lerobot/main/so101).

Our current [UR5e/Robotiq reference](hardware-reference.md) is already compiled and
numerically characterized. Its physical response, shoe-task competence and
serial-specific calibration remain unmeasured.

## What each addition must deliver

1. **Pinned simulation profile:** source revision and notices, named arm/gripper
joints, independent actuators and couplings, units, mounting/TCP frames,
geometry/collision coverage, masses, sensors and controller assumptions.
2. **Physical adapter:** calibration, requested/applied commands, measured feedback,
actual acquisition/arrival clocks, observation age and fault reporting.
3. **Simulation benchmark:** native mechanical checks, numerical sensitivity,
latency/throughput/memory with stated measurement boundaries, and all attempted
nominal, stress, failed-grasp and recovery task trials.
4. **Physical comparison:** matched command replays, independent calibration and
confirmation groups, pose/gap/contact/slip residuals, actual control latency
and retained failures. Importing a mesh cannot satisfy this step.
5. **Installable skill evidence:** a robot-specific policy and supported conditions,
with separate simulation and physical results tied to the exact artifact.

For the first shoe task, choose one actual arm and gripper before broadening
hardware work. Opening, finger geometry, slip, workspace and usable payload drive
that choice. Five-finger hands require a separate geometry, sensing, actuation and
benchmark contract.

Support follows the [execution map](roadmap.md). LeRobot can supply recording and
device connections while DVIDIA develops its own action model. Support for a
device does not establish that any existing skill model transfers to it.
6 changes: 6 additions & 0 deletions docs/roadmap.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,12 @@ motor/gripper response, contact/slip evidence and shoe-specific task training
precede physical skill qualification. Left/right five-finger assets are audited
separately; the two-arm assembly remains to be built.

The [upcoming hardware list](hardware-roadmap.md) now tracks YAM, OpenArm, PiPER,
ARX X5 and PAROL6 alongside the proposed SO-101 entry profile. YAM is the first
six-axis candidate; OpenArm is the later two-arm candidate. Each addition needs a
pinned simulator, physical adapter and its own benchmark/qualification evidence.
These are planned targets; their DVIDIA integrations have not been implemented.

| Order | Build and owner role | Current state | Evidence needed to advance |
| --- | --- | --- | --- |
| 1 | Collection planning — product + data | Local coverage planner, versioned briefs and declared source-group tallies implemented. | First skill's observable goal, operating envelope, actual recordings and reviewed capture lineage. |
Expand Down
2 changes: 1 addition & 1 deletion pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -46,4 +46,4 @@ dvidia-embodiment = "dvidia_training.embodiment_cli:main"
where = ["src"]

[tool.setuptools.package-data]
dvidia_training = ["*.html", "*.json", "LICENSE", "requirements.lock.txt"]
dvidia_training = ["*.html", "*.json", "*.js", "*.txt", "LICENSE", "requirements.lock.txt"]
11 changes: 11 additions & 0 deletions scripts/build-reference-viewer.mjs
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
import {readFile, writeFile} from 'node:fs/promises';
import {createRequire} from 'node:module';
import {fileURLToPath} from 'node:url';
import path from 'node:path';
const root = path.resolve(path.dirname(fileURLToPath(import.meta.url)), '..');
const require = createRequire(path.join(root, 'viewer/package.json'));
const {build} = require('esbuild');
await build({entryPoints:[path.join(root,'src/dvidia_training/reference_replay.js')],outfile:path.join(root,'src/dvidia_training/reference_replay.bundle.js'),bundle:true,minify:true,format:'iife',target:['es2020'],platform:'browser',alias:{'three':path.join(root,'viewer/node_modules/three/build/three.module.js'),'three/addons/controls/OrbitControls.js':path.join(root,'viewer/node_modules/three/examples/jsm/controls/OrbitControls.js')},nodePaths:[path.join(root,'viewer/node_modules')],legalComments:'none',charset:'utf8',banner:{js:'/* DVIDIA reference replay. MIT; bundled Three.js r180, MIT. See reference_replay.third-party.txt. */'}});
const license = await readFile(path.join(root,'viewer/node_modules/three/LICENSE'),'utf8');
await writeFile(path.join(root,'src/dvidia_training/reference_replay.third-party.txt'),'DVIDIA self-hosted reference replay\n\nThree.js 0.180.0 and OrbitControls\nhttps://github.com/mrdoob/three.js/tree/r180\n\n'+license);
console.log('Built reference_replay.bundle.js and reference_replay.third-party.txt with pinned local dependencies.');
7 changes: 7 additions & 0 deletions src/dvidia_training/embodiment_cli.py
Original file line number Diff line number Diff line change
Expand Up @@ -105,10 +105,17 @@ def main(argv=None):
score.add_argument("--output", type=Path, required=True)
inspect = commands.add_parser("inspect", help="Verify bounded report artifact integrity; no native simulator needed")
inspect.add_argument("directory", type=Path)
replay = commands.add_parser("export-replay", help="Export actual native geometry and recorded poses to an offline interactive 3D replay")
replay.add_argument("assets", type=Path)
replay.add_argument("bundle", type=Path)
replay.add_argument("--output", type=Path, required=True)
args = parser.parse_args(argv)
try:
if args.command == "inspect":
result = inspect_bundle(args.directory)
elif args.command == "export-replay":
from .reference_replay import export_replay
result = export_replay(args.assets, args.bundle, args.output)
elif args.command == "score-paired":
from .embodiment_calibration import load_trace, score_trace
result = score_trace(load_trace(args.trace))
Expand Down
3,835 changes: 3,835 additions & 0 deletions src/dvidia_training/reference_replay.bundle.js

Large diffs are not rendered by default.

Loading
Loading