Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
51 changes: 38 additions & 13 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,10 +1,12 @@
<p align="center">
<a href="https://github.com/openrobokit/retargetingkit"><img src="https://raw.githubusercontent.com/openrobokit/retargetingkit/main/docs/assets/retargetingkit-banner.webp" alt="RetargetingKit" width="100%"></a>
<a href="https://github.com/openrobokit/retargetingkit"><img src="https://openrobokit.com/assets/docs/retargetingkit/assets/retargetingkit-banner.webp" alt="RetargetingKit" width="100%"></a>
</p>

<p align="center">
<a href="pyproject.toml"><img src="https://img.shields.io/badge/python-3.10%2B-blue" alt="Python 3.10+"></a>
<a href="https://github.com/openrobokit/retargetingkit/actions/workflows/build.yml"><img src="https://img.shields.io/badge/CI-GitHub_Actions-blue" alt="GitHub Actions"></a>
<a href="https://openrobokit.com/retargetingkit"><img src="https://img.shields.io/badge/homepage-RetargetingKit-blue" alt="Homepage"></a>
<a href="https://openrobokit.com/docs/retargetingkit/overview/"><img src="https://img.shields.io/badge/docs-RetargetingKit-blue" alt="Docs"></a>
</p>

RetargetingKit maps live or recorded human motion to robot hands, arms, and humanoids using [RoboKit](https://github.com/openrobokit/robokit) optimization and neural hand models.
Expand All @@ -14,8 +16,19 @@ The two main applications are:
- 🎮 **Real-time teleoperation**
- ⚡ **Offline retargeting in parallel**

<p align="center">
<a href="https://openrobokit.com/retargetingkit"><img src="https://openrobokit.com/assets/docs/retargetingkit/assets/retargetingkit-paper-teaser.png" alt="RetargetingKit paper teaser showing hand manipulation and humanoid teleoperation" width="100%"></a><br>
<em>Real-time teleoperation with dexterous hands and humanoids.</em>
</p>

<p align="center">
<a href="https://openrobokit.com/retargetingkit"><img src="https://openrobokit.com/assets/docs/retargetingkit/assets/showcase/full-body-scene.png" alt="Humanoid retargeting in a furnished scene" width="100%"></a><br>
<em>Body-and-hand retargeting in a furnished scene.</em>
</p>

## Quick links

- [Homepage](https://openrobokit.com/retargetingkit)
- [Docs](https://github.com/openrobokit/retargetingkit/blob/main/docs/guide/overview.md)
- [Quickstart](https://github.com/openrobokit/retargetingkit/blob/main/docs/guide/quickstart.md)
- [Installation](https://github.com/openrobokit/retargetingkit/blob/main/docs/guide/installation.md)
Expand All @@ -35,34 +48,46 @@ See [Installation](https://github.com/openrobokit/retargetingkit/blob/main/docs/

<table width="100%">
<tr>
<td width="50%" align="center" valign="top">
<a href="https://github.com/openrobokit/retargetingkit/blob/main/docs/guide/hand_retargeting.md#work-with-trajectories"><img src="https://raw.githubusercontent.com/openrobokit/retargetingkit/main/docs/assets/built-in-tasks/hand-retargeting.webp" alt="Tracked hand skeleton and retargeted robot hand side by side" width="300"></a><br>
<td width="33%" align="center" valign="top">
<a href="https://github.com/openrobokit/retargetingkit/blob/main/docs/guide/hand_retargeting.md#work-with-trajectories"><img src="https://openrobokit.com/assets/docs/retargetingkit/assets/built-in-tasks/hand-retargeting.webp" alt="Tracked hand skeleton and retargeted robot hand side by side" width="300"></a><br>
<strong><a href="https://github.com/openrobokit/retargetingkit/blob/main/docs/guide/hand_retargeting.md#work-with-trajectories">Hand<br>retargeting</a></strong>
</td>
<td width="50%" align="center" valign="top">
<a href="src/retargetingkit/gallery/hand_contact.py"><img src="https://raw.githubusercontent.com/openrobokit/retargetingkit/main/docs/assets/built-in-tasks/hand-geometry-sim.webp" alt="Robot hand grasping a sage cylinder in a white studio" width="300"></a><br>
<td width="33%" align="center" valign="top">
<a href="src/retargetingkit/gallery/hand_contact.py"><img src="https://openrobokit.com/assets/docs/retargetingkit/assets/built-in-tasks/hand-geometry-sim.webp" alt="Robot hand grasping a sage cylinder in a white studio" width="300"></a><br>
<strong><a href="src/retargetingkit/gallery/hand_contact.py">Geometry-aware hand<br>retargeting</a></strong>
</td>
<td width="33%" align="center" valign="top">
<a href="https://github.com/openrobokit/retargetingkit/blob/main/docs/guide/hand_retargeting.md#preview-live-motion"><img src="https://openrobokit.com/assets/docs/retargetingkit/assets/built-in-tasks/hand-teleoperation.webp" alt="Physical robot hand rotating a purple cube" width="300"></a><br>
<strong><a href="https://github.com/openrobokit/retargetingkit/blob/main/docs/guide/hand_retargeting.md#preview-live-motion">Real-time hand<br>teleoperation</a></strong>
</td>
</tr>
<tr>
<td width="50%" align="center" valign="top">
<a href="https://github.com/openrobokit/retargetingkit/blob/main/docs/guide/humanoid_retargeting.md"><img src="https://raw.githubusercontent.com/openrobokit/retargetingkit/main/docs/assets/built-in-tasks/humanoid-retargeting.webp" alt="Human body skeleton and retargeted humanoid side by side" width="300"></a><br>
<td width="33%" align="center" valign="top">
<a href="https://github.com/openrobokit/retargetingkit/blob/main/docs/guide/humanoid_retargeting.md"><img src="https://openrobokit.com/assets/docs/retargetingkit/assets/built-in-tasks/humanoid-retargeting.webp" alt="Human body skeleton and retargeted humanoid side by side" width="300"></a><br>
<strong><a href="https://github.com/openrobokit/retargetingkit/blob/main/docs/guide/humanoid_retargeting.md">Humanoid<br>retargeting</a></strong>
</td>
<td width="50%" align="center" valign="top">
<a href="src/retargetingkit/gallery/humanoid_terrain.py"><img src="https://raw.githubusercontent.com/openrobokit/retargetingkit/main/docs/assets/built-in-tasks/humanoid-geometry.webp" alt="Rendered humanoid crouching and climbing on stepped terrain" width="300"></a><br>
<td width="33%" align="center" valign="top">
<a href="src/retargetingkit/gallery/humanoid_terrain.py"><img src="https://openrobokit.com/assets/docs/retargetingkit/assets/built-in-tasks/humanoid-geometry.webp" alt="Rendered humanoid crouching and climbing on stepped terrain" width="300"></a><br>
<strong><a href="src/retargetingkit/gallery/humanoid_terrain.py">Geometry-aware humanoid<br>retargeting</a></strong>
</td>
<td width="33%" align="center" valign="top">
<a href="https://github.com/openrobokit/retargetingkit/blob/main/docs/guide/humanoid_retargeting.md"><img src="https://openrobokit.com/assets/docs/retargetingkit/assets/built-in-tasks/humanoid-teleoperation.webp" alt="Physical humanoid crouching to pick up a green bottle" width="300"></a><br>
<strong><a href="https://github.com/openrobokit/retargetingkit/blob/main/docs/guide/humanoid_retargeting.md">Real-time humanoid<br>teleoperation</a></strong>
</td>
</tr>
<tr>
<td width="50%" align="center" valign="top">
<a href="https://github.com/openrobokit/retargetingkit/blob/main/docs/guide/full_body_retargeting.md"><img src="https://raw.githubusercontent.com/openrobokit/retargetingkit/main/docs/assets/built-in-tasks/full-body-retargeting.webp" alt="Standing humanoid turning and taking small steps on a white background" width="300"></a><br>
<td width="33%" align="center" valign="top">
<a href="https://github.com/openrobokit/retargetingkit/blob/main/docs/guide/full_body_retargeting.md"><img src="https://openrobokit.com/assets/docs/retargetingkit/assets/built-in-tasks/full-body-retargeting.webp" alt="Standing humanoid turning and taking small steps on a white background" width="300"></a><br>
<strong><a href="https://github.com/openrobokit/retargetingkit/blob/main/docs/guide/full_body_retargeting.md">Full-body<br>retargeting</a></strong>
</td>
<td width="50%" align="center" valign="top">
<a href="https://github.com/openrobokit/retargetingkit/blob/main/docs/guide/full_body_retargeting.md#retarget-an-indoor-interaction"><img src="https://raw.githubusercontent.com/openrobokit/retargetingkit/main/docs/assets/built-in-tasks/full-body-geometry.webp" alt="Rendered humanoid interacting with objects and furniture in a kitchen on a white background" width="300"></a><br>
<td width="33%" align="center" valign="top">
<a href="https://github.com/openrobokit/retargetingkit/blob/main/docs/guide/full_body_retargeting.md#retarget-an-indoor-interaction"><img src="https://openrobokit.com/assets/docs/retargetingkit/assets/built-in-tasks/full-body-geometry.webp" alt="Rendered humanoid interacting with objects and furniture in a kitchen on a white background" width="300"></a><br>
<strong><a href="https://github.com/openrobokit/retargetingkit/blob/main/docs/guide/full_body_retargeting.md#retarget-an-indoor-interaction">Geometry-aware full-body<br>retargeting</a></strong>
</td>
<td width="33%" align="center" valign="top">
<a href="https://github.com/openrobokit/retargetingkit/blob/main/docs/guide/humanoid_retargeting.md#control-both-inspire-hands"><img src="https://openrobokit.com/assets/docs/retargetingkit/assets/built-in-tasks/full-body-teleoperation.webp" alt="Physical humanoid handling flowers at a table" width="300"></a><br>
<strong><a href="https://github.com/openrobokit/retargetingkit/blob/main/docs/guide/humanoid_retargeting.md#control-both-inspire-hands">Real-time full-body<br>teleoperation</a></strong>
</td>
</tr>
</table>

Expand Down
Binary file removed docs/assets/built-in-tasks/full-body-geometry.webp
Binary file not shown.
Binary file not shown.
Binary file removed docs/assets/built-in-tasks/hand-geometry-sim.webp
Binary file not shown.
Binary file removed docs/assets/built-in-tasks/hand-retargeting.webp
Binary file not shown.
Binary file removed docs/assets/built-in-tasks/humanoid-geometry.webp
Binary file not shown.
Binary file not shown.
Binary file removed docs/assets/retargetingkit-banner.webp
Binary file not shown.
Binary file removed docs/assets/sources/cloudxr-web-client-howto.png
Binary file not shown.
Binary file not shown.
Binary file removed docs/assets/sources/disable_proximity_sensor.png
Binary file not shown.
Binary file removed docs/assets/sources/sidequest_tool.png
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
2 changes: 2 additions & 0 deletions docs/guide/arm_retargeting.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@
# Arm Retargeting

![An xArm7 carrying an XHand in the interactive retargeting gallery](https://openrobokit.com/assets/docs/retargetingkit/assets/gallery/hand_retargeting.webp)

Map a tracked wrist to an xArm6 or xArm7 end effector. The arm solver uses RoboKit inverse kinematics with position, orientation, joint-limit, and smoothness terms.

## How Retargeting Works
Expand Down
3 changes: 2 additions & 1 deletion docs/guide/full_body_retargeting.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,8 @@
# Full-Body Retargeting

| Body, hand, and object interaction. | Body and hand motion in a furnished scene. |
| ![SAGA body and hand interaction retargeted in Viser](https://openrobokit.com/assets/docs/retargetingkit/assets/gallery/full_body_hoi.webp) | ![ParaHome body and scene interaction retargeted in Viser](https://openrobokit.com/assets/docs/retargetingkit/assets/gallery/full_body_hsi.webp) |
| --- | --- |
| Body, hand, and object interaction. | Body and hand motion in a furnished scene. |

Retarget body motion, articulated fingers, and object interaction together to a G1 with Inspire hands. RetargetingKit loads SAGA or ParaHome clips; RoboKit solves the robot motion and object poses.

Expand Down
3 changes: 2 additions & 1 deletion docs/guide/hand_retargeting.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,8 @@
# Hand Retargeting

| Hand motion on a robot assembly. | Hand-object contact refinement. |
| ![Tracked hand retargeted to an xArm7 and XHand](https://openrobokit.com/assets/docs/retargetingkit/assets/gallery/hand_retargeting.webp) | ![Hand-object contact refinement in Viser](https://openrobokit.com/assets/docs/retargetingkit/assets/gallery/hand_contact.webp) |
| --- | --- |
| Hand motion on a robot assembly. | Hand-object contact refinement. |

Map human hand motion to robot joint positions with RoboKit's online or offline optimization. Start with a recorded example, then connect a [Manus glove](../reference/sources/manus.md) or [Apple Vision Pro](../reference/sources/vision_pro.md).

Expand Down
3 changes: 2 additions & 1 deletion docs/guide/humanoid_retargeting.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,8 @@
# Humanoid Retargeting

| Human motion mapped to a humanoid. | Humanoid motion on terrain. |
| ![Human motion retargeted to a G1 in Viser](https://openrobokit.com/assets/docs/retargetingkit/assets/gallery/humanoid.webp) | ![Humanoid adapting motion to terrain in Viser](https://openrobokit.com/assets/docs/retargetingkit/assets/gallery/humanoid_terrain.webp) |
| --- | --- |
| Human motion mapped to a humanoid. | Humanoid motion on terrain. |

Map recorded or live body motion to G1 at 50 Hz. The runtime sends robot root pose and joint positions to its configured targets; the SONIC target derives joint velocities for whole-body control.

Expand Down
2 changes: 2 additions & 0 deletions docs/guide/neural_hand_retargeting.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@
# Neural Hand Retargeting

![Neural hand inference in the interactive gallery](https://openrobokit.com/assets/docs/retargetingkit/assets/gallery/hand_neural.webp)

Train a model for one robot hand, then map wrist-local MANO-21 keypoints to joint positions in URDF order. Curated checkpoints support Allegro, Sharpa Wave, and XHand.

## How Inference Works
Expand Down
22 changes: 12 additions & 10 deletions docs/guide/overview.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,6 @@
![RetargetingKit](../assets/retargetingkit-banner.webp)
![RetargetingKit](https://openrobokit.com/assets/docs/retargetingkit/assets/retargetingkit-banner.webp)

[![Dexterous hand and humanoid motion](https://openrobokit.com/assets/docs/retargetingkit/assets/retargetingkit-paper-teaser.png)](https://openrobokit.com/retargetingkit)

RetargetingKit maps human motion to robot hands, arms, and humanoids. It combines RoboKit optimization with neural hand models for recorded motion and real-time teleoperation.

Expand Down Expand Up @@ -37,17 +39,17 @@ flowchart LR

## Features

| [![Hand retargeting](../assets/built-in-tasks/hand-retargeting.webp)](hand_retargeting.md) | [![Geometry-aware hand retargeting](../assets/built-in-tasks/hand-geometry-sim.webp)](hand_retargeting.md#inspect-object-contact) |
| --- | --- |
| **[Hand retargeting](hand_retargeting.md)** Map tracked hand motion to robot joints. | **[Geometry-aware hand retargeting](hand_retargeting.md#inspect-object-contact)** Refine hands against object geometry. |
| [![Hand retargeting](https://openrobokit.com/assets/docs/retargetingkit/assets/built-in-tasks/hand-retargeting.webp)](hand_retargeting.md) | [![Geometry-aware hand retargeting](https://openrobokit.com/assets/docs/retargetingkit/assets/built-in-tasks/hand-geometry-sim.webp)](hand_retargeting.md#inspect-object-contact) | [![Real-time hand teleoperation](https://openrobokit.com/assets/docs/retargetingkit/assets/built-in-tasks/hand-geometry.webp)](hand_retargeting.md#preview-live-motion) |
| --- | --- | --- |
| **[Hand retargeting](hand_retargeting.md)** Map tracked hand motion to robot joints. | **[Geometry-aware hand retargeting](hand_retargeting.md#inspect-object-contact)** Refine hands against object geometry. | **[Real-time hand teleoperation](hand_retargeting.md#preview-live-motion)** Drive a robot hand from live tracking. |

| [![Humanoid retargeting](../assets/built-in-tasks/humanoid-retargeting.webp)](humanoid_retargeting.md) | [![Geometry-aware humanoid retargeting](../assets/built-in-tasks/humanoid-geometry.webp)](humanoid_retargeting.md) |
| --- | --- |
| **[Humanoid retargeting](humanoid_retargeting.md)** Map captured body motion to humanoids. | **[Geometry-aware humanoid retargeting](humanoid_retargeting.md)** Account for terrain and contact. |
| [![Humanoid retargeting](https://openrobokit.com/assets/docs/retargetingkit/assets/built-in-tasks/humanoid-retargeting.webp)](humanoid_retargeting.md) | [![Geometry-aware humanoid retargeting](https://openrobokit.com/assets/docs/retargetingkit/assets/built-in-tasks/humanoid-geometry.webp)](humanoid_retargeting.md) | [![Real-time humanoid teleoperation](https://openrobokit.com/assets/docs/retargetingkit/assets/built-in-tasks/humanoid-teleoperation.webp)](humanoid_retargeting.md) |
| --- | --- | --- |
| **[Humanoid retargeting](humanoid_retargeting.md)** Map captured body motion to humanoids. | **[Geometry-aware humanoid retargeting](humanoid_retargeting.md)** Account for terrain and contact. | **[Real-time humanoid teleoperation](humanoid_retargeting.md)** Drive a humanoid from live motion. |

| [![Full-body retargeting](../assets/built-in-tasks/full-body-retargeting.webp)](full_body_retargeting.md) | [![Geometry-aware full-body retargeting](../assets/built-in-tasks/full-body-geometry.webp)](full_body_retargeting.md#retarget-an-indoor-interaction) |
| --- | --- |
| **[Full-body retargeting](full_body_retargeting.md)** Combine body and hand motion. | **[Geometry-aware full-body retargeting](full_body_retargeting.md#retarget-an-indoor-interaction)** Preserve scene and object interactions. |
| [![Full-body retargeting](https://openrobokit.com/assets/docs/retargetingkit/assets/built-in-tasks/full-body-retargeting.webp)](full_body_retargeting.md) | [![Geometry-aware full-body retargeting](https://openrobokit.com/assets/docs/retargetingkit/assets/built-in-tasks/full-body-geometry.webp)](full_body_retargeting.md#retarget-an-indoor-interaction) | [![Real-time full-body teleoperation](https://openrobokit.com/assets/docs/retargetingkit/assets/built-in-tasks/full-body-teleoperation.webp)](humanoid_retargeting.md#control-both-inspire-hands) |
| --- | --- | --- |
| **[Full-body retargeting](full_body_retargeting.md)** Combine body and hand motion. | **[Geometry-aware full-body retargeting](full_body_retargeting.md#retarget-an-indoor-interaction)** Preserve scene and object interactions. | **[Real-time full-body teleoperation](humanoid_retargeting.md#control-both-inspire-hands)** Drive body and hands together. |

## Quick links

Expand Down
12 changes: 8 additions & 4 deletions docs/guide/quickstart.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,17 +11,21 @@ uvx --from 'retargetingkit[vis]' \

Open the Viser URL printed in the terminal. Select a robot and motion, then play or scrub the result. Replace `hand_retargeting` with any gallery below. Each preview records that gallery's Viser scene.

| **`hand_retargeting`** Retarget tracked hands. | **`hand_neural`** Compare neural hand models. | **`hand_contact`** Refine hand-object contact. |
| ![Hand retargeting in Viser](https://openrobokit.com/assets/docs/retargetingkit/assets/gallery/hand_retargeting.webp) | ![Neural hand retargeting in Viser](https://openrobokit.com/assets/docs/retargetingkit/assets/gallery/hand_neural.webp) | ![Hand contact retargeting in Viser](https://openrobokit.com/assets/docs/retargetingkit/assets/gallery/hand_contact.webp) |
| --- | --- | --- |
| **`hand_retargeting`** Retarget tracked hands. | **`hand_neural`** Compare neural hand models. | **`hand_contact`** Refine hand-object contact. |

| **`hand_object_dexmachina`** Explore DexMachina interactions. | **`hand_object_gigahands`** Explore GigaHands interactions. | **`hand_object_oakink2`** Explore OakInk2 interactions. |
| ![DexMachina hand-object retargeting in Viser](https://openrobokit.com/assets/docs/retargetingkit/assets/gallery/hand_object_dexmachina.webp) | ![GigaHands hand-object retargeting in Viser](https://openrobokit.com/assets/docs/retargetingkit/assets/gallery/hand_object_gigahands.webp) | ![OakInk2 hand-object retargeting in Viser](https://openrobokit.com/assets/docs/retargetingkit/assets/gallery/hand_object_oakink2.webp) |
| --- | --- | --- |
| **`hand_object_dexmachina`** Explore DexMachina interactions. | **`hand_object_gigahands`** Explore GigaHands interactions. | **`hand_object_oakink2`** Explore OakInk2 interactions. |

| **`humanoid`** Browse motions and robot presets. | **`humanoid_contact`** Retarget object interactions. | **`humanoid_terrain`** Adapt motion to terrain. |
| ![Humanoid retargeting in Viser](https://openrobokit.com/assets/docs/retargetingkit/assets/gallery/humanoid.webp) | ![Humanoid contact retargeting in Viser](https://openrobokit.com/assets/docs/retargetingkit/assets/gallery/humanoid_contact.webp) | ![Humanoid terrain retargeting in Viser](https://openrobokit.com/assets/docs/retargetingkit/assets/gallery/humanoid_terrain.webp) |
| --- | --- | --- |
| **`humanoid`** Browse motions and robot presets. | **`humanoid_contact`** Retarget object interactions. | **`humanoid_terrain`** Adapt motion to terrain. |

| **`full_body_hoi`** Retarget body and hand interactions. | **`full_body_hsi`** Retarget body motion in furnished scenes. |
| ![Full-body object interaction in Viser](https://openrobokit.com/assets/docs/retargetingkit/assets/gallery/full_body_hoi.webp) | ![Full-body scene interaction in Viser](https://openrobokit.com/assets/docs/retargetingkit/assets/gallery/full_body_hsi.webp) |
| --- | --- |
| **`full_body_hoi`** Retarget body and hand interactions. | **`full_body_hsi`** Retarget body motion in furnished scenes. |

To list gallery module names from an installed checkout, run `uv run python -m retargetingkit.gallery`. Gallery data, robot models, and checkpoints download on first use; complete [asset setup](assets.md) first.

Expand Down
Loading
Loading