diff --git a/README.md b/README.md index 63692ee..72892f0 100644 --- a/README.md +++ b/README.md @@ -1,10 +1,12 @@

- RetargetingKit + RetargetingKit

Python 3.10+ GitHub Actions + Homepage + Docs

RetargetingKit maps live or recorded human motion to robot hands, arms, and humanoids using [RoboKit](https://github.com/openrobokit/robokit) optimization and neural hand models. @@ -14,8 +16,19 @@ The two main applications are: - 🎮 **Real-time teleoperation** - ⚡ **Offline retargeting in parallel** +

+ RetargetingKit paper teaser showing hand manipulation and humanoid teleoperation
+ Real-time teleoperation with dexterous hands and humanoids. +

+ +

+ Humanoid retargeting in a furnished scene
+ Body-and-hand retargeting in a furnished scene. +

+ ## Quick links +- [Homepage](https://openrobokit.com/retargetingkit) - [Docs](https://github.com/openrobokit/retargetingkit/blob/main/docs/guide/overview.md) - [Quickstart](https://github.com/openrobokit/retargetingkit/blob/main/docs/guide/quickstart.md) - [Installation](https://github.com/openrobokit/retargetingkit/blob/main/docs/guide/installation.md) @@ -35,34 +48,46 @@ See [Installation](https://github.com/openrobokit/retargetingkit/blob/main/docs/ - - + - - + - - +
- Tracked hand skeleton and retargeted robot hand side by side
+
+ Tracked hand skeleton and retargeted robot hand side by side
Hand
retargeting
- Robot hand grasping a sage cylinder in a white studio
+
+ Robot hand grasping a sage cylinder in a white studio
Geometry-aware hand
retargeting
+ Physical robot hand rotating a purple cube
+ Real-time hand
teleoperation
+
- Human body skeleton and retargeted humanoid side by side
+
+ Human body skeleton and retargeted humanoid side by side
Humanoid
retargeting
- Rendered humanoid crouching and climbing on stepped terrain
+
+ Rendered humanoid crouching and climbing on stepped terrain
Geometry-aware humanoid
retargeting
+ Physical humanoid crouching to pick up a green bottle
+ Real-time humanoid
teleoperation
+
- Standing humanoid turning and taking small steps on a white background
+
+ Standing humanoid turning and taking small steps on a white background
Full-body
retargeting
- Rendered humanoid interacting with objects and furniture in a kitchen on a white background
+
+ Rendered humanoid interacting with objects and furniture in a kitchen on a white background
Geometry-aware full-body
retargeting
+ Physical humanoid handling flowers at a table
+ Real-time full-body
teleoperation
+
diff --git a/docs/assets/built-in-tasks/full-body-geometry.webp b/docs/assets/built-in-tasks/full-body-geometry.webp deleted file mode 100644 index 495c026..0000000 Binary files a/docs/assets/built-in-tasks/full-body-geometry.webp and /dev/null differ diff --git a/docs/assets/built-in-tasks/full-body-retargeting.webp b/docs/assets/built-in-tasks/full-body-retargeting.webp deleted file mode 100644 index 8b7500b..0000000 Binary files a/docs/assets/built-in-tasks/full-body-retargeting.webp and /dev/null differ diff --git a/docs/assets/built-in-tasks/hand-geometry-sim.webp b/docs/assets/built-in-tasks/hand-geometry-sim.webp deleted file mode 100644 index 6c2ea64..0000000 Binary files a/docs/assets/built-in-tasks/hand-geometry-sim.webp and /dev/null differ diff --git a/docs/assets/built-in-tasks/hand-retargeting.webp b/docs/assets/built-in-tasks/hand-retargeting.webp deleted file mode 100644 index 6c1ba29..0000000 Binary files a/docs/assets/built-in-tasks/hand-retargeting.webp and /dev/null differ diff --git a/docs/assets/built-in-tasks/humanoid-geometry.webp b/docs/assets/built-in-tasks/humanoid-geometry.webp deleted file mode 100644 index a793033..0000000 Binary files a/docs/assets/built-in-tasks/humanoid-geometry.webp and /dev/null differ diff --git a/docs/assets/built-in-tasks/humanoid-retargeting.webp b/docs/assets/built-in-tasks/humanoid-retargeting.webp deleted file mode 100644 index 337867d..0000000 Binary files a/docs/assets/built-in-tasks/humanoid-retargeting.webp and /dev/null differ diff --git a/docs/assets/retargetingkit-banner.webp b/docs/assets/retargetingkit-banner.webp deleted file mode 100644 index 57a5a13..0000000 Binary files a/docs/assets/retargetingkit-banner.webp and /dev/null differ diff --git a/docs/assets/sources/cloudxr-web-client-howto.png b/docs/assets/sources/cloudxr-web-client-howto.png deleted file mode 100644 index 232c08e..0000000 Binary files a/docs/assets/sources/cloudxr-web-client-howto.png and /dev/null differ diff --git a/docs/assets/sources/cloudxr_accept_cert_accepted.png b/docs/assets/sources/cloudxr_accept_cert_accepted.png deleted file mode 100644 index c93954f..0000000 Binary files a/docs/assets/sources/cloudxr_accept_cert_accepted.png and /dev/null differ diff --git a/docs/assets/sources/disable_proximity_sensor.png b/docs/assets/sources/disable_proximity_sensor.png deleted file mode 100644 index 6d4eb0c..0000000 Binary files a/docs/assets/sources/disable_proximity_sensor.png and /dev/null differ diff --git a/docs/assets/sources/sidequest_tool.png b/docs/assets/sources/sidequest_tool.png deleted file mode 100644 index d77c502..0000000 Binary files a/docs/assets/sources/sidequest_tool.png and /dev/null differ diff --git a/docs/assets/sources/vision-pro-tracking-streamer-app.jpg b/docs/assets/sources/vision-pro-tracking-streamer-app.jpg deleted file mode 100644 index 9acebdb..0000000 Binary files a/docs/assets/sources/vision-pro-tracking-streamer-app.jpg and /dev/null differ diff --git a/docs/assets/sources/vision-pro-tracking-streamer-start.jpg b/docs/assets/sources/vision-pro-tracking-streamer-start.jpg deleted file mode 100644 index 2568921..0000000 Binary files a/docs/assets/sources/vision-pro-tracking-streamer-start.jpg and /dev/null differ diff --git a/docs/assets/sources/xrobotoolkit-pico-fullbody-settings.jpeg b/docs/assets/sources/xrobotoolkit-pico-fullbody-settings.jpeg deleted file mode 100644 index 380bd2f..0000000 Binary files a/docs/assets/sources/xrobotoolkit-pico-fullbody-settings.jpeg and /dev/null differ diff --git a/docs/guide/arm_retargeting.md b/docs/guide/arm_retargeting.md index 4be24a8..9c75e5f 100644 --- a/docs/guide/arm_retargeting.md +++ b/docs/guide/arm_retargeting.md @@ -1,5 +1,7 @@ # Arm Retargeting +![An xArm7 carrying an XHand in the interactive retargeting gallery](https://openrobokit.com/assets/docs/retargetingkit/assets/gallery/hand_retargeting.webp) + Map a tracked wrist to an xArm6 or xArm7 end effector. The arm solver uses RoboKit inverse kinematics with position, orientation, joint-limit, and smoothness terms. ## How Retargeting Works diff --git a/docs/guide/full_body_retargeting.md b/docs/guide/full_body_retargeting.md index 4549eb0..b056a83 100644 --- a/docs/guide/full_body_retargeting.md +++ b/docs/guide/full_body_retargeting.md @@ -1,7 +1,8 @@ # Full-Body Retargeting -| Body, hand, and object interaction. | Body and hand motion in a furnished scene. | +| ![SAGA body and hand interaction retargeted in Viser](https://openrobokit.com/assets/docs/retargetingkit/assets/gallery/full_body_hoi.webp) | ![ParaHome body and scene interaction retargeted in Viser](https://openrobokit.com/assets/docs/retargetingkit/assets/gallery/full_body_hsi.webp) | | --- | --- | +| Body, hand, and object interaction. | Body and hand motion in a furnished scene. | Retarget body motion, articulated fingers, and object interaction together to a G1 with Inspire hands. RetargetingKit loads SAGA or ParaHome clips; RoboKit solves the robot motion and object poses. diff --git a/docs/guide/hand_retargeting.md b/docs/guide/hand_retargeting.md index d4aeec1..b6156b4 100644 --- a/docs/guide/hand_retargeting.md +++ b/docs/guide/hand_retargeting.md @@ -1,7 +1,8 @@ # Hand Retargeting -| Hand motion on a robot assembly. | Hand-object contact refinement. | +| ![Tracked hand retargeted to an xArm7 and XHand](https://openrobokit.com/assets/docs/retargetingkit/assets/gallery/hand_retargeting.webp) | ![Hand-object contact refinement in Viser](https://openrobokit.com/assets/docs/retargetingkit/assets/gallery/hand_contact.webp) | | --- | --- | +| Hand motion on a robot assembly. | Hand-object contact refinement. | Map human hand motion to robot joint positions with RoboKit's online or offline optimization. Start with a recorded example, then connect a [Manus glove](../reference/sources/manus.md) or [Apple Vision Pro](../reference/sources/vision_pro.md). diff --git a/docs/guide/humanoid_retargeting.md b/docs/guide/humanoid_retargeting.md index f0c9ca6..1b57d40 100644 --- a/docs/guide/humanoid_retargeting.md +++ b/docs/guide/humanoid_retargeting.md @@ -1,7 +1,8 @@ # Humanoid Retargeting -| Human motion mapped to a humanoid. | Humanoid motion on terrain. | +| ![Human motion retargeted to a G1 in Viser](https://openrobokit.com/assets/docs/retargetingkit/assets/gallery/humanoid.webp) | ![Humanoid adapting motion to terrain in Viser](https://openrobokit.com/assets/docs/retargetingkit/assets/gallery/humanoid_terrain.webp) | | --- | --- | +| Human motion mapped to a humanoid. | Humanoid motion on terrain. | Map recorded or live body motion to G1 at 50 Hz. The runtime sends robot root pose and joint positions to its configured targets; the SONIC target derives joint velocities for whole-body control. diff --git a/docs/guide/neural_hand_retargeting.md b/docs/guide/neural_hand_retargeting.md index 9b7d734..c9c69c8 100644 --- a/docs/guide/neural_hand_retargeting.md +++ b/docs/guide/neural_hand_retargeting.md @@ -1,5 +1,7 @@ # Neural Hand Retargeting +![Neural hand inference in the interactive gallery](https://openrobokit.com/assets/docs/retargetingkit/assets/gallery/hand_neural.webp) + Train a model for one robot hand, then map wrist-local MANO-21 keypoints to joint positions in URDF order. Curated checkpoints support Allegro, Sharpa Wave, and XHand. ## How Inference Works diff --git a/docs/guide/overview.md b/docs/guide/overview.md index 7d39782..4ffdfc3 100644 --- a/docs/guide/overview.md +++ b/docs/guide/overview.md @@ -1,4 +1,6 @@ -![RetargetingKit](../assets/retargetingkit-banner.webp) +![RetargetingKit](https://openrobokit.com/assets/docs/retargetingkit/assets/retargetingkit-banner.webp) + +[![Dexterous hand and humanoid motion](https://openrobokit.com/assets/docs/retargetingkit/assets/retargetingkit-paper-teaser.png)](https://openrobokit.com/retargetingkit) RetargetingKit maps human motion to robot hands, arms, and humanoids. It combines RoboKit optimization with neural hand models for recorded motion and real-time teleoperation. @@ -37,17 +39,17 @@ flowchart LR ## Features -| [![Hand retargeting](../assets/built-in-tasks/hand-retargeting.webp)](hand_retargeting.md) | [![Geometry-aware hand retargeting](../assets/built-in-tasks/hand-geometry-sim.webp)](hand_retargeting.md#inspect-object-contact) | -| --- | --- | -| **[Hand retargeting](hand_retargeting.md)** Map tracked hand motion to robot joints. | **[Geometry-aware hand retargeting](hand_retargeting.md#inspect-object-contact)** Refine hands against object geometry. | +| [![Hand retargeting](https://openrobokit.com/assets/docs/retargetingkit/assets/built-in-tasks/hand-retargeting.webp)](hand_retargeting.md) | [![Geometry-aware hand retargeting](https://openrobokit.com/assets/docs/retargetingkit/assets/built-in-tasks/hand-geometry-sim.webp)](hand_retargeting.md#inspect-object-contact) | [![Real-time hand teleoperation](https://openrobokit.com/assets/docs/retargetingkit/assets/built-in-tasks/hand-geometry.webp)](hand_retargeting.md#preview-live-motion) | +| --- | --- | --- | +| **[Hand retargeting](hand_retargeting.md)** Map tracked hand motion to robot joints. | **[Geometry-aware hand retargeting](hand_retargeting.md#inspect-object-contact)** Refine hands against object geometry. | **[Real-time hand teleoperation](hand_retargeting.md#preview-live-motion)** Drive a robot hand from live tracking. | -| [![Humanoid retargeting](../assets/built-in-tasks/humanoid-retargeting.webp)](humanoid_retargeting.md) | [![Geometry-aware humanoid retargeting](../assets/built-in-tasks/humanoid-geometry.webp)](humanoid_retargeting.md) | -| --- | --- | -| **[Humanoid retargeting](humanoid_retargeting.md)** Map captured body motion to humanoids. | **[Geometry-aware humanoid retargeting](humanoid_retargeting.md)** Account for terrain and contact. | +| [![Humanoid retargeting](https://openrobokit.com/assets/docs/retargetingkit/assets/built-in-tasks/humanoid-retargeting.webp)](humanoid_retargeting.md) | [![Geometry-aware humanoid retargeting](https://openrobokit.com/assets/docs/retargetingkit/assets/built-in-tasks/humanoid-geometry.webp)](humanoid_retargeting.md) | [![Real-time humanoid teleoperation](https://openrobokit.com/assets/docs/retargetingkit/assets/built-in-tasks/humanoid-teleoperation.webp)](humanoid_retargeting.md) | +| --- | --- | --- | +| **[Humanoid retargeting](humanoid_retargeting.md)** Map captured body motion to humanoids. | **[Geometry-aware humanoid retargeting](humanoid_retargeting.md)** Account for terrain and contact. | **[Real-time humanoid teleoperation](humanoid_retargeting.md)** Drive a humanoid from live motion. | -| [![Full-body retargeting](../assets/built-in-tasks/full-body-retargeting.webp)](full_body_retargeting.md) | [![Geometry-aware full-body retargeting](../assets/built-in-tasks/full-body-geometry.webp)](full_body_retargeting.md#retarget-an-indoor-interaction) | -| --- | --- | -| **[Full-body retargeting](full_body_retargeting.md)** Combine body and hand motion. | **[Geometry-aware full-body retargeting](full_body_retargeting.md#retarget-an-indoor-interaction)** Preserve scene and object interactions. | +| [![Full-body retargeting](https://openrobokit.com/assets/docs/retargetingkit/assets/built-in-tasks/full-body-retargeting.webp)](full_body_retargeting.md) | [![Geometry-aware full-body retargeting](https://openrobokit.com/assets/docs/retargetingkit/assets/built-in-tasks/full-body-geometry.webp)](full_body_retargeting.md#retarget-an-indoor-interaction) | [![Real-time full-body teleoperation](https://openrobokit.com/assets/docs/retargetingkit/assets/built-in-tasks/full-body-teleoperation.webp)](humanoid_retargeting.md#control-both-inspire-hands) | +| --- | --- | --- | +| **[Full-body retargeting](full_body_retargeting.md)** Combine body and hand motion. | **[Geometry-aware full-body retargeting](full_body_retargeting.md#retarget-an-indoor-interaction)** Preserve scene and object interactions. | **[Real-time full-body teleoperation](humanoid_retargeting.md#control-both-inspire-hands)** Drive body and hands together. | ## Quick links diff --git a/docs/guide/quickstart.md b/docs/guide/quickstart.md index 29930af..2816901 100644 --- a/docs/guide/quickstart.md +++ b/docs/guide/quickstart.md @@ -11,17 +11,21 @@ uvx --from 'retargetingkit[vis]' \ Open the Viser URL printed in the terminal. Select a robot and motion, then play or scrub the result. Replace `hand_retargeting` with any gallery below. Each preview records that gallery's Viser scene. -| **`hand_retargeting`** Retarget tracked hands. | **`hand_neural`** Compare neural hand models. | **`hand_contact`** Refine hand-object contact. | +| ![Hand retargeting in Viser](https://openrobokit.com/assets/docs/retargetingkit/assets/gallery/hand_retargeting.webp) | ![Neural hand retargeting in Viser](https://openrobokit.com/assets/docs/retargetingkit/assets/gallery/hand_neural.webp) | ![Hand contact retargeting in Viser](https://openrobokit.com/assets/docs/retargetingkit/assets/gallery/hand_contact.webp) | | --- | --- | --- | +| **`hand_retargeting`** Retarget tracked hands. | **`hand_neural`** Compare neural hand models. | **`hand_contact`** Refine hand-object contact. | -| **`hand_object_dexmachina`** Explore DexMachina interactions. | **`hand_object_gigahands`** Explore GigaHands interactions. | **`hand_object_oakink2`** Explore OakInk2 interactions. | +| ![DexMachina hand-object retargeting in Viser](https://openrobokit.com/assets/docs/retargetingkit/assets/gallery/hand_object_dexmachina.webp) | ![GigaHands hand-object retargeting in Viser](https://openrobokit.com/assets/docs/retargetingkit/assets/gallery/hand_object_gigahands.webp) | ![OakInk2 hand-object retargeting in Viser](https://openrobokit.com/assets/docs/retargetingkit/assets/gallery/hand_object_oakink2.webp) | | --- | --- | --- | +| **`hand_object_dexmachina`** Explore DexMachina interactions. | **`hand_object_gigahands`** Explore GigaHands interactions. | **`hand_object_oakink2`** Explore OakInk2 interactions. | -| **`humanoid`** Browse motions and robot presets. | **`humanoid_contact`** Retarget object interactions. | **`humanoid_terrain`** Adapt motion to terrain. | +| ![Humanoid retargeting in Viser](https://openrobokit.com/assets/docs/retargetingkit/assets/gallery/humanoid.webp) | ![Humanoid contact retargeting in Viser](https://openrobokit.com/assets/docs/retargetingkit/assets/gallery/humanoid_contact.webp) | ![Humanoid terrain retargeting in Viser](https://openrobokit.com/assets/docs/retargetingkit/assets/gallery/humanoid_terrain.webp) | | --- | --- | --- | +| **`humanoid`** Browse motions and robot presets. | **`humanoid_contact`** Retarget object interactions. | **`humanoid_terrain`** Adapt motion to terrain. | -| **`full_body_hoi`** Retarget body and hand interactions. | **`full_body_hsi`** Retarget body motion in furnished scenes. | +| ![Full-body object interaction in Viser](https://openrobokit.com/assets/docs/retargetingkit/assets/gallery/full_body_hoi.webp) | ![Full-body scene interaction in Viser](https://openrobokit.com/assets/docs/retargetingkit/assets/gallery/full_body_hsi.webp) | | --- | --- | +| **`full_body_hoi`** Retarget body and hand interactions. | **`full_body_hsi`** Retarget body motion in furnished scenes. | To list gallery module names from an installed checkout, run `uv run python -m retargetingkit.gallery`. Gallery data, robot models, and checkpoints download on first use; complete [asset setup](assets.md) first. diff --git a/docs/guide/teleoperation.md b/docs/guide/teleoperation.md index 5f0abeb..f2e8037 100644 --- a/docs/guide/teleoperation.md +++ b/docs/guide/teleoperation.md @@ -1,5 +1,9 @@ # Teleoperation +| ![Real-time robot hand teleoperation](https://openrobokit.com/assets/docs/retargetingkit/assets/built-in-tasks/hand-geometry.webp) | ![Real-time humanoid teleoperation](https://openrobokit.com/assets/docs/retargetingkit/assets/built-in-tasks/humanoid-teleoperation.webp) | +| --- | --- | +| Dexterous hand control. | Humanoid motion control. | + A named rig connects a source, retargeter, and target in one command. Choose the flow that matches the motion you want to control, then complete the linked device setup. ## Choose a flow diff --git a/docs/reference/sources/pico.md b/docs/reference/sources/pico.md index 597c1e7..cbd5c41 100644 --- a/docs/reference/sources/pico.md +++ b/docs/reference/sources/pico.md @@ -1,6 +1,6 @@ # PICO -![XRoboToolkit full-body settings. Use your workstation IP; the screenshot shows an example.](../../assets/sources/xrobotoolkit-pico-fullbody-settings.jpeg) +![XRoboToolkit full-body settings. Use your workstation IP; the screenshot shows an example.](https://openrobokit.com/assets/docs/retargetingkit/assets/sources/xrobotoolkit-pico-fullbody-settings.jpeg) Use PICO full-body tracking through XRoboToolkit to retarget a 24-joint body to Unitree G1. Record sessions for offline replay, or use controller buttons to open and close two Inspire hands alongside body motion. diff --git a/docs/reference/sources/quest3.md b/docs/reference/sources/quest3.md index 1fab2f1..6c8960e 100644 --- a/docs/reference/sources/quest3.md +++ b/docs/reference/sources/quest3.md @@ -1,6 +1,6 @@ # Quest 3 -![CloudXR Web Client: enter the workstation IP, accept its certificate, then connect.](../../assets/sources/cloudxr-web-client-howto.png) +![CloudXR Web Client: enter the workstation IP, accept its certificate, then connect.](https://openrobokit.com/assets/docs/retargetingkit/assets/sources/cloudxr-web-client-howto.png) Use Quest 3 controllers to move one or two xArm6 or xArm7 end effectors. Tracking comes through NVIDIA Isaac Teleop and CloudXR. The standalone playground supports a Viser preview, controller calibration, and hardware control. @@ -30,7 +30,7 @@ sudo ufw allow 49100,48322/tcp Open the [Isaac Teleop WebXR client](https://nvidia.github.io/IsaacTeleop/client) in the headset browser. Enter the workstation IP, accept its generated certificate, and select **Connect**, then **Play**. -![CloudXR confirms that the certificate was accepted.](../../assets/sources/cloudxr_accept_cert_accepted.png) +![CloudXR confirms that the certificate was accepted.](https://openrobokit.com/assets/docs/retargetingkit/assets/sources/cloudxr_accept_cert_accepted.png) ## Check tracking @@ -77,8 +77,8 @@ Use `--arm xarm6` to select xArm6. Each arm moves to its home pose during startu ## Keep the headset awake -![Open Device Settings & Tools in SideQuest.](../../assets/sources/sidequest_tool.png) +![Open Device Settings & Tools in SideQuest.](https://openrobokit.com/assets/docs/retargetingkit/assets/sources/sidequest_tool.png) -![Disable the proximity sensor for an uninterrupted session.](../../assets/sources/disable_proximity_sensor.png) +![Disable the proximity sensor for an uninterrupted session.](https://openrobokit.com/assets/docs/retargetingkit/assets/sources/disable_proximity_sensor.png) For long sessions, enable developer mode, connect SideQuest over USB, and disable the proximity sensor in **Device Settings & Tools**. diff --git a/docs/reference/sources/vision_pro.md b/docs/reference/sources/vision_pro.md index 8bd4d11..c661230 100644 --- a/docs/reference/sources/vision_pro.md +++ b/docs/reference/sources/vision_pro.md @@ -1,6 +1,6 @@ # Apple Vision Pro -![Tracking Streamer: note the headset IP and select Start.](../../assets/sources/vision-pro-tracking-streamer-start.jpg) +![Tracking Streamer: note the headset IP and select Start.](https://openrobokit.com/assets/docs/retargetingkit/assets/sources/vision-pro-tracking-streamer-start.jpg) Stream head, wrist, and finger tracking from Apple Vision Pro with `avp_stream`. RetargetingKit can drive robot hands, robot arms, or synchronized arm-and-hand assemblies, and replay the same tracking from a recording. @@ -12,7 +12,7 @@ Stream head, wrist, and finger tracking from Apple Vision Pro with `avp_stream`. ## Connect the headset -![Open Tracking Streamer from the Vision Pro app launcher.](../../assets/sources/vision-pro-tracking-streamer-app.jpg) +![Open Tracking Streamer from the Vision Pro app launcher.](https://openrobokit.com/assets/docs/retargetingkit/assets/sources/vision-pro-tracking-streamer-app.jpg) 1. Connect the workstation and Vision Pro to the same Wi-Fi network. 2. Open **Tracking Streamer** on Vision Pro.