🎯 Project Overview

🤗 This repository is a fork of lerobot by XenseRobotics, used for Xense's multimodal tactile data acquisition system. This branch tracks upstream lerobot v5.1, with Xense-specific robots (Flexiv Rizon4 RT, Elite CS66 RT, reBot B601 RT, and ARX5 — each single-arm and bimanual; plus TacCap tactile grippers), teleoperators (Pico4 VR, dual SpaceMouse, TRLC leader, gamepad) and tactile cameras layered on top. For generic lerobot usage (datasets, policies, training scripts) refer to the upstream README.

🔧 Installation

Tested on Ubuntu 22.04 and 24.04, NVIDIA driver ≥ 570.144. Use Mamba (strongly recommended over plain conda — it's much faster on the robostack-staging channel that ships ROS Humble + SOEM). v5.1 pins Python 3.12 and PyTorch ≥ 2.2 with CUDA 12.8.

curl -L -O "https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-$(uname)-$(uname -m).sh"
bash Miniforge3-$(uname)-$(uname -m).sh

📦 Environment Setup

Step 1: 📂 Clone the repository with all submodules:

git clone \
  --recurse-submodules \
  https://github.com/XenseRobotics-AI/lerobot-xense.git
cd lerobot-xense

If you already cloned without submodules, initialize them manually:

git submodule update --init --recursive --progress

Optional — internal network (GitLab mirror). The default clone above uses the public GitHub mirrors (git@github.com:XenseRobotics-AI/*). Company-network developers can instead point the submodules at the internal GitLab server; the internal URLs live in .gitmodules.gitlab and a helper repoints your local remotes without touching the committed .gitmodules (the submodules listed in .gitmodules.gitlab have matching pins on both remotes; libpyrebot currently stays on GitHub):

XENSE_GITLAB_HOST=<host> scripts/submodule-remote.sh gitlab   # `github` switches back
git submodule update --init --recursive

The GitLab host is not committed — this repository is public, so the address comes from XENSE_GITLAB_HOST instead. Ask a teammate for the value and export it in your shell profile.

This repository uses third_party/ git submodules to manage hardware SDK dependencies:

Submodule	Installed package
third_party/ARX5_SDK	pyarx
third_party/libpyflexiv	flexiv_rt
third_party/libpyrebot	rebot_rt
third_party/XGripper	xgripper
third_party/libpyelite	libpyelite (Elite CS SDK + native Pink IK)
third_party/taccap-gripper	xense.taccap (TacCap UMI tactile gripper SDK)

The libpyrebot submodule provides C++ 100 Hz Cartesian control and IK for reBot B601 DM and RS arms. Build its pybind11 extension for the active Python environment before using the rebot_b601_rt or bi_rebot_b601_rt robot types (see the build steps below and its README). It supports DM serial bridges and SocketCAN, and RS over SocketCAN. Complete DM and RS model assets are bundled. Platform 09 uses the identified DM_TacCap00 v1 assembly: its dm-taccap00-09.yaml recipes explicitly set each arm's urdf_path and end_frame: taccap_eef_link. Other platforms must select the URDF/TCP matching their actual arm, mount and payload; DM does not imply TacCap00. The gripper uses its separate driver. These recipes enable model gravity plus matching _offset.yaml holding biases via gravity_use_joint_offsets: true. Legacy configurations without urdf_path retain the family/gripper fallback (DM_TacCap00 for DM TacCap, RS_TacCap05 for RS TacCap, otherwise the original DM/RS model).

Platform 09 recording includes the RealSense D415 head camera (RGB 640×480 at 30 Hz), plus the discovered wrist and tactile cameras. Use the bimanual recording recipe and matching teleoperation recipe; see platform/model configuration for commands and adapting a recipe to another platform.

Both control modes use native timed Cartesian interpolation by default (interpolate_cartesian_targets: true): translation is linear, orientation uses SLERP, and each 100 Hz substep runs IK from the last limited command. The DM and RS TacCap recipes select mit; pos_vel uses the same interpolation. Their joint command caps are 1.5 rad/s, with A reset capped at 0.5 rad/s.

Switching an arm to control_mode: pos_vel automatically sets its effective gravity_compensation_scale to 0.0 during configuration parsing and logs the override. The recipe's MIT gravity settings can remain in the YAML; gravity and identified torque offsets are not configured in the native controller in pos_vel mode. For dual-arm recipes, this is applied independently to each arm. MIT mode still requires explicit gravity_torque_limits_nm when gravity feedforward is enabled.

DM and RS share the same Cartesian interpolation and joint coordination. For TCP speed, set max_cartesian_linear_speed (m/s) and max_cartesian_angular_speed (rad/s) under each robot/arm config. All DM TacCap recipes use 1.0 m/s and 6.28 rad/s, with coordinate_cartesian_speed: true. Each 100 Hz step scales translation and rotation together; joint constraints can reduce the whole step further. max_joint_speed: 1.5 remains the joint cap in rad/s, and reset remains 0.5 rad/s. TCP limits apply to the native commanded trajectory, including catch-up, and require interpolate_cartesian_targets: true. They are distinct from the Pico output filters teleop.max_pos_velocity / max_rot_velocity; actual speed can be lower under joint constraints. Both robot fields default to None (disabled) for existing configurations. Rebuild libpyrebot to use these fields.

Streaming Cartesian tracking approaches unreachable targets with bounded local IK steps instead of rejecting the entire waypoint. Each accepted step reduces the sampled pose error and stays within joint position limits and the per-cycle speed bound. At a local boundary it holds when no improving step is available; returning targets are followed from the last command without queuing a distant joint solution. This preserves the Pico pose mapping and applies independently to both arms in either control mode. It is a local approximation, not a global nearest-pose or collision-avoidance planner. Rebuild libpyrebot to use this behavior; tracking logs expose tracking_limited and tracking_progress with the updated native library.

The teleoperation and recording recipes set reset_position_rad: [0, 0.8, 0.8, -0.5, 0, 0] (J1..J6 in radians) for startup and the A-button reset. Their reset_tolerance_rad: 0.06 allows the observed MIT steady-state joint offsets of about 0.04–0.05 rad at this pose. The command must reach the target and all measured joints must remain within tolerance for 0.1 s. This changes the completion criterion; the commanded pose is unchanged. The B601 driver waits for completion before connect() returns, so Pico is seeded from the measured pose after startup movement. Both arms connect before moving together. connect(go_to_start=False) skips startup movement, as does teleoperation dryrun.

These recipes enable robot.reset_on_exit: true: after Ctrl+C or normal completion, the arm(s) return to six zero joint angles at reset_joint_speed (0.5 rad/s), then disconnect. This also applies to Ctrl+C or a settling timeout during startup movement, after the arm(s) have connected. The exit target is independent of reset_position_rad. MIT and position/velocity modes use the same reset path. Exit feedback must settle within the separate exit_reset_tolerance_rad (0.03 rad); reset_timeout_sec bounds the wait. While waiting, the driver reports measured joint errors and the CAN channel every two seconds, including the joints outside tolerance. A second Ctrl+C aborts the return and disconnects immediately. Connection failures before startup movement, teleoperation errors and dry runs skip the exit reset; a fault during reset stops it. Use --robot.reset_on_exit=false to disable this behavior. The driver handles the return in disconnect(); disconnect(reset=False) closes without returning.

The RS TacCap recipes set mit_velocity_feedforward: false: MIT tracks the interpolated position with desired velocity dq=0, retaining its configured stiffness and damping. This follows libtron2rt's Cartesian command behavior and avoids sending waypoint-boundary finite-difference velocity pulses into MIT's velocity term. --robot.mit_velocity_feedforward=true restores the previous behavior for comparison. This option does not affect pos_vel; native API and general robot-config defaults remain true for compatibility. Tracking logs distinguish velocity (position-ramp slope) from mit_dq (MIT desired velocity before encoding). Physical smoothness and load tracking still require a hardware check.

B601 uses local tactile-camera adapters for cached observations and bounded USB reopen retries, plus local TacCap discovery for gripper hubs sharing an upstream USB hub. Other robots retain the original shared camera and USB grouping behavior. The B601 native SDK is loaded only when its driver is requested.

B601 DM and RobStride (RS) motor zeros can be set separately from RT control. Install motorbridge>=0.5.6,<0.6 (or the rebot-calibration extra), then run:

lerobot-calibrate \
  --robot.type=rebot_b601_rt \
  --robot.motor_family=dm \
  --robot.channel=/dev/ttyACM0 \
  --robot.id=follower1 \
  --robot.has_gripper=false

# RobStride / 灵足, over SocketCAN
lerobot-calibrate \
  --robot.type=rebot_b601_rt \
  --robot.motor_family=rs \
  --robot.channel=can0 \
  --robot.id=follower1 \
  --robot.has_gripper=false

For both arms, run the sequential calibration script with a bimanual teleop or recording recipe:

python -m lerobot.robots.bi_rebot_b601_rt.bi_calibration \
  --config_path=recipes/teleop/bi_rebot_b601_rt/dm-taccap00-09.yaml

or

python -m lerobot.robots.bi_rebot_b601_rt.bi_calibration \
  --config_path=recipes/teleop/bi_rebot_b601_rt/dm-taccap00-10.yaml

It calibrates the right arm first, then the left, with a separate mechanical-zero confirmation for each arm. An error or interrupt stops the sequence; a completed right-arm calibration remains saved if the left arm fails. It uses the recipe's motor settings and <robot.id>_right / <robot.id>_left record IDs without starting RT control or peripherals.

The calibration path in rebot_b601_rt disables the selected motors, checks their communication, then waits for you to manually place the arm in its mechanical zero pose before sending zero commands. It does not enable motors, switch control modes, or automatically move the arm. It reuses channel, dm_serial_baud, motor_configs, and has_gripper from the RT configuration. The example above calibrates the six arm motors with a separate USB/TacCap gripper; use --robot.has_gripper=true to include CAN gripper motor 7. Calibration records are stored separately under calibration/robots/rebot_b601_rt/dm/ and rs/. An existing follower1.json is kept unless you enter c at the recalibration prompt. After setting zeros, calibration requests a fresh position register reply from each motor (DM p_m, RS mechPos) and prints its position in radians/degrees and PASS/FAIL. Keep the arm supported in the mechanical zero pose until this finishes. All positions must be finite and within ±0.01 rad (about ±0.57°) before the record is saved; this check is independent of reset_tolerance_rad. A read timeout or out-of-tolerance position fails calibration and leaves any existing JSON intact, although the hardware zero commands have already been sent. This readback verifies the current zero, not persistence across a power cycle or correct manual alignment. The JSON uses the legacy Seeed format; its range fields are placeholders, not measured travel limits. Use rebot_b601_rt / bi_rebot_b601_rt for teleoperation and recording.

B601 recording recipes are available for one arm and two arms. Both explicitly select the RS + TacCap05 v2 URDF and use MIT with TacCap camera discovery; the recording recipe maps the left arm to can1 and the right arm to can0. Set these to match your adapters. For example:

lerobot-record \
  --config_path=recipes/record/bi_rebot_b601_rt/rs-taccap05.yaml \
  --dataset.repo_id=Xense/bi_rebot_pick_cube_001 \
  --dataset.single_task="Pick up the cube and place it in the box" \
  --dataset.num_episodes=10

The recipes save locally (push_to_hub: false); use a new dataset name for a new run, or --resume=true to append to a compatible existing dataset. Recording uses dataset.fps, with a default of 30. Single-arm recording also supports SpaceMouse by overriding --teleop.type=spacemouse.

For B601 recording, right arrow ends the current recording/preparation phase, left arrow requests a retake, and Esc stops the session. During recording, Space or the Pico A button returns the arm(s) to the configured joint pose; SpaceMouse uses both buttons. B/X/Y are not mapped to episode controls. The preparation interval uses the same generic loop as other robots: teleoperation without adding frames or handling reset buttons. Startup and reset completion seed the teleoperator from the measured TCP pose(s), and no VR commands are sent during a native reset. An active reset finishes before saving or leaving a phase, which can extend that phase beyond its configured duration.

Normal frames pair the current observation/images with the accepted action returned by send_action; native Cartesian interpolation can accept it asynchronously. B601 uses the shared run_rt_record_loop with a device policy; after reset it resynchronizes the teleoperator before reading the next action. During reset, the action is the measured TCP and gripper, paired with the previous observation as in the other RT recording loops; the reset-trigger frame is skipped. Single-arm actions have 10 values and dual-arm actions have 20 (position, two rotation columns, and gripper per arm). Keep use_joint_observation: false on each arm, as in the recipes: the shared reset-frame logic requires TCP observation keys.

Normal completion, Esc, and Ctrl+C honor robot.reset_on_exit. Like the other robots, Ctrl+C interrupts recording without saving the current episode; previously saved episodes are retained. A second Ctrl+C during the return aborts it. Recording uses the common cleanup order: video encoding, dataset finalization, then disconnect() for connected devices. B601's driver handles the configured exit return and checks the controller for faults before starting it; the recording script does not select a different disconnect mode based on the exception. Image-writer cleanup uses the same exception decorator as the other recording loops. These recording paths have offline regression coverage; validate them on your rig before a collection run.

xensesdk is not a submodule — it is installed from PyPI (xensesdk==2.1.2, the published cp312 manylinux wheel, which bundles the patched libxense_c.so flash reader). libpyelite vendors the Elite CS C++ SDK as its own submodule (third_party/libpyelite/vendor/elite_cs_sdk), so no network fetch of the C++ source.

xensevr_pc_service_sdk (Pico4 teleop/tracker) has no submodule either. Its pybind11 sources live in-repo under src/lerobot/teleoperators/pico4/xensevr-pc-service-pybind/, and the C SDK they link against — PXREARobotSDK.h plus libPXREARobotSDK.so — is copied straight out of the xensevr-pc-service .deb that --install fetches (see Step 3). The daemon in that .deb is what the teleop talks to at runtime anyway, so it was never optional; carrying a 33 MiB checkout of the service's Qt tree and prebuilt gRPC archives just to rebuild a library we were already downloading was the part that was.

Step 2: 🐍 Create and activate the mamba environment:

bash ./setup_env.sh --mamba lerobot-xense
mamba activate lerobot-xense

The default env name baked into conda_environment.yaml is lerobot-xense-py312. You can pass a different name to --mamba, but the rest of this README and the openpi project assume lerobot-xense-py312.

Step 3: 📦 Install LeRobot-Xense and hardware SDK bindings:

bash ./setup_env.sh --install

This step will:

• Update the conda environment from conda_environment.yaml
• Install the main package from pyproject.toml
• Install xensesdk from PyPI (xensesdk==2.1.2)
• Install the XenseVR PC Service daemon from its .deb (~116 MB, fetched from the v0.2.1 release into /opt/apps/roboticsservice; override with XENSEVR_DEB_URL, or point XENSEVR_DEB at a local file for offline installs) — only with --pico4 / --bi_pico4 or a full install
• Build and install the SDK packages supported by setup_env.sh: pyarx, rebot_rt, flexiv_rt, xensevr_pc_service_sdk (built against the .deb's client SDK), xgripper, libpyelite (Elite CS SDK + native Pink IK, built from its submodule), and xense.taccap (TacCap UMI gripper)
• Configure SpaceMouse udev rules and HID permissions automatically

You will be prompted for sudo password during installation (for ARX5 real-time capability and udev rules).

For reBot B601 DM/RS, an existing LeRobot environment can install or rebuild just rebot_rt with the following command from the repository root:

git submodule update --init --recursive third_party/libpyrebot
bash scripts/install_rebot.sh

On Linux x86_64, the installer creates a separate conda-forge environment for Pinocchio 3.9 and its build tools (GCC 12 for Ubuntu 22.04/24.04 compatibility), compiles for the active Python ABI, builds a wheel locally with --no-build-isolation --no-deps --no-index, then installs and verifies it in the LeRobot environment. Verification loads both DM/RS models and computes gravity without connecting to hardware. Pinocchio 3.9 needs Boost 1.88, which conflicts with the ROS Jazzy dependencies; do not install it into the LeRobot conda environment. No ROS/CUDA packages are changed by this standalone installer. If the target interpreter loads an older system C++ runtime first, the installer validates a compatible runtime and writes lerobot_rebot_runtime.pth into that Python environment to preload it before NumPy. It prefers the target conda environment's own runtime, then the isolated dependency environment's runtime; it does not replace sitecustomize.py or any system library.

The dependency environment defaults to ~/.local/share/lerobot-xense/rebot-rt-deps/py<major.minor> (under XDG_DATA_HOME when set). Keep this directory after installation: the compiled module links to its shared libraries. Override it with REBOT_DEPS_PREFIX, or use --skip-deps to reuse an already provisioned environment. A complete dependency environment is reused without network access. The first run needs access to conda-forge; this avoids the failing PyPI build-dependency download but does not repair a broken network connection.

Re-run the installer after changing the submodule or Python version, and on each new computer. Build directories and native binaries from another computer are not reused. pip install -e third_party/libpyrebot alone does not compile the extension. For a full reBot + TacCap + Pico4 station install, use bash ./setup_env.sh --install --rebot --taccap --pico4.

🎛️ Selective hardware install (build only the SDKs you need)

By default --install builds every hardware SDK supported by the script. On a station that only uses one or two devices, pass per-hardware-family selectors after --install to build core + only those SDKs — faster, and it skips SDKs you can't (or don't want to) build on that host:

# core + Flexiv + TacCap only (arms auto-include the xense gripper stack)
bash ./setup_env.sh --install --flexiv --taccap

# core + Elite only
bash ./setup_env.sh --install --elite

# core only — no hardware SDK bindings
bash ./setup_env.sh --install --core

# list every selector
bash ./setup_env.sh --install --help

Selector	Builds	Robots / teleoperators enabled
--rebot, --bi_rebot	rebot_rt (isolated Pinocchio dependencies)	rebot_b601_rt, bi_rebot_b601_rt
--flexiv, --bi_flexiv	flexiv_rt (+ xense)	flexiv_rizon4_rt, bi_flexiv_rizon4_rt
--elite, --bi_elite	libpyelite (+ xense)	elite_cs66_rt, bi_elite_cs66_rt
--taccap, --bi_taccap	xense.taccap (+ xense)	taccap_follower gripper (on any arm)
--xense	xensesdk + xgripper (XGripper)	serial gripper + tactile sensors
--arx5, --bi_arx5	pyarx	arx5_follower, bi_arx5
--pico4, --bi_pico4	xensevr_pc_service_sdk (+ PC Service .deb)	pico4, bi_pico4 teleop
--spacemouse	pyspacemouse	spacemouse teleop
--dynamixel, --trlc	dynamixel-sdk	trlc_leader, bi_trlc teleop
--all	everything (explicit)	—
--core, --none	nothing (core only)	—
(no selector)	everything (default, backward compatible)	—

Notes:

• No selector = install all — the existing bash ./setup_env.sh --install behavior is unchanged.
• Arms auto-include xense — --flexiv / --elite / --taccap also build the xense gripper stack, because those arms drive xense grippers.
• Post-install verification only checks the SDKs you selected.
• Code stays compatible with a partial install. import lerobot and the CLIs (lerobot-teleoperate, lerobot-record, …) start fine even when an SDK is absent — a device whose SDK isn't installed simply won't appear as a --robot.type / --teleop.type choice (and only errors, with a rebuild hint, if you try to construct it).

Step 4: ✅ Verify the installation. These checks assume a full --install; on a selective install (e.g. --flexiv --taccap) only the SDKs you selected are built, so verify just those — the installer already prints a per-SDK verification summary at the end.

python -c 'import pyarx; print("pyarx OK ->", pyarx.__file__)'
python -c 'import flexiv_rt; print("flexiv_rt OK ->", flexiv_rt.__file__)'
python -c 'import xensevr_pc_service_sdk; print("xensevr_pc_service_sdk OK ->", xensevr_pc_service_sdk.__file__)'
python -c 'import xensesdk; print("xensesdk OK ->", xensesdk.__file__)'
python -c 'import xgripper; print("xgripper OK ->", xgripper.__file__)'
python -c 'import libpyelite; print("libpyelite OK ->", libpyelite.__file__, libpyelite.IK_BACKEND)'
python -c 'import xense.taccap; print("xense.taccap OK ->", xense.taccap.__file__)'

Step 5: 📌 Note on FFmpeg / video: v5.1 no longer pins ffmpeg through conda (the robostack ICU pin conflicted with newer ffmpeg builds). Video encoding/decoding is handled by torchcodec + av wheels installed via setup_env.sh --install. If you need a system ffmpeg with libsvtav1, install it separately (apt or upstream static build):

# Optional: verify torchcodec wheel is loadable
python -c 'import torchcodec; print("torchcodec OK ->", torchcodec.__version__)'

ARX5 Real-time Thread Permissions

The ARX5 SDK requires CAP_SYS_NICE on the Python interpreter for real-time CAN thread scheduling. This is handled by setup_env.sh --install, but can be set manually:

PY_EXE=$(python -c 'import sys, os; p = sys.executable; print(os.path.realpath(p))')
sudo setcap cap_sys_nice+ep "$PY_EXE"
getcap "$PY_EXE"  # should show: cap_sys_nice+ep

🚀 Running teleop & record (recipes)

Teleoperation and recording are driven by recipe YAML files under recipes/ (see recipes/README.md for the full guide). Pass one with --config_path:

lerobot-teleoperate --config_path=recipes/teleop/bi_elite_cs66_rt/diagonal-07-taccap.yaml
lerobot-record     --config_path=recipes/record/bi_flexiv_rizon4_rt/assemble_box-xgripper.yaml

A recipe is self-contained: it carries both the bench hardware (controller IPs/SNs, camera SNs, mount geometry, per-arm home/start poses) and the run's tuning (control mode, servo gains, guards, gripper force, dataset fields). One file is everything a run needs; adding a bench is a new recipe, no Python change.

ls recipes/teleop/bi_flexiv_rizon4_rt/  # forward-04, forward-05, forward-dewu, diagonal-02
ls recipes/teleop/bi_elite_cs66_rt/     # diagonal-07, diagonal-08

Precedence is dataclass default < recipe < CLI, so an explicit --robot.left_robot_sn=… overrides the recipe for a one-off run.

Gripper (gripper:). Every robot takes one typed gripper block; a bimanual arm writes it once and both sides get a copy with side stamped in (the two are always a matched pair). Two backends:

• serial — XenseSerialGripper, a parallel jaw over USB serial. Left/right are auto-discovered by board-SN parity (odd SN → left, even SN → right) at connect, so no gripper SN is configured.
• taccap_follower — xense.taccap FollowerGripper, the centric TacCap gripper (MIT impedance). Left/right resolved from the firmware-burned SN, and its wrist
  • GSPS tactile cameras auto-discovered at connect.

robot:
  gripper:
    type: taccap_follower
    close_speed_radps: 3.0 # setpoint-ramp rate during travel (rad/s)
    auto_discover_cameras: true # sniff wrist + tactile off this gripper's hub
    enable_tactile: true
    undistort_wrist: true # rectify the wrist fisheye from the MCU's intrinsics
    fisheye_balance: 0.0 # 0 = calibrated focal length; 1 = 0.70x, widest FOV

What the gripper carries is configured on the gripper, not on the arm — swap a TacCap for an XGripper and the sensors on the hub change while the arm does not. undistort_wrist and fisheye_balance are taccap-only, because the wrist lens' intrinsics live in that gripper's own MCU flash.

The block is decoded through the gripper registry, so a knob belonging to the other backend — or a typo — is rejected at parse time rather than silently ignored. That is not cosmetic here: undistort_wrist on an XGripper block, or with auto_discover_cameras off, is refused rather than accepted and quietly ignored. See src/lerobot/grippers/README.md for the full field lists and the driver contract.

🐭 SpaceMouse Teleoperation System

This project includes advanced SpaceMouse support with both single and dual-device modes for precise robotic control.

Dependencies

System Requirements:

• Ubuntu 22.04 / 24.04 (tested) or other Linux distributions
• Python 3.12+
• libhidapi (installed via apt)

Python Packages:

• pyspacemouse - Modern cross-platform SpaceMouse library
• hidapi - Python wrapper for HID API
• easyhid - Easy-to-use HID library (dependency of pyspacemouse)

All Python dependencies are automatically installed by setup_env.sh --install.

Permissions Setup

SpaceMouse requires proper udev rules to allow non-root access. This is configured automatically by setup_env.sh --install (see Step 3 in the Installation section above).

Testing Your SpaceMouse

After installation and permissions setup, test your SpaceMouse:

# Basic functionality test (prints real-time 6-DoF values)
python src/lerobot/teleoperators/spacemouse/examples/01_basic.py

# Device discovery (lists connected SpaceMice by path/serial)
python src/lerobot/teleoperators/spacemouse/examples/05_discovery.py

The test script will display real-time position (x, y, z) and orientation (roll, pitch, yaw) values as you move the SpaceMouse.

📝 Note: If you're using a 3Dconnexion Universal Receiver (wireless), you may see multiple devices listed (e.g., 14 "UniversalReceiver" entries). This is normal - the receiver exposes multiple HID interfaces for different functions. PySpaceMouse will automatically select the correct interface for 6-DoF input.

Features

• ✅ Modern PySpaceMouse Integration: Uses PySpaceMouse library for cross-platform SpaceMouse support
• ✅ No System Services Required: Direct HID communication, no need for spacenavd daemon
• ✅ Single Device Mode: Traditional 6-DoF control with one SpaceMouse
• ✅ Dual Device Mode: Advanced left/right hand coordination for complex manipulation
• ✅ Flexible Axis Assignment: Configure which device controls position vs orientation
• ✅ Independent Sensitivity: Per-device sensitivity settings for optimal control

Single Device Configuration

from lerobot.teleoperators.spacemouse import SpacemouseConfig, SpacemouseTeleop

# Standard single SpaceMouse setup (default)
config = SpacemouseConfig(
    pos_sensitivity=0.8,  # Position control sensitivity
    ori_sensitivity=1.5,  # Orientation control sensitivity
    deadzone=0.1,  # Deadzone threshold
    frequency=200,  # Polling frequency (Hz)
)

teleop = SpacemouseTeleop(config)

Dual Device Configuration

Perfect for complex robotic tasks requiring precise position and orientation control:

from lerobot.teleoperators.spacemouse import SpacemouseConfig, DeviceConfig

# Left hand controls position, right hand controls orientation
config = SpacemouseConfig(
    multi_device_mode=True,
    left_device=DeviceConfig(
        device_index=0,
        enabled_axes=(True, True, True, False, False, False),  # X, Y, Z position only
        pos_sensitivity=0.8,
        ori_sensitivity=0.0,  # Disabled
    ),
    right_device=DeviceConfig(
        device_index=1,
        enabled_axes=(False, False, False, True, True, True),  # Roll, pitch, yaw only
        pos_sensitivity=0.0,  # Disabled
        ori_sensitivity=1.5,
    ),
)

teleop = SpacemouseTeleop(config)

Example Configurations

See examples/09_custom_config.py and examples/03_multi_device.py (under src/lerobot/teleoperators/spacemouse/) for complete configuration examples including:

• Position/Orientation split control
• Dual-arm robot control
• Fine/Coarse movement control

Use Cases

• 🤖 Dual-Arm Robots: Independent control of two robotic arms
• 🎯 Precision Manipulation: Decouple position and orientation control for fine tasks
• 🔄 Complex Assembly: Left hand positions, right hand orients components
• 🏭 Industrial Applications: Enhanced ergonomics and control precision

Hardware Requirements

• Single Mode: Any 3Dconnexion SpaceMouse device
• Dual Mode: Two identical SpaceMouse devices (e.g., two SpaceNavigators)

Supported Devices

All 3Dconnexion devices supported by PySpaceMouse:

• SpaceNavigator
• SpaceMouse Pro
• SpaceMouse Wireless
• SpaceMouse Compact
• And more...

🤖 Recording on Flexiv Rizon4

Bimanual Flexiv Rizon4 RT + BiPico4 Record Controls

For lerobot-record with --robot.type=bi_flexiv_rizon4_rt --teleop.type=bi_pico4, the controller buttons are mapped as follows:

Controller button	Keyboard equivalent	Action
Right A	go_start	Reset both arms to start pose (RT non-blocking, recording continues)
Left X	rerecord_episode	Discard current episode and re-record
Left Y	exit_early	Finish current episode early
Right B	stop_recording	Stop the recording session

Button state is refreshed via BiPico4.poll_buttons() at the top of each loop iteration, before event checks. Keyboard events and controller buttons are unified into the same events[] checks — both are equal-priority input sources.

During RT reset, the record loop keeps running: observations are still sampled, teleop actions are still read, and the teleop pose is re-synced to the robot once the reset trajectory finishes.

Record Loop Implementation (flexiv_rizon4_rt_record_loop)

This section documents the dataset construction logic in flexiv_rizon4_rt_record_loop, which handles both normal teleoperation recording and the RT reset trajectory recording for bi_flexiv_rizon4_rt + bi_pico4.

Loop Structure

while timestamp < control_time_s:
    poll_buttons()              # lightweight button refresh (no pose computation)

    # Unified event checks — keyboard OR controller button, symmetric
    stop_recording  / B  →  break
    rerecord        / X  →  break
    exit_early      / Y  →  break
    go_start        / A  →  reset_to_initial_position(), recording continues

    get_observation()
    check robot_is_moving + sync teleop if reset just finished
    get_action()
    send_action / dataset write

State Variables

Variable	Role
reset_triggered	Per-frame flag. Set True the frame reset is triggered. Skips send_action and dataset write for that frame only. Resets to False at the start of every iteration.
prev_rt_moving	Edge-detection flag. Set True while robot.rt_moving is True. Cleared to False when movement stops, triggering one call to _sync_rt_teleop_to_robot_pose().
prev_observation_frame	Holds the previous frame's observation. Used by shifted-frame logic to pair obs[t-1] with the robot's actual position at obs[t] as the action.

Three Frame Modes

1. Normal teleoperation (robot_is_moving=False, reset_triggered=False)

robot.send_action(teleop_action) → sent_action
dataset: { obs[t],  action = sent_action[t] }   # direct frame
prev_observation_frame = obs[t]

2. Reset trigger frame (reset_triggered=True)

robot.reset_to_initial_position()   # C++ RT thread takes over arm control
send_action  → skipped
dataset      → skipped
prev_observation_frame = obs[T]     # saved as anchor for next iteration

obs[T] is intentionally not written to the dataset. It is used as prev_observation_frame for the first shifted frame on the next iteration, so it appears exactly once — without this skip it would appear twice (once as a direct frame, once as the prev of the first shifted frame).

3. RT reset in progress (robot_is_moving=True)

send_action  → skipped (C++ RT thread drives the arm autonomously)
current_as_action = { key: obs[t][key] for key in robot.action_features }
# robot.action_features = left/right TCP pose (9D each) + gripper (1D each) = 20D total
# Iterates action_features keys only — image keys in obs[t] are excluded automatically.
dataset: { obs[t-1],  action = current_as_action }   # shifted frame
prev_observation_frame = obs[t]

The action is extracted from the current observation using the same keys as robot.action_features. This records where the robot actually moved to, not what the teleop commanded — the same shifted-frame convention used by bi_arx5_record_loop.

Per-Frame Decision Table

Frame	reset_triggered	robot_is_moving	send_action	Dataset write	prev_obs updated to
T-1 (normal teleop)	False	False	✓	{obs[T-1], action[T-1]} direct	obs[T-1]
T (reset triggered)	True	False	skipped	skipped	obs[T]
T+1 (RT moving)	False	True	skipped	{obs[T], state_20d[T+1]} shifted	obs[T+1]
T+2 (RT moving)	False	True	skipped	{obs[T+1], state_20d[T+2]} shifted	obs[T+2]
…	False	True	skipped	shifted	…
N+1 (reset done, teleop synced)	False	False	✓	{obs[N+1], action[N+1]} direct	obs[N+1]

state_20d[t] = {k: obs[t][k] for k in robot.action_features} — left/right TCP pose (9D each) + gripper (1D each), image keys excluded.

Complete Frame Sequence Around a Reset

frame T-2  normal teleop  →  dataset: { obs[T-2], action[T-2] }
frame T-1  normal teleop  →  dataset: { obs[T-1], action[T-1] },  prev=obs[T-1]
frame T    reset trigger  →  dataset: skipped,                     prev=obs[T]
frame T+1  rt_moving      →  dataset: { obs[T],   state_20d[T+1] },  prev=obs[T+1]
frame T+2  rt_moving      →  dataset: { obs[T+1], state_20d[T+2] },  prev=obs[T+2]
  ...
frame N    rt_moving      →  dataset: { obs[N-1], state_20d[N] },    prev=obs[N]
frame N+1  reset done     →  _sync_rt_teleop_to_robot_pose()
           normal teleop  →  dataset: { obs[N+1], action[N+1] }

Post-Reset Teleop Sync

When prev_rt_moving transitions True → False (frame N+1), _sync_rt_teleop_to_robot_pose() is called once. This reads the robot's current TCP pose (now at start position) and calls teleop.reset_to_pose(), updating the Pico4's internal _start_pos reference. Without this sync the teleop would compute position deltas from the pre-reset pose, causing the arm to jump on the first grip after reset.

🔑 The LeRobotDataset format

A dataset in LeRobotDataset format is very simple to use. It can be loaded from a repository on the Hugging Face hub or a local folder simply with e.g. dataset = LeRobotDataset("lerobot/aloha_static_coffee") and can be indexed into like any Hugging Face and PyTorch dataset. For instance dataset[0] will retrieve a single temporal frame from the dataset containing observation(s) and an action as PyTorch tensors ready to be fed to a model.

A specificity of LeRobotDataset is that, rather than retrieving a single frame by its index, we can retrieve several frames based on their temporal relationship with the indexed frame, by setting delta_timestamps to a list of relative times with respect to the indexed frame. For example, with delta_timestamps = {"observation.image": [-1, -0.5, -0.2, 0]} one can retrieve, for a given index, 4 frames: 3 "previous" frames 1 second, 0.5 seconds, and 0.2 seconds before the indexed frame, and the indexed frame itself (corresponding to the 0 entry). See example 1_load_lerobot_dataset.py for more details on delta_timestamps.

Under the hood, the LeRobotDataset format makes use of several ways to serialize data which can be useful to understand if you plan to work more closely with this format. We tried to make a flexible yet simple dataset format that would cover most type of features and specificities present in reinforcement learning and robotics, in simulation and in real-world, with a focus on cameras and robot states but easily extended to other types of sensory inputs as long as they can be represented by a tensor.

Here are the important details and internal structure organization of a typical LeRobotDataset instantiated with dataset = LeRobotDataset("lerobot/aloha_static_coffee"). The exact features will change from dataset to dataset but not the main aspects:

dataset attributes:
  ├ hf_dataset: a Hugging Face dataset (backed by Arrow/parquet). Typical features example:
  │  ├ observation.images.cam_high (VideoFrame):
  │  │   VideoFrame = {'path': path to a mp4 video, 'timestamp' (float32): timestamp in the video}
  │  ├ observation.state (list of float32): position of an arm joints (for instance)
  │  ... (more observations)
  │  ├ action (list of float32): goal position of an arm joints (for instance)
  │  ├ episode_index (int64): index of the episode for this sample
  │  ├ frame_index (int64): index of the frame for this sample in the episode ; starts at 0 for each episode
  │  ├ timestamp (float32): timestamp in the episode
  │  ├ next.done (bool): indicates the end of an episode ; True for the last frame in each episode
  │  └ index (int64): general index in the whole dataset
  ├ meta: a LeRobotDatasetMetadata object containing:
  │  ├ info: a dictionary of metadata on the dataset
  │  │  ├ codebase_version (str): this is to keep track of the codebase version the dataset was created with
  │  │  ├ fps (int): frame per second the dataset is recorded/synchronized to
  │  │  ├ features (dict): all features contained in the dataset with their shapes and types
  │  │  ├ total_episodes (int): total number of episodes in the dataset
  │  │  ├ total_frames (int): total number of frames in the dataset
  │  │  ├ robot_type (str): robot type used for recording
  │  │  ├ data_path (str): formattable string for the parquet files
  │  │  └ video_path (str): formattable string for the video files (if using videos)
  │  ├ episodes: a DataFrame containing episode metadata with columns:
  │  │  ├ episode_index (int): index of the episode
  │  │  ├ tasks (list): list of tasks for this episode
  │  │  ├ length (int): number of frames in this episode
  │  │  ├ dataset_from_index (int): start index of this episode in the dataset
  │  │  └ dataset_to_index (int): end index of this episode in the dataset
  │  ├ stats: a dictionary of statistics (max, mean, min, std) for each feature in the dataset, for instance
  │  │  ├ observation.images.front_cam: {'max': tensor with same number of dimensions (e.g. `(c, 1, 1)` for images, `(c,)` for states), etc.}
  │  │  └ ...
  │  └ tasks: a DataFrame containing task information with task names as index and task_index as values
  ├ root (Path): local directory where the dataset is stored
  ├ image_transforms (Callable): optional image transformations to apply to visual modalities
  └ delta_timestamps (dict): optional delta timestamps for temporal queries

A LeRobotDataset is serialised using several widespread file formats for each of its parts, namely:

• hf_dataset stored using Hugging Face datasets library serialization to parquet
• videos are stored in mp4 format to save space
• metadata are stored in plain json/jsonl files

Dataset can be uploaded/downloaded from the HuggingFace hub seamlessly. To work on a local dataset, you can specify its location with the root argument if it's not in the default ~/.cache/huggingface/lerobot location.

📝 Recent Updates

SpaceMouse System Upgrade (2025-01-23)

🎉 Major SpaceMouse System Overhaul:

• Modern Library Migration: Migrated from legacy spnav to modern PySpaceMouse library
• Cross-Platform Support: Now supports Linux, macOS, and Windows
• No System Dependencies: Removed requirement for spacenavd system service
• Dual-Device Support: Revolutionary dual SpaceMouse mode for advanced manipulation
• Flexible Configuration: Per-device sensitivity and axis assignment
• Hardware Independence: Direct HID communication for better reliability

Breaking Changes:

• spacenavd service is no longer required
• Configuration options have been expanded with new dual-device parameters
• Old single-device configurations remain fully compatible

Migration Benefits:

• ✅ Easier setup (no system services to configure)
• ✅ Better cross-platform compatibility
• ✅ More responsive input handling
• ✅ Advanced dual-hand control capabilities
• ✅ Future-proof with active library maintenance

Citation

If you use this codebase, please cite the original LeRobot project:

@misc{cadene2024lerobot,
    author = {Cadene, Remi and Alibert, Simon and Soare, Alexander and Gallouedec, Quentin and Zouitine, Adil and Palma, Steven and Kooijmans, Pepijn and Aractingi, Michel and Shukor, Mustafa and Aubakirova, Dana and Russi, Martino and Capuano, Francesco and Pascal, Caroline and Choghari, Jade and Moss, Jess and Wolf, Thomas},
    title = {LeRobot: State-of-the-art Machine Learning for Real-World Robotics in Pytorch},
    howpublished = "\url{https://github.com/huggingface/lerobot}",
    year = {2024}
}

If you use this fork (LeRobot-Xense) specifically, please also cite:

@misc{vertax2026lerobotxense,
    author = {vertax42 and Xense Robotics Team},
    title = {LeRobot-Xense: LeRobot with Xense Tactile Robotics Support},
    howpublished = "\url{https://github.com/XenseRobotics-AI/lerobot-xense}",
    year = {2026}
}