MuJoCo Manipulation

A compact example stack for building MuJoCo manipulation applications with different robots, inverse-kinematics backends, and teleoperation devices.

The refactored core supports UR5e, AgileX Piper, Franka Panda, and Franka FR3. Native damped-least-squares (DLS) IK is included; Mink is optional. The backend registry is intentionally open so a future cuRobo adapter can be added without changing scene or teleoperation code.

Setup

python3 -m venv .venv
source .venv/bin/activate
python3 -m pip install --upgrade pip
python3 -m pip install -e ".[dev]"
make test

For Mink:

python3 -m pip install -e ".[dev,mink]"

The same commands are available as make install, make install-mink, and make check.

Try it

Run the robot-specific teleoperation examples:

python3 -m examples.ur5e.teleop --ik dls
python3 -m examples.piper.teleop --ik dls
python3 -m examples.teleop --robot panda --ik mink

Dual-arm tabletop setups use the same DLS/Mink switch:

python3 -m examples.ur5e.dual_arm_table --ik dls
python3 -m examples.ur5e.dual_arm_table --ik mink
python3 -m examples.piper.dual_arm_table --ik dls

The UR5e dual-arm examples use the physical lab layout: a 1.92 × 0.92 m table at 0.74 m, inverted bases at x = ±0.50 m and z = 1.75 m, and mirrored left/right starting joint poses. The RGB-D variant also retains the measured camera configuration:

python3 -m examples.ur5e.lab_dual_arm_rgbd --ik mink
python3 examples/load_dual_ur5e_table_rgbd_scene.py  # Mink + draggable mocap targets

# Optional headless point-cloud export:
MUJOCO_GL=egl python3 -m examples.ur5e.lab_dual_arm_rgbd \
  --headless --save-pointcloud outputs/lab_scene.ply

Single- and dual-arm pick-and-place programs:

python3 -m examples.ur5e.pick_place --ik dls
python3 -m examples.ur5e.dual_arm_pick_place --ik mink
python3 -m examples.piper.pick_place --ik dls
python3 -m examples.piper.dual_arm_pick_place --ik mink

The pick-and-place examples use a deterministic carry attachment after the close waypoint. This keeps the examples focused on scene composition, IK, and task sequencing rather than contact-tuning a different grasp for every robot.

Run a viewer-free smoke test:

python3 -m examples.teleop --robot fr3 --headless --steps 200
python3 -m examples.ur5e.dual_arm_table --headless

Keyboard controls are W/S, A/D, R/F for translation; U/O, I/K, J/L for rotation; [ and ] for the gripper; 0 to reset the target; and Q to quit.

The Mink RGB-D loader adds a colored mocap target at each end effector. Double-click a target, then use Ctrl + right-drag to move it vertically, Ctrl + Shift + right-drag to move it horizontally, or Ctrl + left-drag to rotate it.

Architecture

RobotDescription ──> SceneBuilder ──> RobotInstance
                                           │
Cartesian Pose ──> TeleopController ──> IKSolver ──> joint targets
                                      ├── DLS (built in)
                                      ├── Mink (optional)
                                      └── external backend (for example cuRobo)

• mujoco_manipulation/robots: declarative robot metadata, registry, and compiled model bindings.
• mujoco_manipulation/scene.py: generic prefixed one- or multi-robot assembly, plus hooks for task-specific objects and sensors.
• mujoco_manipulation/ik: a small solver contract, native DLS, and lazy optional backend registration.
• mujoco_manipulation/teleop: backend-neutral commands and target state.
• examples/ur5e and examples/piper: robot-specific entry points backed by the shared CLI in examples/teleop.py.

Add a robot

Register a RobotDescription containing the MJCF path, arm joint and actuator names, home position, and end-effector site. Attachable tools such as Robotiq use a GripperDescription; the UR5e example attaches robotiq_2f85 by default. If an MJCF has no tool site, provide its parent body and local position. No IK or teleop code needs to change.

Add an IK backend

Implement mujoco_manipulation.ik.IKSolver, then register a factory:

from mujoco_manipulation.ik import register_ik_backend

register_ik_backend("curobo", make_curobo_solver)

Keep large optional libraries inside the factory or adapter module so the base package still imports without them.