The project environment can be recreated from requirements.yaml:
conda env create -f requirements.yaml
conda activate mm_plannerTo verify the core packages:
python -c "import torch, mujoco, casadi, PySide6; print(torch.__version__)"requirements.yaml is a compact, project-oriented environment file. It lists
top-level dependencies only; low-level CUDA/runtime packages such as nvidia-*
are installed automatically by the PyTorch CUDA wheels when needed.
This package implements a modular PyTorch planner for a flying-crawling drone. It fuses onboard RGB observations, historical self-motion, and ordered task waypoints, then predicts motion mode, future flight waypoints, crawling action, and transition actions.
mm_planner/
├── config.py
├── main_test.py
├── models/
│ ├── vision_encoder.py
│ ├── trajectory_encoder.py
│ ├── waypoint_encoder.py
│ ├── fusion_transformer.py
│ ├── heads.py
│ └── planner.py
└── utils/
└── tensor_utils.py
The default vision encoder loads DINOv2 with torch.hub.load.
python -m mm_planner.main_testFor an offline smoke test, use the local mock vision backbone:
python -m mm_planner.main_test --mock-visionThe MuJoCo teaching environment can record clips while you fly with the joystick. Push joystick axis 4 above zero to start a clip, then push it below zero to end and save that clip. You can repeat this many times in one run.
Run it from this package directory:
conda run -n mm_planner python env/collect_clip_data.py \
--output-dir data/demos \
--episode-name perching_demo \
--fps 20 \
--width 224 \
--height 224You can add --clip-seconds 10 if you also want a maximum automatic segment
length. The default 0 means only axis 4 decides when clips end.
For state-only debugging without camera rendering:
conda run -n mm_planner python env/collect_clip_data.py --no-rgbThe PySide collection app provides live world-follow and FPV views plus editable task waypoints. Waypoints are shown only in the world-follow view and saved into each clip when joystick axis 4 starts recording:
conda run -n mm_planner python env/AppClipCollector.py \
--output-dir data/demos \
--episode-name perching_demoEach saved clip includes task_waypoints_world ([N, 3]) and
task_waypoints ([N, 7], planner feature format).
Each episode is written to data/demos/<episode-name>/ and split into
clip_000000.npz, clip_000001.npz, and so on. Each clip contains:
rgb:[T, H, W, 3]uint8 camera frames fromfpv_camera, unless--no-rgbtime:[T]qpos,qvel,ctrl,sensordatamode_id: current teaching control modejoystick_axes: roll, pitch, throttle, yaw, record axis, control-mode axis, takeoff/land axisroll_command,pitch_command,yaw_rate_command,throttle_commandthrust_command,target_omega,target_velocityyaw_command,position_target
To visualize recorded data in MuJoCo:
conda run -n mm_planner python env/replay_clip_data.py data/demos/perching_demoThis opens the MuJoCo viewer plus a separate first-person OpenCV window rendered
from fpv_camera. Press q or Esc in the first-person window to stop replay.
You can also replay a single clip and loop it:
conda run -n mm_planner python env/replay_clip_data.py \
data/demos/perching_demo/clip_000000.npz \
--loop \
--speed 0.5Useful replay options:
--fpv-camera fpv_camera--fpv-width 640 --fpv-height 480--no-fpv-window
For a richer dataset browser, use the PySide viewer:
conda run -n mm_planner python env/clip_viewer_app.py \
--data-dir data/demos/perching_demoThe app provides a clip list, synchronized world/FPV MuJoCo renders, playback
controls, a frame slider, and tabs for position, joystick, control, and command
curves. In the world render, enable Future points and set N to draw red
markers at the future UAV positions for 0.2s, 0.4s, ..., 0.2*N s. It needs a
graphical desktop/OpenGL context.
rgb:[B, 3, H, W]traj_mode_ids:[B, n_traj_encoder]traj_continuous:[B, n_traj_encoder, traj_continuous_dim]task_waypoints:[B, n_waypoints, waypoint_dim]task_waypoint_mask: optional[B, n_waypoints], whereTruemeans padding
mode_logits:[B, num_modes]flight_waypoints.mu/sigma:[B, pred_num_flight_waypoints, pred_waypoint_dim]flight_velocity_deltas.mu/sigma:[B, pred_num_flight_waypoints, 4]flight_acceleration_deltas.mu/sigma:[B, pred_num_flight_waypoints, 3]crawl_action:[B, 3]transition.contact_pos:[B, 3]transition.surface_normal:[B, 3]transition.yaw:[B, 1]transition.approach_speed:[B, 1]fused_feature:[B, fusion_dim]
The flight waypoint, velocity-delta, and acceleration-delta heads all generate
Gaussian action distributions autoregressively. Each returns mu and positive
sigma, and each supports optional teacher forcing during training.