A complete 3D humanoid physics simulation where an agent learns to walk using reinforcement learning. Built with PyBullet for physics simulation, Stable-Baselines3 for PPO training, and Gymnasium for the environment interface.
- Realistic Physics: PyBullet-based simulation with a 22kg bipedal humanoid (torso, upper/lower legs, feet)
- PPO Training: Proximal Policy Optimization with parallel environments and observation normalization
- 25-Dimensional Observations: Joint states, torso pose/velocity, foot contacts
- Customizable Reward Function: Forward velocity, upright bonus, energy penalty, fall penalty
- Visualization: Real-time PyBullet GUI, TensorBoard monitoring, video recording
- Production Ready: CLI scripts for training and evaluation with extensive configuration options
Note: Add a GIF or video here showing the trained humanoid walking.
After training (~500k-1M steps), you should see the humanoid develop a stable walking gait, maintaining balance while moving forward.
- Python 3.8+
- OpenGL support (required by PyBullet for rendering)
# Clone the repository
git clone <repo-url>
cd humanoid_rl_walk
# Create virtual environment (recommended)
python -m venv venv
source venv/bin/activate # Linux/Mac
# or: venv\Scripts\activate # Windows
# Install dependencies
pip install -r requirements.txt| Package | Version | Purpose |
|---|---|---|
| pybullet | ≥3.2.5 | Physics simulation engine |
| stable-baselines3 | ≥2.0.0 | PPO algorithm implementation |
| gymnasium | ≥0.29.0 | RL environment interface |
| numpy | ≥1.24.0 | Numerical computations |
| matplotlib | ≥3.7.0 | Plotting training curves |
| tensorboard | ≥2.13.0 | Training visualization |
| torch | ≥2.0.0 | Neural network backend |
# Basic training (1M timesteps, 4 parallel environments)
python scripts/train_humanoid.py
# Quick test run
python scripts/train_humanoid.py --total-timesteps 50000 --n-envs 2# Run with visualization
python scripts/eval_humanoid.py --model-path ./models/humanoid_ppo_<timestamp>/best_model/model.zip --render
# Batch evaluation (10 episodes, no rendering)
python scripts/eval_humanoid.py --model-path ./models/humanoid_ppo_<timestamp>/final_model/model.zip --n-episodes 10You need a trained model first. Here's what to expect:
| Training Progress | Time (~4 envs) | What You'll See |
|---|---|---|
| 0 - 100k steps | ~5-10 min | Agent falling over randomly |
| 100k - 300k steps | ~15-30 min | Learning to balance/stand |
| 300k - 500k steps | ~30-60 min | Basic walking gait emerges |
| 500k - 1M+ steps | 1-2 hours | Stable, smooth walking |
To visualize: Run eval_humanoid.py with --render after training. Use --slow-motion 2.0 to slow it down.
The training script provides extensive configuration options:
python scripts/train_humanoid.py \
--total-timesteps 1000000 \
--n-envs 4 \
--learning-rate 3e-4 \
--batch-size 128 \
--n-steps 2048 \
--gamma 0.99 \
--save-path ./models \
--log-path ./logs \
--eval-freq 10000 \
--checkpoint-freq 100000| Option | Default | Description |
|---|---|---|
--total-timesteps |
1000000 | Total environment steps to train |
--n-envs |
4 | Number of parallel environments |
--learning-rate |
3e-4 | PPO learning rate |
--batch-size |
128 | Minibatch size |
--n-steps |
2048 | Steps per environment per update |
--n-epochs |
10 | PPO epochs per update |
--gamma |
0.99 | Discount factor |
--gae-lambda |
0.95 | GAE lambda parameter |
--clip-range |
0.2 | PPO clip range |
--ent-coef |
0.01 | Entropy coefficient |
--normalize |
False | Enable observation/reward normalization |
--resume |
None | Path to checkpoint to resume from |
--seed |
0 | Random seed |
tensorboard --logdir ./logsThen open http://localhost:6006 in your browser.
python scripts/eval_humanoid.py \
--model-path ./models/humanoid_ppo_<timestamp>/final_model/model.zip \
--vec-normalize ./models/humanoid_ppo_<timestamp>/final_model/vec_normalize.pkl \
--n-episodes 10 \
--render \
--slow-motion 1.5
⚠️ Important: If you trained with--normalize, you must provide the--vec-normalizeflag pointing to thevec_normalize.pklfile. Without it, the model will fail because observations won't be normalized correctly. Thefinal_model/folder contains bothmodel.zipandvec_normalize.pkl.
| Option | Default | Description |
|---|---|---|
--model-path |
(required) | Path to trained model .zip file |
--vec-normalize |
None | Path to VecNormalize stats .pkl |
--n-episodes |
10 | Number of evaluation episodes |
--render |
False | Enable PyBullet GUI visualization |
--slow-motion |
1.0 | Slow-motion factor (2.0 = half speed) |
--record |
None | Path to save video (e.g., ./eval.mp4) |
--deterministic |
True | Use deterministic actions |
--seed |
0 | Random seed |
python scripts/eval_humanoid.py \
--model-path ./models/best_model.zip \
--record ./videos/walking.mp4 \
--n-episodes 3 \
--fps 30import gymnasium as gym
from humanoid_rl.envs import HumanoidWalkEnv
from humanoid_rl.models.ppo_train import create_ppo_model, make_vec_env, train
# Create environment
env = HumanoidWalkEnv(render_mode="human")
# Basic interaction
obs, info = env.reset()
for _ in range(1000):
action = env.action_space.sample()
obs, reward, terminated, truncated, info = env.step(action)
if terminated or truncated:
obs, info = env.reset()
env.close()
# Training with PPO
vec_env = make_vec_env(HumanoidWalkEnv, n_envs=4)
model = create_ppo_model(vec_env, learning_rate=3e-4)
model = train(model, total_timesteps=100000)
model.save("humanoid_model")| Index | Component | Dimensions | Description |
|---|---|---|---|
| 0-7 | Joint Positions | 8 | Angles in radians: left_hip_yaw, left_hip_pitch, left_knee, left_ankle, right_hip_yaw, right_hip_pitch, right_knee, right_ankle |
| 8-15 | Joint Velocities | 8 | Angular velocities (rad/s) for each joint |
| 16-19 | Torso Orientation | 4 | Quaternion (x, y, z, w) |
| 20-22 | Torso Linear Velocity | 3 | (vx, vy, vz) in m/s |
| 23-25 | Torso Angular Velocity | 3 | (wx, wy, wz) in rad/s |
| 26-27 | Foot Contacts | 2 | Binary contact indicators (left, right) |
| 28 | Forward Velocity | 1 | X-component of torso velocity |
| Index | Joint | Range | Limits |
|---|---|---|---|
| 0 | Left Hip Yaw | [-1, 1] | -45° to +45° |
| 1 | Left Hip Pitch | [-1, 1] | -60° to +60° |
| 2 | Left Knee | [-1, 1] | -90° to 0° |
| 3 | Left Ankle | [-1, 1] | -30° to +30° |
| 4 | Right Hip Yaw | [-1, 1] | -45° to +45° |
| 5 | Right Hip Pitch | [-1, 1] | -60° to +60° |
| 6 | Right Knee | [-1, 1] | -90° to 0° |
| 7 | Right Ankle | [-1, 1] | -30° to +30° |
Actions are normalized to [-1, 1] and scaled internally to joint limits.
reward = forward_reward + upright_reward + energy_penalty + fall_penalty
where:
forward_reward = forward_reward_weight × forward_velocity
upright_reward = upright_reward_weight × upright_bonus (0 to 1)
energy_penalty = -energy_cost_weight × sum(actions²)
fall_penalty = -100 (if terminated due to fall)
| Component | Default Weight | Purpose |
|---|---|---|
| Forward Velocity | 1.0 | Encourage forward movement |
| Upright Bonus | 0.1 | Maintain vertical posture |
| Energy Cost | 0.01 | Penalize excessive actions |
| Fall Penalty | -100 | Large penalty for falling |
The episode terminates when:
- Torso height falls below 0.3 meters
- Torso tilt exceeds 60° (pitch or roll)
The episode truncates when:
- Maximum steps (default: 1000) is reached (~16.7 seconds at 60Hz)
┌─────────┐
│ TORSO │ ← 0.3m × 0.2m × 0.4m, 10kg
│ │
└────┬────┘
╱ │ ╲
(○) (○) ← Hip connectors (yaw rotation), 0.5kg each
│ │
┌───┴──┐ ┌──┴───┐
│UPPER │ │UPPER │ ← Capsule: r=0.05m, L=0.3m, 3kg each
│ LEG │ │ LEG │
└───┬──┘ └──┬───┘
│ │
┌───┴──┐ ┌──┴───┐
│LOWER │ │LOWER │ ← Capsule: r=0.04m, L=0.3m, 2kg each
│ LEG │ │ LEG │
└───┬──┘ └──┬───┘
│ │
┌───┴──┐ ┌──┴───┐
│ FOOT │ │ FOOT │ ← Box: 0.1m × 0.05m × 0.15m, 1kg each
└──────┘ └──────┘
| Body Part | Shape | Dimensions | Mass |
|---|---|---|---|
| Torso | Box | 0.3 × 0.2 × 0.4 m | 10 kg |
| Hip Connectors (×2) | Sphere | radius=0.04m | 0.5 kg each |
| Upper Legs (×2) | Capsule | radius=0.05m, length=0.3m | 3 kg each |
| Lower Legs (×2) | Capsule | radius=0.04m, length=0.3m | 2 kg each |
| Feet (×2) | Box | 0.1 × 0.05 × 0.15 m | 1 kg each |
| Total | 23 kg |
| Joint | Axis | Lower Limit | Upper Limit | Purpose |
|---|---|---|---|---|
| Hip Yaw (×2) | Yaw (Z) | -45° | +45° | Turning left/right |
| Hip Pitch (×2) | Pitch (Y) | -60° | +60° | Leg swing fwd/back |
| Knee (×2) | Pitch (Y) | -90° | 0° | Leg bending |
| Ankle (×2) | Pitch (Y) | -30° | +30° | Foot flexion |
| Phase | Timesteps | Behavior |
|---|---|---|
| Initial | 0 - 100k | Agent falls repeatedly, random exploration |
| Balancing | 100k - 300k | Learns to stand, tentative stepping |
| Walking | 300k - 500k | Develops basic walking gait |
| Refinement | 500k - 1M+ | Smoother, faster, more stable walking |
For stable training:
python scripts/train_humanoid.py \
--total-timesteps 2000000 \
--n-envs 8 \
--learning-rate 3e-4 \
--batch-size 128 \
--n-steps 2048 \
--normalize- Use observation normalization (
--normalize): Significantly improves learning stability - More parallel environments (
--n-envs 8): Better sample efficiency and exploration - Patient training: Walking emerges after ~300k-500k steps; don't stop too early
- Monitor TensorBoard: Watch for reward plateau or collapse
- Checkpoint frequently: Training can be unstable; save progress often
| Issue | Possible Cause | Solution |
|---|---|---|
| Agent never learns to stand | Learning rate too high | Reduce to 1e-4 |
| Agent stands but doesn't walk | Reward weights imbalanced | Increase forward_reward_weight |
| Training unstable/collapses | Observations not normalized | Add --normalize |
| Slow training | Too few environments | Increase --n-envs |
The basic environment (V1) trains the humanoid to walk straight forward. This often results in a "dragging" gait where one leg leads and the other follows.
HumanoidWalkEnvV2 addresses this by using a target-following task that encourages more human-like walking:
- Random Target Placement: A target (red sphere) is placed randomly around the humanoid
- Velocity Toward Target: Rewards moving toward the target, not just forward
- Target Reached: When the humanoid reaches the target, a new one spawns in a random direction
- Forced Turning: Random placement forces the humanoid to turn, using both legs more equally
- Gait Symmetry Rewards: Explicit reward for alternating left-right foot contacts
# Basic V2 training
python scripts/train_humanoid_v2.py --normalize --total-timesteps 1000000
# Adjust gait symmetry emphasis (higher = more symmetric walking)
python scripts/train_humanoid_v2.py --gait-symmetry-weight 0.2 --normalize
# Longer training for better results
python scripts/train_humanoid_v2.py --total-timesteps 2000000 --n-envs 8 --normalize| Parameter | Default | Description |
|---|---|---|
--target-reward-weight |
1.0 | Weight for velocity-toward-target reward |
--target-reach-bonus |
10.0 | Bonus points for reaching a target |
--gait-symmetry-weight |
0.1 | Weight for leg usage symmetry reward |
--alternation-bonus-weight |
0.05 | Bonus for proper L-R-L-R foot alternation |
--target-distance-min |
1.0 | Minimum target spawn distance (meters) |
--target-distance-max |
3.0 | Maximum target spawn distance (meters) |
V2 extends the observation space with target information:
| Index | Component | Dimensions | Description |
|---|---|---|---|
| 0-28 | Base observations | 29 | Same as V1 |
| 29-30 | Relative target | 2 | [dx, dy] to target in local frame |
| 31 | Target distance | 1 | Distance to target in meters |
- Hip yaw joints: Enable actual turning by rotating legs horizontally
- Turning requirement: Random targets force weight shifting and balanced leg use
- No "forward only" bias: Agent can't exploit always leading with one leg
- Symmetry rewards: Explicit incentive for alternating foot contacts
- More exploration: Varied directions prevent local optima
humanoid_rl_walk/
├── humanoid_rl/
│ ├── envs/
│ │ ├── humanoid_walk_env.py # V1: Basic forward walking
│ │ └── humanoid_walk_env_v2.py # V2: Target-following for human-like gait
│ ├── models/
│ │ ├── ppo_train.py # Training: create_ppo_model, make_vec_env, train
│ │ └── ppo_eval.py # Evaluation: load_model, evaluate_model
│ ├── sim/
│ │ ├── humanoid_model.py # PyBullet humanoid: HumanoidModel class
│ │ └── physics_utils.py # setup_physics, create_ground_plane
│ └── vis/
│ ├── live_viewer.py # LiveViewer, camera tracking
│ └── plots.py # plot_training_curves, TensorboardReader
└── scripts/
├── train_humanoid.py # CLI training (V1 environment)
├── train_humanoid_v2.py # CLI training (V2 target-following)
└── eval_humanoid.py # CLI evaluation
| Module | Purpose |
|---|---|
envs/humanoid_walk_env.py |
Gymnasium-compatible environment with observation/action spaces, reward, termination |
sim/humanoid_model.py |
Programmatic PyBullet humanoid construction with joints and collision shapes |
sim/physics_utils.py |
Physics engine setup, ground plane, center-of-mass utilities |
models/ppo_train.py |
PPO model creation, vectorized environments, training loop |
models/ppo_eval.py |
Model loading, batch evaluation, visual episodes, video recording |
vis/live_viewer.py |
Real-time camera tracking, debug overlays |
vis/plots.py |
Training curve visualization, TensorBoard log parsing |
python scripts/eval_humanoid.py --model-path <path> --renderFeatures:
- Real-time 3D visualization
- Interactive camera control
- Slow-motion playback with
--slow-motion 2.0
| Control | Action |
|---|---|
| Mouse wheel | Zoom in/out |
| Ctrl + Left mouse drag | Rotate the view (orbit around target) |
| Ctrl + Middle mouse drag | Pan the view (move camera sideways) |
| Alt + Left mouse drag | Alternative rotation |
On macOS, you may need to use Cmd instead of Ctrl. You can also use the sliders in the PyBullet GUI panel to adjust camera distance, yaw, and pitch.
tensorboard --logdir ./logsMetrics tracked:
- Episode reward (mean/min/max)
- Episode length
- Policy loss, value loss, entropy
- Learning rate schedule
- Custom metrics (forward velocity, fall rate)
python scripts/eval_humanoid.py --model-path <path> --record ./video.mp4 --fps 30from humanoid_rl.vis.plots import plot_training_curves, TensorboardReader
reader = TensorboardReader("./logs/humanoid_ppo_<timestamp>")
plot_training_curves(reader, save_path="./training_curves.png")- Python: 3.8 or higher
- Operating System: Linux, macOS, Windows
- OpenGL: Required for PyBullet rendering (GUI and rgb_array modes)
- GPU: CUDA recommended; CPU often faster than MPS for this project
| Platform | GPU | Device Flag | Notes |
|---|---|---|---|
| Windows/Linux | NVIDIA | --device cuda |
Recommended if available |
| macOS | Apple Silicon | --device mps |
Available but slower than CPU |
| Any | CPU | --device cpu |
Default, fast for MLP policies |
Why CPU is often faster: PPO with small MLP networks (like this project) is CPU-bound. The overhead of transferring data to/from GPU outweighs the compute benefit. CUDA GPUs are still faster due to better optimization, but MPS on Apple Silicon is typically slower than CPU for this workload.
# Default (auto-detects CUDA, otherwise uses CPU)
python scripts/train_humanoid.py
# Force specific device
python scripts/train_humanoid.py --device cuda # NVIDIA GPU (fastest)
python scripts/train_humanoid.py --device cpu # CPU (fast for MLP)
python scripts/train_humanoid.py --device mps # Apple Silicon (test only)
# Test your device performance
python scripts/test_gpu.pyPyBullet requires OpenGL for visualization. On headless servers, use:
render_mode=Nonefor training (no display needed)- Virtual framebuffer (xvfb) for video recording
# On headless Linux server
xvfb-run -a python scripts/eval_humanoid.py --model-path <path> --record video.mp4If you encounter a compilation error like 'fdopen' is a macro when installing PyBullet on macOS, use this workaround:
# Download PyBullet source to a temporary directory
pip download pybullet --no-binary :all: -d /tmp/pybullet_src
cd /tmp/pybullet_src
tar -xzf pybullet-*.tar.gz
cd pybullet-*
# Remove the conflicting fdopen macro definition
sed -i '' 's/#define fdopen.*NULL.*$//' examples/ThirdPartyLibs/zlib/zutil.h
# Build and install
pip install .Alternatively, you can try installing an older version that doesn't have this issue:
pip install pybullet==3.2.5source venv/bin/activate && pip download pybullet --no-binary :all: -d /tmp/pybulletsrc && cd /tmp/pybullet_src && tar -xzf pybullet-.tar.gz && cd pybullet-_ && sed -i '' 's/#define fdopen._NULL._$//' examples/ThirdPartyLibs/zlib/zutil.h && pip install . 2>&1 | tail -30
MIT License
- PyBullet - Physics simulation engine
- Stable-Baselines3 - RL algorithms
- Gymnasium - RL environment API
- OpenAI - Original Gym environment specification
If you use this project in your research, please cite:
@software{humanoid_rl_walk,
title = {Humanoid RL Walk: 3D Biped Walking with Reinforcement Learning},
year = {2024},
url = {https://github.com/your-username/humanoid_rl_walk}
}