Skip to content

About

No description, website, or topics provided.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Latest commit

 

History

4 Commits

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Humanoid RL Walk - 3D Biped Walking with Reinforcement Learning

A complete 3D humanoid physics simulation where an agent learns to walk using reinforcement learning. Built with PyBullet for physics simulation, Stable-Baselines3 for PPO training, and Gymnasium for the environment interface.

Key Features

  • Realistic Physics: PyBullet-based simulation with a 22kg bipedal humanoid (torso, upper/lower legs, feet)
  • PPO Training: Proximal Policy Optimization with parallel environments and observation normalization
  • 25-Dimensional Observations: Joint states, torso pose/velocity, foot contacts
  • Customizable Reward Function: Forward velocity, upright bonus, energy penalty, fall penalty
  • Visualization: Real-time PyBullet GUI, TensorBoard monitoring, video recording
  • Production Ready: CLI scripts for training and evaluation with extensive configuration options

Demo

Note: Add a GIF or video here showing the trained humanoid walking.

After training (~500k-1M steps), you should see the humanoid develop a stable walking gait, maintaining balance while moving forward.


Installation

Prerequisites

  • Python 3.8+
  • OpenGL support (required by PyBullet for rendering)

Setup

# Clone the repository
git clone <repo-url>
cd humanoid_rl_walk

# Create virtual environment (recommended)
python -m venv venv
source venv/bin/activate  # Linux/Mac
# or: venv\Scripts\activate  # Windows

# Install dependencies
pip install -r requirements.txt

Dependencies

Package Version Purpose
pybullet ≥3.2.5 Physics simulation engine
stable-baselines3 ≥2.0.0 PPO algorithm implementation
gymnasium ≥0.29.0 RL environment interface
numpy ≥1.24.0 Numerical computations
matplotlib ≥3.7.0 Plotting training curves
tensorboard ≥2.13.0 Training visualization
torch ≥2.0.0 Neural network backend

Quick Start

Train an Agent

# Basic training (1M timesteps, 4 parallel environments)
python scripts/train_humanoid.py

# Quick test run
python scripts/train_humanoid.py --total-timesteps 50000 --n-envs 2

Evaluate a Trained Agent

# Run with visualization
python scripts/eval_humanoid.py --model-path ./models/humanoid_ppo_<timestamp>/best_model/model.zip --render

# Batch evaluation (10 episodes, no rendering)
python scripts/eval_humanoid.py --model-path ./models/humanoid_ppo_<timestamp>/final_model/model.zip --n-episodes 10

When Can I See the Humanoid Walk?

You need a trained model first. Here's what to expect:

Training Progress Time (~4 envs) What You'll See
0 - 100k steps ~5-10 min Agent falling over randomly
100k - 300k steps ~15-30 min Learning to balance/stand
300k - 500k steps ~30-60 min Basic walking gait emerges
500k - 1M+ steps 1-2 hours Stable, smooth walking

To visualize: Run eval_humanoid.py with --render after training. Use --slow-motion 2.0 to slow it down.


Detailed Usage

Training

The training script provides extensive configuration options:

python scripts/train_humanoid.py \
    --total-timesteps 1000000 \
    --n-envs 4 \
    --learning-rate 3e-4 \
    --batch-size 128 \
    --n-steps 2048 \
    --gamma 0.99 \
    --save-path ./models \
    --log-path ./logs \
    --eval-freq 10000 \
    --checkpoint-freq 100000

Training Options

Option Default Description
--total-timesteps 1000000 Total environment steps to train
--n-envs 4 Number of parallel environments
--learning-rate 3e-4 PPO learning rate
--batch-size 128 Minibatch size
--n-steps 2048 Steps per environment per update
--n-epochs 10 PPO epochs per update
--gamma 0.99 Discount factor
--gae-lambda 0.95 GAE lambda parameter
--clip-range 0.2 PPO clip range
--ent-coef 0.01 Entropy coefficient
--normalize False Enable observation/reward normalization
--resume None Path to checkpoint to resume from
--seed 0 Random seed

Monitor Training with TensorBoard

tensorboard --logdir ./logs

Then open http://localhost:6006 in your browser.

Evaluation

python scripts/eval_humanoid.py \
    --model-path ./models/humanoid_ppo_<timestamp>/final_model/model.zip \
    --vec-normalize ./models/humanoid_ppo_<timestamp>/final_model/vec_normalize.pkl \
    --n-episodes 10 \
    --render \
    --slow-motion 1.5

⚠️ Important: If you trained with --normalize, you must provide the --vec-normalize flag pointing to the vec_normalize.pkl file. Without it, the model will fail because observations won't be normalized correctly. The final_model/ folder contains both model.zip and vec_normalize.pkl.

Evaluation Options

Option Default Description
--model-path (required) Path to trained model .zip file
--vec-normalize None Path to VecNormalize stats .pkl
--n-episodes 10 Number of evaluation episodes
--render False Enable PyBullet GUI visualization
--slow-motion 1.0 Slow-motion factor (2.0 = half speed)
--record None Path to save video (e.g., ./eval.mp4)
--deterministic True Use deterministic actions
--seed 0 Random seed

Record Video

python scripts/eval_humanoid.py \
    --model-path ./models/best_model.zip \
    --record ./videos/walking.mp4 \
    --n-episodes 3 \
    --fps 30

Using as a Library

import gymnasium as gym
from humanoid_rl.envs import HumanoidWalkEnv
from humanoid_rl.models.ppo_train import create_ppo_model, make_vec_env, train

# Create environment
env = HumanoidWalkEnv(render_mode="human")

# Basic interaction
obs, info = env.reset()
for _ in range(1000):
    action = env.action_space.sample()
    obs, reward, terminated, truncated, info = env.step(action)
    if terminated or truncated:
        obs, info = env.reset()
env.close()

# Training with PPO
vec_env = make_vec_env(HumanoidWalkEnv, n_envs=4)
model = create_ppo_model(vec_env, learning_rate=3e-4)
model = train(model, total_timesteps=100000)
model.save("humanoid_model")

Environment Details

Observation Space (29 dimensions)

Index Component Dimensions Description
0-7 Joint Positions 8 Angles in radians: left_hip_yaw, left_hip_pitch, left_knee, left_ankle, right_hip_yaw, right_hip_pitch, right_knee, right_ankle
8-15 Joint Velocities 8 Angular velocities (rad/s) for each joint
16-19 Torso Orientation 4 Quaternion (x, y, z, w)
20-22 Torso Linear Velocity 3 (vx, vy, vz) in m/s
23-25 Torso Angular Velocity 3 (wx, wy, wz) in rad/s
26-27 Foot Contacts 2 Binary contact indicators (left, right)
28 Forward Velocity 1 X-component of torso velocity

Action Space (8 dimensions)

Index Joint Range Limits
0 Left Hip Yaw [-1, 1] -45° to +45°
1 Left Hip Pitch [-1, 1] -60° to +60°
2 Left Knee [-1, 1] -90° to 0°
3 Left Ankle [-1, 1] -30° to +30°
4 Right Hip Yaw [-1, 1] -45° to +45°
5 Right Hip Pitch [-1, 1] -60° to +60°
6 Right Knee [-1, 1] -90° to 0°
7 Right Ankle [-1, 1] -30° to +30°

Actions are normalized to [-1, 1] and scaled internally to joint limits.

Reward Function

reward = forward_reward + upright_reward + energy_penalty + fall_penalty

where:
  forward_reward = forward_reward_weight × forward_velocity
  upright_reward = upright_reward_weight × upright_bonus (0 to 1)
  energy_penalty = -energy_cost_weight × sum(actions²)
  fall_penalty   = -100 (if terminated due to fall)
Component Default Weight Purpose
Forward Velocity 1.0 Encourage forward movement
Upright Bonus 0.1 Maintain vertical posture
Energy Cost 0.01 Penalize excessive actions
Fall Penalty -100 Large penalty for falling

Termination Conditions

The episode terminates when:

  • Torso height falls below 0.3 meters
  • Torso tilt exceeds 60° (pitch or roll)

The episode truncates when:

  • Maximum steps (default: 1000) is reached (~16.7 seconds at 60Hz)

Humanoid Model

Physical Structure

          ┌─────────┐
          │  TORSO  │  ← 0.3m × 0.2m × 0.4m, 10kg
          │         │
          └────┬────┘
         ╱     │     ╲
       (○)           (○)  ← Hip connectors (yaw rotation), 0.5kg each
        │             │
    ┌───┴──┐       ┌──┴───┐
    │UPPER │       │UPPER │  ← Capsule: r=0.05m, L=0.3m, 3kg each
    │ LEG  │       │ LEG  │
    └───┬──┘       └──┬───┘
        │             │
    ┌───┴──┐       ┌──┴───┐
    │LOWER │       │LOWER │  ← Capsule: r=0.04m, L=0.3m, 2kg each
    │ LEG  │       │ LEG  │
    └───┬──┘       └──┬───┘
        │             │
    ┌───┴──┐       ┌──┴───┐
    │ FOOT │       │ FOOT │  ← Box: 0.1m × 0.05m × 0.15m, 1kg each
    └──────┘       └──────┘

Specifications

Body Part Shape Dimensions Mass
Torso Box 0.3 × 0.2 × 0.4 m 10 kg
Hip Connectors (×2) Sphere radius=0.04m 0.5 kg each
Upper Legs (×2) Capsule radius=0.05m, length=0.3m 3 kg each
Lower Legs (×2) Capsule radius=0.04m, length=0.3m 2 kg each
Feet (×2) Box 0.1 × 0.05 × 0.15 m 1 kg each
Total 23 kg

Joint Configuration

Joint Axis Lower Limit Upper Limit Purpose
Hip Yaw (×2) Yaw (Z) -45° +45° Turning left/right
Hip Pitch (×2) Pitch (Y) -60° +60° Leg swing fwd/back
Knee (×2) Pitch (Y) -90° 0° Leg bending
Ankle (×2) Pitch (Y) -30° +30° Foot flexion

Training Tips & Expected Behavior

Training Phases

Phase Timesteps Behavior
Initial 0 - 100k Agent falls repeatedly, random exploration
Balancing 100k - 300k Learns to stand, tentative stepping
Walking 300k - 500k Develops basic walking gait
Refinement 500k - 1M+ Smoother, faster, more stable walking

Recommended Hyperparameters

For stable training:

python scripts/train_humanoid.py \
    --total-timesteps 2000000 \
    --n-envs 8 \
    --learning-rate 3e-4 \
    --batch-size 128 \
    --n-steps 2048 \
    --normalize

Tips

  1. Use observation normalization (--normalize): Significantly improves learning stability
  2. More parallel environments (--n-envs 8): Better sample efficiency and exploration
  3. Patient training: Walking emerges after ~300k-500k steps; don't stop too early
  4. Monitor TensorBoard: Watch for reward plateau or collapse
  5. Checkpoint frequently: Training can be unstable; save progress often

Common Issues

Issue Possible Cause Solution
Agent never learns to stand Learning rate too high Reduce to 1e-4
Agent stands but doesn't walk Reward weights imbalanced Increase forward_reward_weight
Training unstable/collapses Observations not normalized Add --normalize
Slow training Too few environments Increase --n-envs

V2 Environment: Human-Like Walking with Target-Following

The basic environment (V1) trains the humanoid to walk straight forward. This often results in a "dragging" gait where one leg leads and the other follows.

HumanoidWalkEnvV2 addresses this by using a target-following task that encourages more human-like walking:

How It Works

  1. Random Target Placement: A target (red sphere) is placed randomly around the humanoid
  2. Velocity Toward Target: Rewards moving toward the target, not just forward
  3. Target Reached: When the humanoid reaches the target, a new one spawns in a random direction
  4. Forced Turning: Random placement forces the humanoid to turn, using both legs more equally
  5. Gait Symmetry Rewards: Explicit reward for alternating left-right foot contacts

Training with V2

# Basic V2 training
python scripts/train_humanoid_v2.py --normalize --total-timesteps 1000000

# Adjust gait symmetry emphasis (higher = more symmetric walking)
python scripts/train_humanoid_v2.py --gait-symmetry-weight 0.2 --normalize

# Longer training for better results
python scripts/train_humanoid_v2.py --total-timesteps 2000000 --n-envs 8 --normalize

V2-Specific Parameters

Parameter Default Description
--target-reward-weight 1.0 Weight for velocity-toward-target reward
--target-reach-bonus 10.0 Bonus points for reaching a target
--gait-symmetry-weight 0.1 Weight for leg usage symmetry reward
--alternation-bonus-weight 0.05 Bonus for proper L-R-L-R foot alternation
--target-distance-min 1.0 Minimum target spawn distance (meters)
--target-distance-max 3.0 Maximum target spawn distance (meters)

V2 Observation Space (32 dimensions)

V2 extends the observation space with target information:

Index Component Dimensions Description
0-28 Base observations 29 Same as V1
29-30 Relative target 2 [dx, dy] to target in local frame
31 Target distance 1 Distance to target in meters

Why V2 Produces Better Gaits

  • Hip yaw joints: Enable actual turning by rotating legs horizontally
  • Turning requirement: Random targets force weight shifting and balanced leg use
  • No "forward only" bias: Agent can't exploit always leading with one leg
  • Symmetry rewards: Explicit incentive for alternating foot contacts
  • More exploration: Varied directions prevent local optima

Project Architecture

humanoid_rl_walk/
├── humanoid_rl/
│   ├── envs/
│   │   ├── humanoid_walk_env.py      # V1: Basic forward walking
│   │   └── humanoid_walk_env_v2.py   # V2: Target-following for human-like gait
│   ├── models/
│   │   ├── ppo_train.py              # Training: create_ppo_model, make_vec_env, train
│   │   └── ppo_eval.py               # Evaluation: load_model, evaluate_model
│   ├── sim/
│   │   ├── humanoid_model.py         # PyBullet humanoid: HumanoidModel class
│   │   └── physics_utils.py          # setup_physics, create_ground_plane
│   └── vis/
│       ├── live_viewer.py            # LiveViewer, camera tracking
│       └── plots.py                  # plot_training_curves, TensorboardReader
└── scripts/
    ├── train_humanoid.py             # CLI training (V1 environment)
    ├── train_humanoid_v2.py          # CLI training (V2 target-following)
    └── eval_humanoid.py              # CLI evaluation

Module Responsibilities

Module Purpose
envs/humanoid_walk_env.py Gymnasium-compatible environment with observation/action spaces, reward, termination
sim/humanoid_model.py Programmatic PyBullet humanoid construction with joints and collision shapes
sim/physics_utils.py Physics engine setup, ground plane, center-of-mass utilities
models/ppo_train.py PPO model creation, vectorized environments, training loop
models/ppo_eval.py Model loading, batch evaluation, visual episodes, video recording
vis/live_viewer.py Real-time camera tracking, debug overlays
vis/plots.py Training curve visualization, TensorBoard log parsing

Visualization Options

PyBullet GUI (Real-time)

python scripts/eval_humanoid.py --model-path <path> --render

Features:

  • Real-time 3D visualization
  • Interactive camera control
  • Slow-motion playback with --slow-motion 2.0

Camera Controls

Control Action
Mouse wheel Zoom in/out
Ctrl + Left mouse drag Rotate the view (orbit around target)
Ctrl + Middle mouse drag Pan the view (move camera sideways)
Alt + Left mouse drag Alternative rotation

On macOS, you may need to use Cmd instead of Ctrl. You can also use the sliders in the PyBullet GUI panel to adjust camera distance, yaw, and pitch.

TensorBoard Monitoring

tensorboard --logdir ./logs

Metrics tracked:

  • Episode reward (mean/min/max)
  • Episode length
  • Policy loss, value loss, entropy
  • Learning rate schedule
  • Custom metrics (forward velocity, fall rate)

Video Recording

python scripts/eval_humanoid.py --model-path <path> --record ./video.mp4 --fps 30

Plotting Training Curves

from humanoid_rl.vis.plots import plot_training_curves, TensorboardReader

reader = TensorboardReader("./logs/humanoid_ppo_<timestamp>")
plot_training_curves(reader, save_path="./training_curves.png")

Requirements

  • Python: 3.8 or higher
  • Operating System: Linux, macOS, Windows
  • OpenGL: Required for PyBullet rendering (GUI and rgb_array modes)
  • GPU: CUDA recommended; CPU often faster than MPS for this project

GPU Acceleration

Platform GPU Device Flag Notes
Windows/Linux NVIDIA --device cuda Recommended if available
macOS Apple Silicon --device mps Available but slower than CPU
Any CPU --device cpu Default, fast for MLP policies

Why CPU is often faster: PPO with small MLP networks (like this project) is CPU-bound. The overhead of transferring data to/from GPU outweighs the compute benefit. CUDA GPUs are still faster due to better optimization, but MPS on Apple Silicon is typically slower than CPU for this workload.

# Default (auto-detects CUDA, otherwise uses CPU)
python scripts/train_humanoid.py

# Force specific device
python scripts/train_humanoid.py --device cuda    # NVIDIA GPU (fastest)
python scripts/train_humanoid.py --device cpu     # CPU (fast for MLP)
python scripts/train_humanoid.py --device mps     # Apple Silicon (test only)

# Test your device performance
python scripts/test_gpu.py

Note on PyBullet

PyBullet requires OpenGL for visualization. On headless servers, use:

  • render_mode=None for training (no display needed)
  • Virtual framebuffer (xvfb) for video recording
# On headless Linux server
xvfb-run -a python scripts/eval_humanoid.py --model-path <path> --record video.mp4

Troubleshooting

PyBullet Installation on macOS (SDK 26+)

If you encounter a compilation error like 'fdopen' is a macro when installing PyBullet on macOS, use this workaround:

# Download PyBullet source to a temporary directory
pip download pybullet --no-binary :all: -d /tmp/pybullet_src
cd /tmp/pybullet_src
tar -xzf pybullet-*.tar.gz
cd pybullet-*

# Remove the conflicting fdopen macro definition
sed -i '' 's/#define fdopen.*NULL.*$//' examples/ThirdPartyLibs/zlib/zutil.h

# Build and install
pip install .

Alternatively, you can try installing an older version that doesn't have this issue:

pip install pybullet==3.2.5

Workaround for macOS PyBullet Build Issue

source venv/bin/activate && pip download pybullet --no-binary :all: -d /tmp/pybulletsrc && cd /tmp/pybullet_src && tar -xzf pybullet-.tar.gz && cd pybullet-_ && sed -i '' 's/#define fdopen._NULL._$//' examples/ThirdPartyLibs/zlib/zutil.h && pip install . 2>&1 | tail -30



License

MIT License


Acknowledgments


Citation

If you use this project in your research, please cite:

@software{humanoid_rl_walk,
  title = {Humanoid RL Walk: 3D Biped Walking with Reinforcement Learning},
  year = {2024},
  url = {https://github.com/your-username/humanoid_rl_walk}
}

About

No description, website, or topics provided.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages