[Bug Report] CollisionPipeline: NaN solver divergence and robot-terrain penetration on rough terrain

[rafaelcathomen]

Describe the bug

When using Newton's CollisionPipeline (use_mujoco_contacts=False) with rough terrain (the upstream ROUGH_TERRAINS_CFG), the MJWarp solver produces NaN values in physics state (joint positions, velocities, root state) after a few training iterations. With visualization at low env count, the robot can be seen getting stuck inside terrain geometry:

anymal_rough_newton_bug.mp4

This is separate from the nconmax spurious contact force issue (fixed by camevor/newton:fix-solvermujoco-contact-init).

Steps to reproduce

1. Setup

  # IsaacLab develop branch
  git checkout develop  # commit e69dbf41ef7

  # Newton: install from fork with contact init fix
  git clone git@github.com:newton-physics/newton.git
  git remote add camevor git@github.com:camevor/newton.git
  git fetch camevor
  git checkout camevor/fix-solvermujoco-contact-init  # commit b16bcc88

  # Install
  cd IsaacLab
  ./isaaclab.sh -p -m pip uninstall newton -y
  ./isaaclab.sh -p -m pip install -e ../newton

2. Modify flat_env_cfg.py by replacing with flat_env_cfg.py, which includes:

• rough terrain import
• updated solver config
• increased spawn height
• increased max_triangle_pairs

3. Run

  # 4096 envs:  NaN after ~6 iterations
  python scripts/reinforcement_learning/rsl_rl/train.py --task Isaac-Velocity-Flat-Anymal-C-v0 presets=newton --num_envs 4096

  # 16 envs with viewer — robot visibly penetrates terrain
  python scripts/reinforcement_learning/rsl_rl/train.py --task Isaac-Velocity-Flat-Anymal-C-v0 presets=newton --num_envs 16
  --viz newton

The error I got is:

Traceback (most recent call last):
  File "/home/rafael/Projects/IsaacLab/source/isaaclab_tasks/isaaclab_tasks/utils/sim_launcher.py", line 223, in launch_simulation
    yield
  File "/home/rafael/Projects/IsaacLab/scripts/reinforcement_learning/rsl_rl/train.py", line 205, in main
    runner.learn(num_learning_iterations=agent_cfg.max_iterations, init_at_random_ep_len=True)
  File "/home/rafael/miniconda3/envs/isaaclab_newton_dev/lib/python3.11/site-packages/rsl_rl/runners/on_policy_runner.py", line 90, in learn
    check_nan(obs, rewards, dones)
  File "/home/rafael/miniconda3/envs/isaaclab_newton_dev/lib/python3.11/site-packages/rsl_rl/utils/utils.py", line 279, in check_nan
    raise ValueError(
ValueError: The observation group 'policy' returned by the environment contains NaN values. This usually indicates a bug in the environment's step() or reset() function.
Error executing job with overrides: []
Traceback (most recent call last):
  File "/home/rafael/Projects/IsaacLab/source/isaaclab_tasks/isaaclab_tasks/utils/hydra.py", line 238, in hydra_main
    func(env_cfg, agent_cfg, *args, **kwargs)
  File "/home/rafael/Projects/IsaacLab/scripts/reinforcement_learning/rsl_rl/train.py", line 205, in main
    runner.learn(num_learning_iterations=agent_cfg.max_iterations, init_at_random_ep_len=True)
  File "/home/rafael/miniconda3/envs/isaaclab_newton_dev/lib/python3.11/site-packages/rsl_rl/runners/on_policy_runner.py", line 90, in learn
    check_nan(obs, rewards, dones)
  File "/home/rafael/miniconda3/envs/isaaclab_newton_dev/lib/python3.11/site-packages/rsl_rl/utils/utils.py", line 279, in check_nan
    raise ValueError(
ValueError: The observation group 'policy' returned by the environment contains NaN values. This usually indicates a bug in the environment's step() or reset() function.

System Info

• Commit: e69dbf4 (IsaacLab develop)

• Newton: b16bcc88 (camevor/newton:fix-solvermujoco-contact-init)

• Isaac Sim Version: N/A (Newton-only)

• OS: Ubuntu (Linux 6.17.0)

• GPU: NVIDIA GeForce RTX 4090

• CUDA: 13

• Driver Version: 580.126.09

• I have checked that there is no similar issue in the repo (required)

• I have checked that the issue is not in running Isaac Sim itself and is related to the repo

Acceptance Criteria

• Anymal can train on rough terrain without nan values

[AntoineRichard]

@preist-nvidia @camevor for viz

[preist-nvidia]

@camevor can you please have a first look and/or work with @rafaelcathomen on a repro in Newton?

[camevor]

The terrain mesh has many intersecting components, and the robot feet are getting stuck in-between such layers. A newton repro is available at camevor:repro-mesh-collision.