FastSAC v1: off-policy SAC drop-in for PPO - #1
Open
romesco wants to merge 9 commits into
Open
Conversation
Newer IsaacLab (Newton/warp backend) exposes soft_joint_pos_limits and default_joint_pos as warp arrays, not torch tensors; the ported _compute_action_scaling indexed them torch-style and raised IndexError. Convert via wp.to_torch when needed. Found during the 128-env smoke.
…eatment) The env-side pre-reset obs stash is not always active (e.g. when isaaclab core loads from an editable install that lacks it). Rather than raise or bootstrap from the post-reset observation, degrade safely to terminal treatment and warn once. Correct bootstrapping still applies whenever time_outs_obs is present.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Off-policy SAC for rsl_rl (FastSAC v1)
Adds a SAC off-policy algorithm as a drop-in alternative to PPO for massively-parallel IsaacLab locomotion, targeting the rsl_rl 5.2.0 model API. Ported math from the
rsl_rl_saclineage; integration rewritten for the currentMLPModel/distribution_cfgAPI and grouped (dict) observations.What's here (9 commits, clean stack on
feature/locomotion)storage/replay_buffer.py): per-env circular TensorDict buffer with n-step return assembly.models/sac_mlp_model.py): tanh-Gaussian actor + twin-Q critic on the 5.2.0MLPModel. Overridesoutput_std/output_entropyfor torchNormal(.stddev/.entropy()), which the 5.2.0 base assumes as attributes.algorithms/sac.py): clipped-double-Q via aq_aggregation(min/avg) seam, auto-alpha entropy tuning, n-step returns, gamma-consistency guard, warp→torch conversion in action scaling, and a safe fallback (warn + terminal treatment) whentime_outs_obsis absent.runners/off_policy_runner.py): 5.2.0Loggerintegration; version bump to 5.3.0.Correctness-critical design
time_outs_obs(true pre-reset terminal obs) so truncations bootstrap and terminations don't.bootstrap_mask = bootstrap + 1 - dones.obs_groupsrouting (actor vs critic); >2D obs-leaf mask broadcast handled for height-scan groups.Testing
Companion IsaacLab integration PR: UW-Lab/IsaacLab (branch
rosario/fastsac-v1).