Repository navigation
Conversation
This was referenced Sep 24, 2026
Contributor
Author
This was referenced Sep 24, 2026
The pi0.5 converters stop doing rotation math. Actions now reach the algo
already normalized and already in their model-native layout, so to32_norm_6d /
from32_norm_6d are a pure rearrange into (and out of) openpi's 32-slot block
layout: RobotBimanual's native 20D [L xyz 6d g | R xyz 6d g] IS that layout and
only needs padding, HumanBimanual's 18D inserts a zero gripper slot per arm.
BaseActionConverter raises on both, so an unregistered embodiment fails loudly.
The four single-arm converters (RobotLeft/Right, HumanLeft/Right) are deleted
along with PI._extract_xyz, which existed for the cam-frame MSE. PI's loss
reduction and sample post-processing move into _reduce_loss and
_postprocess_sampled_actions.
The ypr->6D conversion is now a data transform. Eva gains cartesian_6d and
cartesian_wristframe_6d, Human the same pair, each 6D-encoding the action AND
the proprio ee_pose: normalized YPR saturates yaw/roll at +-pi (wraparound), so
per-dim normalization is only meaningful on the continuous representation.
get_revert_transform_list / the _build_*_revert_6d{,_wristframe}_ lists map a
prediction batch back to cam-frame ypr for the viz overlay.
Eva defaults to cartesian_wristframe_6d (eva, cotrain_pi_base, industry_eva_pi;
cotrain_pi_lang_wrist now composes the same as cotrain_pi_lang). Human data
configs move to cartesian_6d with pad_proprio_gripper, which pads the proprio
ee_pose 18 -> 20 (zero grip slots at 9/19) so the pi0.5 State: prompt bins line
up positionally with the robot 20D layout; UnpadGripperZeros inverts it in the
revert lists. HPT dims follow: eva 14 -> 20, human 12 -> proprio 20 / action 18.
With the 6D modes in place, a discontinuous rotation stops being a legal
training target. get_transform_list raises on the cartesian ypr modes and on
Eva's cartesian_wristframe_quat unless the caller passes an explicit
allow_legacy_rotation=True, which only the data/rollout boundary does (the
viz tutorial, rollout.py's legacy-checkpoint path, and the fixtures that use a
cartesian pipeline as scaffolding). test_data_configs_compose asserts that no
shipped data config selects such a mode or sets the opt-out, so the guard
cannot be bypassed from a config. The five *_pickplace_qwen* data configs were
the only ones still on Euler rotations; they and the two hpt_bc_pickplace_qwen
model configs paired with them are deleted. The keypoint modes keep their ypr
and quat variants for now -- the keypoints_*_6d modes do not exist yet.
Also here, because they are the same widening: SplitKeys rejects a layout whose
widths do not sum to the input (an evaluator revert list built for 12-dim ypr
fed an 18/20-dim 6D batch used to slice silently), and vendor-tagged human
embodiment names (MECKA_/SCALE_/ARIA_/LIGHTWHEEL_*) alias onto HUMAN_* through
canonical_embodiment_name, so pins and embodiment checks accept the legacy
metadata. Robot (EVA_*) names are never aliased.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VBmhpVy7ZcMtsBJWh38cJp
… zarrs compute_hand_pose_xyzquat built the palm normal as cross(thumb_dir, pinky_dir). Thumb and pinky are anatomically swapped between hands, so that cross product already mirrors chirality for free; applying rot_left on top of it flipped x/y a second time and landed the LEFT hand on the RIGHT hand's spatial convention. Both hands now use the same rot_right relabel. Reconverting every affected zarr would be the obvious fix, so this avoids it: rot_left == rot_right @ diag(-1, -1, 1), which makes right-multiplying the stored left pose by Rz(180 deg) exactly equivalent to reconverting with rot_right. RotateLocalFrame does that right-multiply (R_new = R_old @ R_fix), relabelling the pose's own axes without moving its origin, on (7,) poses and (T, 7) chunks alike, and leaves zero-norm quaternions alone because those mark padded frames. Human.get_transform_list(fix_left_wrist_convention=True) prepends it to the raw LEFT pose keys before any frame math: ee_pose in the cartesian modes, wrist_pose in the keypoints modes, since the converter wrote the same rotation into both. The flag is a property of the DATA, not of the vendor: measured against the production set (1833 of 1838 episodes probed, 2026-09-18), every mecka episode AND all 20 scale episodes still carry the double-mirrored left wrist, with the right-hand control clean, so data/mecka.yaml and data/scale.yaml both opt in. Data reconverted after this commit must not, or it double-flips. Because that correctness lives in the data, left_wrist_convention() re-derives the verdict from an episode's own keypoints -- recomputing both wrists under each relabel and comparing with the stored quaternions, with the RIGHT hand as the control -- and S3EpisodeResolver samples a few resolved episodes at startup. Only the double-flip raises: the flag on against post-fix data would correct what the converter already corrected, which is a fresh error from a stale setting. The flag off against pre-fix data is what every run did before this commit, and the generic human configs (aria, human, the cotrains) have never set it, so that warns and names the knob rather than turning their startup into a crash. Five legacy 2025 episodes match neither convention (a third converter wrote them); their control fails, so they warn too. data/mecka_scale_cotrain_pi.yaml and data/industry_eva_pi.yaml are deleted: both merge mecka and scale into ONE human_bimanual dataset under a single transform_list, and the correction is per-lab, so neither file has a correct value for the flag. model/pi0.5_cotrain_mecka_scale.yaml goes with them. test_wrist6d_roundtrip runs the real transform lists on synthetic world-frame poses, parametrized over the flag, and compares every stage against plain numpy/scipy SE(3) math that shares no code with the pipeline; the convention checker is tested the same way, on synthetic keypoints. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VBmhpVy7ZcMtsBJWh38cJp
…an default
Human data trains on hand keypoints instead of a palm-origin cartesian pose.
The action is 144-D wrist-first, per hand
[wrist xyz (3) | wrist rot6d (6) | 21 MANO keypoints (63)] x {left, right},
built by the keypoints_{wrist,head}frame_6d modes: the existing ypr keypoint
builders produce the 138-D layout and KeypointsYPRToRot6D re-encodes just the
wrist rotation, leaving the keypoint blocks untouched. Both the action and the
observations.state.keypoints proprio are converted, so neither carries a
wrapping euler angle. MANO keypoint 0 IS the wrist, so per hand the wrist
position rides at both ch 0-2 and ch 9-11; that is kept deliberately (21
keypoints is the MANO interface every consumer reshapes to) and written down
in the bimanual_keypoint_layout docstring.
include_ee_pose additionally builds the cartesian head-frame ee_pose next to
the keypoint action (keymap + transform list), so a keypoint pi0.5 run
discretizes the same State: prompt block as the cartesian runs and differs
from them only in the action space. The cartesian builder runs with
delete_target_world=False to keep obs_head_pose alive for the keypoint
builder, and its actions_cartesian by-product is dropped.
144 does not fit openpi's action vector, whose PyTorch port hard-codes 32-wide
action projections regardless of Pi0Config.action_dim. PI now reads action_dim
and, when it is not 32, swaps in action_dim-wide action_in_proj /
action_out_proj AFTER the strict base-weight load: only those two layers start
fresh, everything else comes from the checkpoint, and sample_actions already
draws noise at config.action_dim. pad_to_width zero-pads narrower embodiments
(eva's 20-D) into the same vector and raises rather than truncating a wider
one. _reduce_loss then restricts the mean to the slots the converter actually
packs, so eva does not average its loss over 112 zero-padded dims and
contribute a fraction of the gradient signal it had at width 32.
HumanBimanualKeypoints is identity-in, slice-out.
HPT human configs move to a 144-D state_keypoints stem. hpt_bc_flow_human
holds the body -- there is no vendor notion at the embodiment level, so the
canonical config is the embodiment one -- and hpt_bc_flow_{mecka,aria,scale}
are two-line aliases of it. HPTModel.add_stem drops null stem entries so a
child config can delete a stem its base declares, which hydra merges cannot
express, and hpt_cotrain_flow_seperate_head uses it to drop state_ee_pose.
With the keypoints_*_6d modes in place, the guard from the previous commit
widens to the keypoints ypr and quat modes, so every remaining discontinuous
rotation target needs an explicit allow_legacy_rotation=True (the aria->MANO
and mecka keypoint viz tutorials, which read pre-6D data). The cartesian
configs that were still pinned to the ypr widths move with it:
hpt_bc_flow_human_cartesian and the cotrain base to human 20/18, the
shared-head cotrains to a 20-wide shared action.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VBmhpVy7ZcMtsBJWh38cJp
Wrist-relative actions made a latent division safe-looking: the t=0 cell of a wrist-frame chunk is exactly the identity pose, so its quantile range is 0 and the historical `(q99 - q1) + 1e-6` denominator scaled any off-convention value by 1e6. A channel whose range is below NORM_MIN_RANGE (1e-4) is now treated as constant -- it normalizes to 0 and unnormalizes to its centre rather than dividing by epsilon. Diffusion Policy / robomimic ignore ranges under 1e-4 and GR00T maps min == max to 0; this is the same rule, applied in the shared _apply_norm_one / _apply_unnorm_one so the dataset and the tests agree. norm_stats.pool_horizon (default false, so nothing changes unless asked) computes one set of stats per channel over every timestep of the chunk, the way openpi does, instead of one per (timestep, channel) cell. The pooled stats are tiled back to the (T, D) shape, so normalize, the bounds check and checkpoints see the same layout either way. It is part of the norm-stat cache key, so flipping it cannot silently reuse stats computed the other way. precompute_norm_stats.py computes and caches the stats for a recipe outside a training run, keyed by the same file + content hash. Which of the two is better is unsettled: the literature is split (LBM per-timestep with a stated rationale, openpi / RDT / GR00T / Diffusion Policy / UMI / lerobot / Octo pooled) and nothing here has compared them on a real recipe, so the default stays the inherited per-timestep one and the config comment says what would make it worth flipping. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VBmhpVy7ZcMtsBJWh38cJp
A 6D rotation is two columns of a rotation matrix: already bounded in [-1, 1], and per-channel centre-and-scale pulls the pair off the manifold without buying any resolution. The two projects that anchor an action chunk at near-identity the way this repo does -- TRI's LBM and UMI -- both exclude rotation from normalization for exactly that reason (LBM: "We avoid this normalization procedure for the 6D rotation in the poses to avoid corrupting the rotation matrix"; UMI sends its rotation channels through an identity normalizer). Per timestep AND normalized was the one combination with no precedent, and it is what manufactures the t=0 zero-range rotation cells NORM_MIN_RANGE exists to catch -- this removes them at the source. rot6d_channels() reads the indices off the existing layout tables, so the rule follows the layout rather than a second hardcoded list: the cartesian 18/20 widths and the 144-D keypoint layout, action and proprio alike. The ypr widths keep the old behaviour; an Euler angle is not a matrix column and those modes are legacy-only now. _apply_norm_one / _apply_unnorm_one take the channels and pass them through untouched, so the dataset path, the evaluator revert and the rollout revert all follow from one place, and the degenerate-channel guard is left to do what it is actually good at: the structural kp0 / kp9 position zeros. Norm-stat files stay valid -- what changes is how they are applied -- but a run before this commit and a run after it are not comparable, and the loss is now weighted differently between translation (unit scale) and rotation (raw). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VBmhpVy7ZcMtsBJWh38cJp
Three independent ways MultiDataset could fail a whole run on data that was
fine, or on data it should have rejected up front.
Bounds check. Quantile bounds on a rotation channel are meaningless: euler ypr
wraps at +-pi and rot6d columns are matrix entries, so neither respects a
quantile interval. _bounds_check_channels uses the layout tables to check only
translation, gripper and keypoint channels; unrecognized widths still get the
full-vector check, and the NaN/Inf check always covers everything. The bounds
themselves get slack -- 1e-6 absolute plus half of each cell's own quantile
range -- because the check is any-cell over ~13 200 cells per keypoint chunk.
Measured on the 2815-frame flagship episode through the real
keypoints_wristframe_6d pipeline (zscore, stats from sample_frac 0.10 over its
own frames, 403 probe frames, any-key rejection): 39.5 % of frames rejected at
slack 0, 17.9 % at 0.05, 11.4 % at 0.1, 5.2 % at 0.25, 3.2 % at 0.5, 1.0 % at
1.0. The 132-channel action chunk drives all of it. Half a range is the knee,
not a cushion -- though the absolute numbers move with which frames the stats
were drawn from, so the comment says how to reproduce them.
Corrupt values sit orders of magnitude outside the range and are still caught.
Cells the normalizer treats as constant are exempt, since a tiny offset there
cannot blow up the loss -- and which stat decides that is norm_mode's business
(std under zscore, the quantile range under quantile), so the exemption asks
_degenerate instead of hardcoding q99 - q1; every shipped config is zscore, and
a channel that is 0 in 99.6 % of samples and 1.0 in the rest has q99 - q1 = 0
but std = 0.063, i.e. it is still divided by and must still be checked.
quantile_0_01 / quantile_99_99 now snap to observed samples ("lower" /
"higher") instead of interpolating: under 10k samples the linear percentile
lands strictly inside the sample range, so the extreme frames of the very
episodes the stats came from were rejected (3-16 % of a single episode in the
overfit runs). A stats/sample shape mismatch is logged once per key instead of
silently disabling the check.
Sampler. Retries stayed inside the failing episode until they exhausted it,
so one wholly-bad episode killed the run and NCCL-timed-out every other rank.
Retries now widen to the global index space after 25 attempts, and 100
consecutive failures raise. The cap is a systemic-failure detector, not a
tolerance dial: attempts resets per __getitem__, so with p = the bad fraction
of the whole index space it fires with probability p**(N - 25), a step function
in p. At N = 100 the firing band sits at p >~ 0.8, so everything it can reach
is systemic; at N = 1000 it needed ~99 % of the store to be bad and got there
through ~975 sequential zarr/S3 reads per call per worker, which reads as a
hang rather than a failure. Retry logging was one WARNING per attempt -- tens
of thousands of lines per run -- now the first 20 per worker and then one in
500.
Annotation cutoff. Many Scale-"completed" episodes have no usable annotation
span because the injection step lagged, which makes the cutoff a silent no-op
on exactly the runs whose point is to clamp chunks at annotation boundaries.
S3AnnotationCutoffEpisodeResolver(require_annotations=True) drops them, logs
the count it dropped and the count it kept, and raises if nothing is left. The
per-episode span line drops to DEBUG: one line per episode per rank is
thousands of lines on a real split, and the resolver's aggregate says the same
thing.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VBmhpVy7ZcMtsBJWh38cJp
Metrics were paired MSE over whatever the action vector happened to be, which mixes metres and rotation-matrix entries into one number that means nothing in either unit. action_metrics dispatches on the layout instead: cartesian widths report xyz MSE (m^2), geodesic rotation error in degrees (after Gram-Schmidt, so it survives a non-orthonormal prediction), raw rot6d MSE as a training-tracking diagnostic, DTW and Frechet; keypoint widths report keypoint L2 in metres, wrist xyz / rotation error and keypoint DTW. Each also splits early / mid / late (0-10 / 10-50 / 50-100 % of the chunk), because a chunk model can be right at t=0 and drift, and one average hides that. DTW forgives temporal misalignment that paired MSE punishes: a correct trajectory executed slightly early scores near zero. These are all frame-invariant -- the cam-frame revert is a rigid transform applied to prediction and ground truth alike -- so the revert no longer runs for metrics at all, only when rendering the overlay. That also removes the cam-frame MSE, which was the same number as the native one. Rendering is CPU-bound at roughly a second a frame and dominated viz-epoch wall time (unbounded rendering added ~2h per viz epoch on the combined pretrain), so viz_every_n_epochs gates it on Lightning's (current_epoch + 1) convention, the one validation actually runs on, with the final epoch always rendering so short runs get a video; viz_max_batches caps frames per rank. Metrics still run on every batch of every pass. Video fps is divided by world_size, since distributed validation deals indices round-robin and rank 0's consecutive frames are world_size source frames apart -- they were being written as a world_size-times timelapse. train_viz adds a second val loader over the train split (metrics train_viz/, videos videos_train_viz/), on by default for training runs with an evaluator, so train and val curves come from the same evaluator on the same axes. A third unseen_op_valid loader and data.valid_prefix let the held-out-operator splits log seen_op_valid/ and unseen_op_valid/ separately, instead of the seen operators landing in the train_viz slot and the unseen ones in Valid/. ModelWrapper dispatches on val_loader_names by dataloader_idx. val_at_start runs a pre-fit baseline validation, skipped on resumes because trainer.validate there would score the un-resumed base model. Reverse KL is removed: M forward passes per val batch for a number nothing was read off. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The fold recipes this stack was built for: mecka_all_6d, mecka_fold_freeform_6d
and its opsplit variant (wrist frame; cam frame is a transform_list.mode
override rather than a near-duplicate config, and test_data_configs_compose
rejects any data config that differs from its parent only in that mode),
mecka_fold_flagship_opsplit_hpt_6d, mecka_fold_flagship_topop_hpt_6d and
mecka_fold_freeform_opsplit_hpt_keypoints, plus the two 300M HPT models and
pi0.5_bc_mecka_6d.
Defaults, each for a failure seen in a run:
- checkpoints every 50 epochs, saved on train-epoch end. With
check_val_every_n_epoch > 1 the validation-end hook only fires on val epochs,
so every_n_epochs silently skipped the others.
- ddp_pi inference_mode: false. The compiled pi sampler does in-place setup on
its static buffers during cudagraph capture, which is illegal on inference
tensors ("Inplace update to inference tensor outside InferenceMode"); no_grad
is equivalent for eval and creates none.
- ddp_pi validates every 20 epochs instead of 200, and eval_hpt renders viz
every 200.
- pi0.5 gets a constant LR after warmup. The cosine schedule decayed to 0 over
num_training_steps=60000, so runs past ~60k steps trained at ~0 lr and
shorter runs never saw the configured lr for most of training.
- The Qwen stem resolves a repo id to its cached snapshot directory when
offline: transformers >= 4.57 probes the Hub while loading a tokenizer by
repo id even under HF_HUB_OFFLINE, which raises on compute nodes with no
internet. Loading from the directory skips the probe. Offline is read the
way huggingface_hub reads it -- 1 / ON / YES / TRUE in any case, from the
live environment -- rather than by matching the literal "1", which left
HF_HUB_OFFLINE=true hitting the probe the helper exists to avoid.
viz_mecka_episode.py and viz_eva_episode.py render an episode straight from
its zarr, which is how the left-wrist convention bug was found. Both take the
dataset dir from egomimic.utils.paths.default_dataset_dir(), which reads
paths/default.yaml, instead of each re-stating the cluster path.
HumanBimanualCartesianEuler / RobotBimanualCartesianEuler are renamed to
HumanBimanualCartesian6D / RobotBimanualCartesian6D: both pack xyz + rot6d and
have not touched an Euler angle since the 6D conversion, and this is the commit
that makes the human one the converter for a new default recipe.
Also prunes unit tests whose assertions another test already makes on the same
code path, and parametrized cases that repeat a stricter test's compose call.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…evert its outputs NOT TESTED ON HARDWARE. The rollout hard-coded cartesian_wristframe_ypr inputs and used the policy's predictions as-is. Now that eva checkpoints can be trained in several modes (cartesian, cartesian_6d, cartesian_wristframe_6d), that silently feeds a model the wrong inputs and sends the wrong frame to the arms. It now reads data.train_datasets.<name>.resolver.transform_list.mode off the checkpoint's own config, builds the matching input pipeline, and maps predictions back to camera-frame xyz+ypr+gripper through Eva.get_revert_transform_list. The revert needs the un-normalized proprio, which process_batch_for_training normalizes in place, so the raw ee_pose is cloned before that call. A checkpoint whose config records no mode keeps exactly the old behaviour -- legacy inputs, no revert -- and says so loudly, so this cannot silently change what an existing checkpoint does. test_eva_revert_for_rollout_recovers_camframe_ypr covers the revert math by cross-checking the wrist-frame revert against the cam-frame cartesian_6d one, which never enters the wrist frame -- so the frame composition is genuinely validated, though not against independently written SE(3) math. The hardware path itself still needs a real rollout. Both get_transform_list calls here pass allow_legacy_rotation=True: this is inference, and a checkpoint that predates the 6D conversion records exactly the modes the training-time guard rejects. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VBmhpVy7ZcMtsBJWh38cJp
The unshuffled val loaders were truncated by limit_val_batches (80 x 64 x W frames) to their leading episode-hash-sorted slice, so at W = 1 the seen-op head scored 25 of 70 episodes and 6 of 11 operators, and the unseen head 33 of 191 episodes with an 81/19 operator mix on a 52/48 split (lane-0 diagnostic, 2026-09-15, measured single-rank). The loaders are unshuffled only so the overlay video is coherent, and the video renders just the first viz_max_batches (10) batches anyway. Split each val head in two: * metric loader = EvenStrideDataset(split, frames_per_episode=K) with K from data.metric_frames_per_episode[head], so every episode -- and so every operator -- is scored, at the same 80 batches and with no rendering; * video loader = <head>_video, the head's split restricted to the episode hashes pinned in data.video_episodes[head] (one per operator), contiguous and unsubsampled, logging no metrics. The pinned episodes are taken out of the already-resolved split, so resolve-once still holds. EvalVideo.on_validation_step gains a mode argument (metrics / video / both); ModelWrapper.validation_step routes <head>_video to the same evaluator in video mode and skips it entirely on a non-viz epoch, before the forward pass. Heads with no pins are called exactly as before, so a data config that declares neither key behaves as today. K = floor(5120 / episodes-in-split), from the measured splits: opsplit valid 73 (70 eps), train_viz 3 (1346 eps), unseen_op_valid 26 (191 eps); topop valid 256 (20 eps), train_viz 13 (385 eps). Those are SINGLE-RANK numbers and trainHydra scales them by trainer.devices x num_nodes, because DistributedSampler strides rather than chunks: the W ranks between them walk the whole split, so the pre-split window was already 80 x 64 x W frames wide. Without the scaling the shipped 4-GPU ddp_pi run would score 5,110 frames of the seen-val head where it previously scored all 13,744 -- a regression on the one head small enough to score whole. EvenStrideDataset keeps an episode entire when K exceeds its length, so the scaling cannot overrun a short split. Expect a shift in the per-batch Frechet _min/_max curves: they now span many episodes instead of ~2 s of one, so they are not comparable with older runs. Eval.on_validation_step, the abstract base, takes the mode kwarg too: the metrics/video path passes it unconditionally, so an evaluator that predates the split would have raised TypeError at the first val step of any config that declares video_episodes. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VBmhpVy7ZcMtsBJWh38cJp
…split Three-operator twin of mecka_fold_flagship_opsplit_hpt_6d (and the wider sibling of the single-operator topop config): train on the 3 largest flagship fold-clothes operators (1,079 eps / 206,276 frames, 65.3 % of flagship), none of them held out. Seen-op val is the seeded 5 % episode split within those operators (53 eps / 9,826 frames); the unseen-operator head is inherited unchanged, so the runs stay comparable. Lane A keys measured from the real split on a compute node 2026-09-15: metric_frames_per_episode valid=floor(5120/53)=96, train_viz=floor(5120/1026)=4 (single-rank values; trainHydra scales them by the world size) (unseen 26 inherited); video_episodes pins one seen-val episode per train operator (191+223+226 = 640 f = exactly the 10-batch video budget) and inherits the train_viz and unseen pins. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VBmhpVy7ZcMtsBJWh38cJp
…only grad-norm MAD Validating one checkpoint twice now gives identical prompts, actions and metrics. * HPT gets a per-pass RNG seed, base + rank * stride + pass counter, where the counter advances once per forward_eval call and is rewound by ModelWrapper.on_validation_start. forward_eval builds a torch.Generator from it and threads it down HPTModel.forward -> DenoisingPolicy.forward -> sample_action, and forks/seeds the global RNG around the BC val-loss call (DenoisingPolicy.predict draws its own noise and timestep, out of reach of any generator kwarg). Every generator kwarg defaults to None, so training and all other callers are untouched. The val-loss seed is offset by EVAL_LOSS_STRIDE rather than by 1: a torch.Generator seeded with S draws the same stream as a global manual_seed(S), so +1 would have handed batch i's val loss exactly the noise batch i+1 samples with. eval_base_seed is a real __init__ kwarg, so a second eval pass can be made to draw different noise on purpose. * _build_prompts picks annotations with that same seeded RNG whenever the algo is in eval mode, regardless of annotation_sampling_mode, so eval keeps prompt variety instead of collapsing to sample[0]. Train mode is unchanged. * The MAD grad-norm spike detector in on_after_backward is now log-only: it still logs the raw norm, the threshold, the flag and the [GRAD_NORM_SPIKE] line, but never rescales gradients, and every step (flagged or not) goes into the rolling median/MAD window. Fixed clipping moves to Lightning: trainer/default.yaml gains gradient_clip_val (null for now) and gradient_clip_algorithm: norm. The on_before_optimizer_step hook goes with it: Precision._after_closure calls that hook BEFORE _clip_gradients, so "policy_grad_norms_clipped" was the pre-clip norm under another name -- bit-identical to policy_grad_norms_raw, and no more informative once gradient_clip_val is set. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VBmhpVy7ZcMtsBJWh38cJp
torchvision draws one random parameter set per call, so applying the configured train_image_augs to the stacked (B, 3, H, W) camera batch gave every image in the batch the same jitter. Wrap train_image_augs in a new PerSampleAugs module so each image draws its own parameters, and move the inline aug block of _robomimic_to_hpt_data into _apply_image_augs with the same gating. Deterministic eval augs stay batched. _apply_image_augs also checks train_image_augs before calling it: __init__ accepts None and train mode would have raised TypeError. The loop is not free: measured on an A40 at batch 64 with the shipped ColorJitter + Normalize, 13.8 -> 92.2 ms at 224x224 and 62.2 -> 105.5 ms at the camera's 360x640, per camera per training step. Correctness first here; the cost is removed further up the stack, where the jitter itself is vectorised over the batch. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VBmhpVy7ZcMtsBJWh38cJp
… LR group Two defects that would confound any backbone comparison: * a frozen `ResNet` backbone kept its BatchNorm layers in train mode, so it normalized with batch statistics and kept updating its running stats. `requires_grad = False` does not touch either. `train()` now puts a frozen backbone in eval mode. Latent today -- no shipped config freezes the ResNet -- so no run to date was affected. * the pretrained backbone and the randomly initialised trunk shared one LR. The new `model.backbone_lr_scale` key gives the encoders' backbones (`backbone_parameters()`) their own param group at `lr * scale`; at the default 1.0 the optimizer is built exactly as before, as a single group over `trainer.model.parameters()`. Nothing shipped sets it to anything else, so the two-group path is exercised only by its tests for now. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VBmhpVy7ZcMtsBJWh38cJp
ZarrDataset emits a real action_pad_mask (1.0 per real frame, 0.0 per repeat-last pad, honouring the annotation-cutoff EOS clamp) next to every horizoned action chunk. A new InterpolatePadMask transform carries it through the 30 -> 100 upsampling on the interpolators' own grid: an output step is real only if both source frames it interpolates between are real. HPT gains a use_pad_mask algo key (default false, declared in the two flagship model yamls; every other HPT config takes the __init__ default). With it on, process_batch_for_training builds pad_mask (B, S, 1) from the dataset mask instead of ones; DenoisingPolicy.loss_fn takes the mask and means the squared error over the valid steps only. With it off nothing changes: the mask is dropped from the batch and pad_mask stays all ones, which is exactly F.mse_loss. It ships off to match openpi, which asks LeRobotDataset for action_is_pad and then ignores it (Pi0.compute_loss means over the whole horizon). LeRobot's own policies do mask, but reduce with .mean() over all elements including the zeroed ones, so their loss shrinks as padding grows; this divides by the valid count instead, which keeps the gradient magnitude constant. Measured on the three pinned flagship episodes, 6.4-7.6 % of supervised action steps are repeat-last padding and 12.8-15.2 % of samples contain some, so turning it on changes the loss: a run before and a run after are not comparable, and the layout-aware eval metrics would have to be masked in the same change or train and val would be measuring different things. Padding is still repeat-last and no tail samples are dropped. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VBmhpVy7ZcMtsBJWh38cJp
…mbedding) The proprio keys the model consumes read K steps instead of one. Every change is gated on K > 1 (data) or history_len > 1 (model), and K = 1 / stride 1 is bit-for-bit today's pipeline. Data: Human.get_keymap gains proprio_history and history_stride; for K > 1 it marks the proprio entries with "history": K plus the stride, and ZarrDataset.__getitem__ reads the window idx - s*(K-1) ... idx in steps of s, clamped at the episode start and front-padded by repeating the first real frame, so the key is always (K, D) with the current frame last. A proprio_history_mask (1 = real, 0 = front padding) travels with it. The stride is why this ships on rather than at K = 1. Measured on a real flagship episode (30 fps, fold_clothes), the median wrist translation is 6.8 mm over 1 frame, 25.7 mm over 3, 42.9 mm over 5 and 80.8 mm over 10: an unstrided K = 3 window is 67 ms of past, close enough to a duplicate of the current pose to be mostly sensor noise, while the action chunk it conditions spans a full second. The flagship recipe therefore ships proprio_history: 3 / history_stride: 5 -- 0.33 s of past -- with the stem at history_len: 3. That changes the observation, so runs before and after this commit are not comparable, and history_dropout: 0.2 stops being inert: it is now a real hyperparameter, and the mitigation for the copycat/causal-confusion risk that proprio history classically brings. Transforms: the cartesian builders already treat a leading axis as batch; ActionChunkCoordinateFrameTransform / PoseCoordinateFrameTransform gain an opt-in target_history so a proprio target frame collapses to the CURRENT step (the action chunk must not move when history is added), and Reshape / QuaternionPoseToYPR tolerate the extra axis. observations.state.ee_pose is (K, 20). Eval: every human revert transform list starts with a SelectCurrentStep on its proprio key, so the val metrics, the cam-frame revert and the overlay video see the single current pose they always saw. No-op at K = 1 and idempotent. Norm stats are the baseline's: per-channel (D,) stats broadcast over (K, D), and both the training and the precompute stats pass force proprio_history=1 so K > 1 and K = 1 runs share one set. Model: MLPPolicyStem gains history_len / history_dropout. For history_len > 1 it owns a zero-init learned time embedding added after the projection, and HPT.stem_process skips the input-side sinusoid for such a stem (a constant [0, 1, 0, 1, ...] on normalized values is amplitude-1 noise on [-1, 1] data). history_dropout replaces the whole past by the current step per sample in training, exactly what an episode-start sample looks like after front padding. The K = 1 baseline stays one override away (proprio_history=1, history_stride=1, history_len=1) and the tests pin that it is bit-identical. Every other shipped config is untouched at K = 1, including the keypoint arm, whose wrist-frame proprio would need per-step framing before it can carry history. A compose test pins every train_*.yaml recipe's data K against its stem's history_len, because a recipe that inherits one model's data config and overrides the model is exactly how the two drift apart, and nothing else composes both halves. _stem_history_len takes the domain rather than matching a stem name by suffix: a cotrain run registers one state_ee_pose stem per domain, and a suffix match returns whichever comes first, so the human side was checked against eva's history_len. Latent while everything shipped K = 1; a blocker now. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VBmhpVy7ZcMtsBJWh38cJp
… padding `HPTModel.finalize_modules` ran `self.apply(self._init_weights)` over the whole model, which xavier-randomised the pretrained HF encoders inside the text stems (the frozen Qwen3-Embedding weights were overwritten in place and their biases zeroed). Modules that own pretrained weights now declare them through the `PretrainedWeights` mixin, and the init pass walks the tree without ever entering such a subtree, so the ResNet backbone, `T5Encoder` and any future pretrained stem are covered by the same mechanism. `HPT.__init__` also called `finalize_modules` twice; the first call is gone. Once the model is built, moved and upcast, `verify_pretrained_weights` hashes every declared pretrained submodule and compares it with the checkpoint read straight off the local HF snapshot, failing the run on a mismatch. It costs a second full copy of the encoder (~1.1 GB fp16 at 0.6B) and ~3 s of hashing per rank, so `verify_pretrained: false` turns it off; it is on by default, because a silently re-initialised encoder trains to a plausible wrong result. Only stems that own an HF snapshot are checked -- `ResNet` returns no reference, so it is protected from the init pass but not hash-verified, which the mixin docstring now says. Finally, `QwenPerTokenEncoder.compute_latent` now passes the tokenizer attention mask into its cross-attention. Padded positions are zeroed before the biased `proj`, so they were the learned constant `proj.bias`, not zero, and got real attention weight: text-conditioned results from before this commit are not comparable with results after it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VBmhpVy7ZcMtsBJWh38cJp
The annotation vocabulary is tiny next to the number of training steps, so a frozen text encoder re-runs the same 600M-parameter forward every step. `_Qwen3BaseEncoder` now keeps a lazy in-memory cache of the per-token hidden states keyed by the exact prompt string: misses go through the tokenizer + encoder in ONE sub-batch of unique strings, their un-padded (L_i, D) rows are stored detached, and the batch is re-assembled with the same LEFT padding the tokenizer produces, so `forward_with_mask` / the pooled path are unchanged. Only legal when the encoder is frozen - a trainable encoder must stay online. Knobs on the stem: `cache_text_features` (default true) and `text_cache_max_entries` (default 8192, LRU); both added to the model yamls with a Qwen stem. The cap counts entries, and an entry is the prompt's un-padded (L, D) states -- 4-25 KB for real annotations, 262 KB at max_length 128 -- so the default is ~100-200 MB of device memory in practice and 2 GB in the worst case; the docstring says so next to the knob. A batch carrying more unique prompts than the cap reads its rows from the miss pass rather than from the cache, because the LRU trim runs first and would otherwise evict entries the very next line looks up. `_apply` (to/cuda/float) clears the cache, and `clear_text_feature_cache()` / `text_feature_cache_stats()` are exposed. Padded positions come back as zeros instead of the encoder's pad-token output; every consumer masks them out. Cached-vs-online on 50 real prompts (fp16, each warmed alone: worst-case padding mismatch): max |diff| 3.6e-3 on the latents and 9.4e-2 on raw hidden states, exactly equal to the online path's own batch-padding noise (9.4e-2); in fp32 the same comparison is 1.1e-6 / 1.6e-4. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VBmhpVy7ZcMtsBJWh38cJp
…ide in seconds Human.get_keymap(image_memory=N, image_memory_stride_s=s) adds <front>_mem: N front frames s seconds apart, the newest s seconds back, plus a (N,) <front>_mem_mask of the steps inside the episode. history_stride_s spaces proprio history in seconds, so windows span the same time at any fps. History windows are now point reads instead of a contiguous span thinned afterwards: on JPEG keys (one frame per chunk) a span would fetch every frame between steps. plan_reads still merges them on numeric keys. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UL4UaiqSDNbhpatNwdvJbe
…-step eval MSE _stem_history_len called .get on the policy's stems, an nn.ModuleDict once built, so every proprio-history (K > 1) run crashed at its first batch. The unit tests used a plain dict. The last-step MSE handed torchmetrics a strided [:, -1] slice, which it .view(-1)s; on GPU .cpu() copied it contiguous, on a CPU run it raised. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UL4UaiqSDNbhpatNwdvJbe
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UL4UaiqSDNbhpatNwdvJbe
…lation arms RDTMemory: N past frames of one camera through a frozen DINOv3 tower (only its proj trains), a Perceiver resampler shared across frames pooling each to M latents, a sin-cos code of each frame's age in seconds, and optionally that step's proprio token (fuse_proprio). The N x M tokens join the DiT's self-attention prefix [t; freq; state; memory; actions] under a key mask, so steps before the episode start and dropped memories (trunk.memory.dropout) are never attended. Without trunk.memory the model is unchanged. Four train configs on mecka 20% 6D, a 2x2 over proprio history and visual memory (10 steps, 1 s apart): _base, _proprio, _vision, _both. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UL4UaiqSDNbhpatNwdvJbe
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UL4UaiqSDNbhpatNwdvJbe
Follows the dual-encoder design of diffusion-policy-experiments: - the memory tower is a second DINOv3, fine-tuned at the backbone LR scale (the param-group walk now covers every module with backbone_parameters), pooling each frame to its CLS token; the Perceiver resampler is gone - the window ends at the current frame (t-9 s .. t) - short_dropout (0.3 in the memory arms) replaces a sample's short image stream by a learned null token, replacing whole-memory dropout Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UL4UaiqSDNbhpatNwdvJbe
Backprop through 320 frames / GPU OOMed an H100; with the tower's blocks recomputed the both arm trains at 1.19 s / step on 8xH100. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UL4UaiqSDNbhpatNwdvJbe
PI.process_batch_for_training kept only keymap keys plus episode_hash, so batch['intrinsics'] never reached viz_gt_preds and Human.viz fell back to ARIA_INTRINSICS. Every PI overlay video on non-Aria data was drawn with the wrong camera (about 58 px low on Mecka). Metrics were unaffected. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NxUsbpS65QBRpBiauTEtTs
The keypoint pipeline resamples 30 raw frames to a 100-step chunk, so a mecka step is 29/99 of a frame, but the replay drew step k on frame a + k*stride with an integer stride (inferred as 1). GT and prediction played ~3.4x slow and the head-motion fit paired different hand poses: 15-50 mm between the drawn GT and the frame's own label on the pinned valid episode, 1.6 mm mean after this change. render_replay takes a fractional stride and draws each real frame with the step that falls on it; infer_stride searches D/(H-1) for whole spans D. Integer strides render as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015Kf5s9frd6T6cbBvapfjFy
aidang3019
force-pushed
the
ryanco/replay-fractional-stride
branch
from
September 30, 2026 01:07
b99295a to
bf4981a
Compare
RyanPCo
force-pushed
the
ryanco/pi-overlay-intrinsics
branch
from
October 12, 2026 20:26
70140ee to
ecf46e5
Compare
Contributor
Author
|
Folded into #690 to keep the stack under Graphite's 50-branch cap; the changes are unchanged there. |
This was referenced Oct 12, 2026
Open
This was referenced Oct 22, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The keypoint pipeline resamples 30 raw frames to a 100-step chunk, so a mecka step is 29/99 of a frame, but the replay drew step k on frame
a + k*stridewith an integer stride (inferred as 1). GT and prediction played ~3.4x slow and the head-motion fit paired different hand poses.render_replaynow takes a fractional stride and draws each real frame with the step that falls on it;infer_stridesearchesD/(H-1)for whole spansD. Integer strides render as before. Viz only: training targets and val metrics don't go through this code.Check on the pinned valid episode (
69b5691e…, mean keypoint distance between the drawn GT and the frame's own label):Tests:
tests/unit/test_replay_viz.py(fractional-stride inference and frame counts).🤖 Generated with Claude Code
https://claude.ai/code/session_015Kf5s9frd6T6cbBvapfjFy