Repository navigation
mcap_process: convert vendor MCAP episodes into the Lance tables - #719
Draft
SaveAanchal wants to merge 79 commits into
Draft
SaveAanchal wants to merge 79 commits into
SaveAanchal wants to merge 79 commits into
Conversation
…an default
Human data trains on hand keypoints instead of a palm-origin cartesian pose.
The action is 144-D wrist-first, per hand
[wrist xyz (3) | wrist rot6d (6) | 21 MANO keypoints (63)] x {left, right},
built by the keypoints_{wrist,head}frame_6d modes: the existing ypr keypoint
builders produce the 138-D layout and KeypointsYPRToRot6D re-encodes just the
wrist rotation, leaving the keypoint blocks untouched. Both the action and the
observations.state.keypoints proprio are converted, so neither carries a
wrapping euler angle. MANO keypoint 0 IS the wrist, so per hand the wrist
position rides at both ch 0-2 and ch 9-11; that is kept deliberately (21
keypoints is the MANO interface every consumer reshapes to) and written down
in the bimanual_keypoint_layout docstring.
include_ee_pose additionally builds the cartesian head-frame ee_pose next to
the keypoint action (keymap + transform list), so a keypoint pi0.5 run
discretizes the same State: prompt block as the cartesian runs and differs
from them only in the action space. The cartesian builder runs with
delete_target_world=False to keep obs_head_pose alive for the keypoint
builder, and its actions_cartesian by-product is dropped.
144 does not fit openpi's action vector, whose PyTorch port hard-codes 32-wide
action projections regardless of Pi0Config.action_dim. PI now reads action_dim
and, when it is not 32, swaps in action_dim-wide action_in_proj /
action_out_proj AFTER the strict base-weight load: only those two layers start
fresh, everything else comes from the checkpoint, and sample_actions already
draws noise at config.action_dim. pad_to_width zero-pads narrower embodiments
(eva's 20-D) into the same vector and raises rather than truncating a wider
one. _reduce_loss then restricts the mean to the slots the converter actually
packs, so eva does not average its loss over 112 zero-padded dims and
contribute a fraction of the gradient signal it had at width 32.
HumanBimanualKeypoints is identity-in, slice-out.
HPT human configs move to a 144-D state_keypoints stem. hpt_bc_flow_human
holds the body -- there is no vendor notion at the embodiment level, so the
canonical config is the embodiment one -- and hpt_bc_flow_{mecka,aria,scale}
are two-line aliases of it. HPTModel.add_stem drops null stem entries so a
child config can delete a stem its base declares, which hydra merges cannot
express, and hpt_cotrain_flow_seperate_head uses it to drop state_ee_pose.
With the keypoints_*_6d modes in place, the guard from the previous commit
widens to the keypoints ypr and quat modes, so every remaining discontinuous
rotation target needs an explicit allow_legacy_rotation=True (the aria->MANO
and mecka keypoint viz tutorials, which read pre-6D data). The cartesian
configs that were still pinned to the ypr widths move with it:
hpt_bc_flow_human_cartesian and the cotrain base to human 20/18, the
shared-head cotrains to a 20-wide shared action.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VBmhpVy7ZcMtsBJWh38cJp
Wrist-relative actions made a latent division safe-looking: the t=0 cell of a wrist-frame chunk is exactly the identity pose, so its quantile range is 0 and the historical `(q99 - q1) + 1e-6` denominator scaled any off-convention value by 1e6. A channel whose range is below NORM_MIN_RANGE (1e-4) is now treated as constant -- it normalizes to 0 and unnormalizes to its centre rather than dividing by epsilon. Diffusion Policy / robomimic ignore ranges under 1e-4 and GR00T maps min == max to 0; this is the same rule, applied in the shared _apply_norm_one / _apply_unnorm_one so the dataset and the tests agree. norm_stats.pool_horizon (default false, so nothing changes unless asked) computes one set of stats per channel over every timestep of the chunk, the way openpi does, instead of one per (timestep, channel) cell. The pooled stats are tiled back to the (T, D) shape, so normalize, the bounds check and checkpoints see the same layout either way. It is part of the norm-stat cache key, so flipping it cannot silently reuse stats computed the other way. precompute_norm_stats.py computes and caches the stats for a recipe outside a training run, keyed by the same file + content hash. Which of the two is better is unsettled: the literature is split (LBM per-timestep with a stated rationale, openpi / RDT / GR00T / Diffusion Policy / UMI / lerobot / Octo pooled) and nothing here has compared them on a real recipe, so the default stays the inherited per-timestep one and the config comment says what would make it worth flipping. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VBmhpVy7ZcMtsBJWh38cJp
A 6D rotation is two columns of a rotation matrix: already bounded in [-1, 1], and per-channel centre-and-scale pulls the pair off the manifold without buying any resolution. The two projects that anchor an action chunk at near-identity the way this repo does -- TRI's LBM and UMI -- both exclude rotation from normalization for exactly that reason (LBM: "We avoid this normalization procedure for the 6D rotation in the poses to avoid corrupting the rotation matrix"; UMI sends its rotation channels through an identity normalizer). Per timestep AND normalized was the one combination with no precedent, and it is what manufactures the t=0 zero-range rotation cells NORM_MIN_RANGE exists to catch -- this removes them at the source. rot6d_channels() reads the indices off the existing layout tables, so the rule follows the layout rather than a second hardcoded list: the cartesian 18/20 widths and the 144-D keypoint layout, action and proprio alike. The ypr widths keep the old behaviour; an Euler angle is not a matrix column and those modes are legacy-only now. _apply_norm_one / _apply_unnorm_one take the channels and pass them through untouched, so the dataset path, the evaluator revert and the rollout revert all follow from one place, and the degenerate-channel guard is left to do what it is actually good at: the structural kp0 / kp9 position zeros. Norm-stat files stay valid -- what changes is how they are applied -- but a run before this commit and a run after it are not comparable, and the loss is now weighted differently between translation (unit scale) and rotation (raw). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VBmhpVy7ZcMtsBJWh38cJp
MultiDataset.to_state now records rot6d_passthrough. A checkpoint state that has stats but no flag predates the passthrough and normalized the rot6d channels, so from_state turns it off for those; fresh stats (empty state) keep it on. Without this, eval/rescore/requeue of such a checkpoint fed raw rot6d proprio and left predicted rotations in normalized space, silently. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GqAcWA2Lk9zubUhKX5DW1v
Three independent ways MultiDataset could fail a whole run on data that was
fine, or on data it should have rejected up front.
Bounds check. Quantile bounds on a rotation channel are meaningless: euler ypr
wraps at +-pi and rot6d columns are matrix entries, so neither respects a
quantile interval. _bounds_check_channels uses the layout tables to check only
translation, gripper and keypoint channels; unrecognized widths still get the
full-vector check, and the NaN/Inf check always covers everything. The bounds
themselves get slack -- 1e-6 absolute plus half of each cell's own quantile
range -- because the check is any-cell over ~13 200 cells per keypoint chunk.
Measured on the 2815-frame flagship episode through the real
keypoints_wristframe_6d pipeline (zscore, stats from sample_frac 0.10 over its
own frames, 403 probe frames, any-key rejection): 39.5 % of frames rejected at
slack 0, 17.9 % at 0.05, 11.4 % at 0.1, 5.2 % at 0.25, 3.2 % at 0.5, 1.0 % at
1.0. The 132-channel action chunk drives all of it. Half a range is the knee,
not a cushion -- though the absolute numbers move with which frames the stats
were drawn from, so the comment says how to reproduce them.
Corrupt values sit orders of magnitude outside the range and are still caught.
Cells the normalizer treats as constant are exempt, since a tiny offset there
cannot blow up the loss -- and which stat decides that is norm_mode's business
(std under zscore, the quantile range under quantile), so the exemption asks
_degenerate instead of hardcoding q99 - q1; every shipped config is zscore, and
a channel that is 0 in 99.6 % of samples and 1.0 in the rest has q99 - q1 = 0
but std = 0.063, i.e. it is still divided by and must still be checked.
quantile_0_01 / quantile_99_99 now snap to observed samples ("lower" /
"higher") instead of interpolating: under 10k samples the linear percentile
lands strictly inside the sample range, so the extreme frames of the very
episodes the stats came from were rejected (3-16 % of a single episode in the
overfit runs). A stats/sample shape mismatch is logged once per key instead of
silently disabling the check.
Sampler. Retries stayed inside the failing episode until they exhausted it,
so one wholly-bad episode killed the run and NCCL-timed-out every other rank.
Retries now widen to the global index space after 25 attempts, and 100
consecutive failures raise. The cap is a systemic-failure detector, not a
tolerance dial: attempts resets per __getitem__, so with p = the bad fraction
of the whole index space it fires with probability p**(N - 25), a step function
in p. At N = 100 the firing band sits at p >~ 0.8, so everything it can reach
is systemic; at N = 1000 it needed ~99 % of the store to be bad and got there
through ~975 sequential zarr/S3 reads per call per worker, which reads as a
hang rather than a failure. Retry logging was one WARNING per attempt -- tens
of thousands of lines per run -- now the first 20 per worker and then one in
500.
Annotation cutoff. Many Scale-"completed" episodes have no usable annotation
span because the injection step lagged, which makes the cutoff a silent no-op
on exactly the runs whose point is to clamp chunks at annotation boundaries.
S3AnnotationCutoffEpisodeResolver(require_annotations=True) drops them, logs
the count it dropped and the count it kept, and raises if nothing is left. The
per-episode span line drops to DEBUG: one line per episode per rank is
thousands of lines on a real split, and the resolver's aggregate says the same
thing.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VBmhpVy7ZcMtsBJWh38cJp
Metrics were paired MSE over whatever the action vector happened to be, which mixes metres and rotation-matrix entries into one number that means nothing in either unit. action_metrics dispatches on the layout instead: cartesian widths report xyz MSE (m^2), geodesic rotation error in degrees (after Gram-Schmidt, so it survives a non-orthonormal prediction), raw rot6d MSE as a training-tracking diagnostic, DTW and Frechet; keypoint widths report keypoint L2 in metres, wrist xyz / rotation error and keypoint DTW. Each also splits early / mid / late (0-10 / 10-50 / 50-100 % of the chunk), because a chunk model can be right at t=0 and drift, and one average hides that. DTW forgives temporal misalignment that paired MSE punishes: a correct trajectory executed slightly early scores near zero. These are all frame-invariant -- the cam-frame revert is a rigid transform applied to prediction and ground truth alike -- so the revert no longer runs for metrics at all, only when rendering the overlay. That also removes the cam-frame MSE, which was the same number as the native one. Rendering is CPU-bound at roughly a second a frame and dominated viz-epoch wall time (unbounded rendering added ~2h per viz epoch on the combined pretrain), so viz_every_n_epochs gates it on Lightning's (current_epoch + 1) convention, the one validation actually runs on, with the final epoch always rendering so short runs get a video; viz_max_batches caps frames per rank. Metrics still run on every batch of every pass. Video fps is divided by world_size, since distributed validation deals indices round-robin and rank 0's consecutive frames are world_size source frames apart -- they were being written as a world_size-times timelapse. train_viz adds a second val loader over the train split (metrics train_viz/, videos videos_train_viz/), on by default for training runs with an evaluator, so train and val curves come from the same evaluator on the same axes. A third unseen_op_valid loader and data.valid_prefix let the held-out-operator splits log seen_op_valid/ and unseen_op_valid/ separately, instead of the seen operators landing in the train_viz slot and the unseen ones in Valid/. ModelWrapper dispatches on val_loader_names by dataloader_idx. val_at_start runs a pre-fit baseline validation, skipped on resumes because trainer.validate there would score the un-resumed base model. Reverse KL is removed: M forward passes per val batch for a number nothing was read off. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The fold recipes this stack was built for: mecka_all_6d, mecka_fold_freeform_6d
and its opsplit variant (wrist frame; cam frame is a transform_list.mode
override rather than a near-duplicate config, and test_data_configs_compose
rejects any data config that differs from its parent only in that mode),
mecka_fold_flagship_opsplit_hpt_6d, mecka_fold_flagship_topop_hpt_6d and
mecka_fold_freeform_opsplit_hpt_keypoints, plus the two 300M HPT models and
pi0.5_bc_mecka_6d.
Defaults, each for a failure seen in a run:
- checkpoints every 50 epochs, saved on train-epoch end. With
check_val_every_n_epoch > 1 the validation-end hook only fires on val epochs,
so every_n_epochs silently skipped the others.
- ddp_pi inference_mode: false. The compiled pi sampler does in-place setup on
its static buffers during cudagraph capture, which is illegal on inference
tensors ("Inplace update to inference tensor outside InferenceMode"); no_grad
is equivalent for eval and creates none.
- ddp_pi validates every 20 epochs instead of 200, and eval_hpt renders viz
every 200.
- pi0.5 gets a constant LR after warmup. The cosine schedule decayed to 0 over
num_training_steps=60000, so runs past ~60k steps trained at ~0 lr and
shorter runs never saw the configured lr for most of training.
- The Qwen stem resolves a repo id to its cached snapshot directory when
offline: transformers >= 4.57 probes the Hub while loading a tokenizer by
repo id even under HF_HUB_OFFLINE, which raises on compute nodes with no
internet. Loading from the directory skips the probe. Offline is read the
way huggingface_hub reads it -- 1 / ON / YES / TRUE in any case, from the
live environment -- rather than by matching the literal "1", which left
HF_HUB_OFFLINE=true hitting the probe the helper exists to avoid.
viz_mecka_episode.py and viz_eva_episode.py render an episode straight from
its zarr, which is how the left-wrist convention bug was found. Both take the
dataset dir from egomimic.utils.paths.default_dataset_dir(), which reads
paths/default.yaml, instead of each re-stating the cluster path.
HumanBimanualCartesianEuler / RobotBimanualCartesianEuler are renamed to
HumanBimanualCartesian6D / RobotBimanualCartesian6D: both pack xyz + rot6d and
have not touched an Euler angle since the 6D conversion, and this is the commit
that makes the human one the converter for a new default recipe.
Also prunes unit tests whose assertions another test already makes on the same
code path, and parametrized cases that repeat a stricter test's compose call.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…evert its outputs NOT TESTED ON HARDWARE. The rollout hard-coded cartesian_wristframe_ypr inputs and used the policy's predictions as-is. Now that eva checkpoints can be trained in several modes (cartesian, cartesian_6d, cartesian_wristframe_6d), that silently feeds a model the wrong inputs and sends the wrong frame to the arms. It now reads data.train_datasets.<name>.resolver.transform_list.mode off the checkpoint's own config, builds the matching input pipeline, and maps predictions back to camera-frame xyz+ypr+gripper through Eva.get_revert_transform_list. The revert needs the un-normalized proprio, which process_batch_for_training normalizes in place, so the raw ee_pose is cloned before that call. A checkpoint whose config records no mode keeps exactly the old behaviour -- legacy inputs, no revert -- and says so loudly, so this cannot silently change what an existing checkpoint does. test_eva_revert_for_rollout_recovers_camframe_ypr covers the revert math by cross-checking the wrist-frame revert against the cam-frame cartesian_6d one, which never enters the wrist frame -- so the frame composition is genuinely validated, though not against independently written SE(3) math. The hardware path itself still needs a real rollout. Both get_transform_list calls here pass allow_legacy_rotation=True: this is inference, and a checkpoint that predates the 6D conversion records exactly the modes the training-time guard rejects. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VBmhpVy7ZcMtsBJWh38cJp
…eckpoint
The rollout looked for data.train_datasets.<eva>.resolver.transform_list.mode
in the checkpoint's config tree, but _build_model_config_tree saves only
model, so every checkpoint fell back to the legacy 14-D ypr pipeline and a
6D policy would have produced garbage commands. trainHydra now records
{dataset: transform_list.mode} as data_transform_modes in that tree, and the
rollout reads it from there. A checkpoint without it (every one trained
before this commit) is taken as legacy only when its saved actions_cartesian
width is 14; otherwise the rollout refuses to guess and asks for the new
--transform-mode flag.
EMBODIMENT_MAP still held the pre-collapse ids (8/7/6 for both/left/right);
it now reads them from EMBODIMENT.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GqAcWA2Lk9zubUhKX5DW1v
The unshuffled val loaders were truncated by limit_val_batches (80 x 64 x W frames) to their leading episode-hash-sorted slice, so at W = 1 the seen-op head scored 25 of 70 episodes and 6 of 11 operators, and the unseen head 33 of 191 episodes with an 81/19 operator mix on a 52/48 split (lane-0 diagnostic, 2026-09-15, measured single-rank). The loaders are unshuffled only so the overlay video is coherent, and the video renders just the first viz_max_batches (10) batches anyway. Split each val head in two: * metric loader = EvenStrideDataset(split, frames_per_episode=K) with K from data.metric_frames_per_episode[head], so every episode -- and so every operator -- is scored, at the same 80 batches and with no rendering; * video loader = <head>_video, the head's split restricted to the episode hashes pinned in data.video_episodes[head] (one per operator), contiguous and unsubsampled, logging no metrics. The pinned episodes are taken out of the already-resolved split, so resolve-once still holds. EvalVideo.on_validation_step gains a mode argument (metrics / video / both); ModelWrapper.validation_step routes <head>_video to the same evaluator in video mode and skips it entirely on a non-viz epoch, before the forward pass. Heads with no pins are called exactly as before, so a data config that declares neither key behaves as today. K = floor(5120 / episodes-in-split), from the measured splits: opsplit valid 73 (70 eps), train_viz 3 (1346 eps), unseen_op_valid 26 (191 eps); topop valid 256 (20 eps), train_viz 13 (385 eps). Those are SINGLE-RANK numbers and trainHydra scales them by trainer.devices x num_nodes, because DistributedSampler strides rather than chunks: the W ranks between them walk the whole split, so the pre-split window was already 80 x 64 x W frames wide. Without the scaling the shipped 4-GPU ddp_pi run would score 5,110 frames of the seen-val head where it previously scored all 13,744 -- a regression on the one head small enough to score whole. EvenStrideDataset keeps an episode entire when K exceeds its length, so the scaling cannot overrun a short split. Expect a shift in the per-batch Frechet _min/_max curves: they now span many episodes instead of ~2 s of one, so they are not comparable with older runs. Eval.on_validation_step, the abstract base, takes the mode kwarg too: the metrics/video path passes it unconditionally, so an evaluator that predates the split would have raised TypeError at the first val step of any config that declares video_episodes. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VBmhpVy7ZcMtsBJWh38cJp
…split Three-operator twin of mecka_fold_flagship_opsplit_hpt_6d (and the wider sibling of the single-operator topop config): train on the 3 largest flagship fold-clothes operators (1,079 eps / 206,276 frames, 65.3 % of flagship), none of them held out. Seen-op val is the seeded 5 % episode split within those operators (53 eps / 9,826 frames); the unseen-operator head is inherited unchanged, so the runs stay comparable. Lane A keys measured from the real split on a compute node 2026-09-15: metric_frames_per_episode valid=floor(5120/53)=96, train_viz=floor(5120/1026)=4 (single-rank values; trainHydra scales them by the world size) (unseen 26 inherited); video_episodes pins one seen-val episode per train operator (191+223+226 = 640 f = exactly the 10-batch video budget) and inherits the train_viz and unseen pins. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VBmhpVy7ZcMtsBJWh38cJp
stem input_dim, infer_ac_dims, act_dim and enc-dec output_dim now interpolate
robomimic_model.dims.{<emb>.proprio,<emb>.action,action_width}. Resolved values
are unchanged (pinned by tests/unit/test_model_dims.py). Pi configs untouched.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MWtR1x5tjREwoxQZAnUGrh
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BbQNNaMAbTNMypegmJkv4s
…only grad-norm MAD Validating one checkpoint twice now gives identical prompts, actions and metrics. * HPT gets a per-pass RNG seed, base + rank * stride + pass counter, where the counter advances once per forward_eval call and is rewound by ModelWrapper.on_validation_start. forward_eval builds a torch.Generator from it and threads it down HPTModel.forward -> DenoisingPolicy.forward -> sample_action, and forks/seeds the global RNG around the BC val-loss call (DenoisingPolicy.predict draws its own noise and timestep, out of reach of any generator kwarg). Every generator kwarg defaults to None, so training and all other callers are untouched. The val-loss seed is offset by EVAL_LOSS_STRIDE rather than by 1: a torch.Generator seeded with S draws the same stream as a global manual_seed(S), so +1 would have handed batch i's val loss exactly the noise batch i+1 samples with. eval_base_seed is a real __init__ kwarg, so a second eval pass can be made to draw different noise on purpose. * _build_prompts picks annotations with that same seeded RNG whenever the algo is in eval mode, regardless of annotation_sampling_mode, so eval keeps prompt variety instead of collapsing to sample[0]. Train mode is unchanged. * The MAD grad-norm spike detector in on_after_backward is now log-only: it still logs the raw norm, the threshold, the flag and the [GRAD_NORM_SPIKE] line, but never rescales gradients, and every step (flagged or not) goes into the rolling median/MAD window. Fixed clipping moves to Lightning: trainer/default.yaml gains gradient_clip_val (null for now) and gradient_clip_algorithm: norm. The on_before_optimizer_step hook goes with it: Precision._after_closure calls that hook BEFORE _clip_gradients, so "policy_grad_norms_clipped" was the pre-clip norm under another name -- bit-identical to policy_grad_norms_raw, and no more informative once gradient_clip_val is set. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VBmhpVy7ZcMtsBJWh38cJp
stem input_dim, infer_ac_dims, act_dim and enc-dec output_dim now interpolate
robomimic_model.dims.{<emb>.proprio,<emb>.action,action_width}. Resolved values
are unchanged (pinned by tests/unit/test_model_dims.py). Pi configs untouched.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MWtR1x5tjREwoxQZAnUGrh
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BbQNNaMAbTNMypegmJkv4s
torchvision draws one random parameter set per call, so applying the configured train_image_augs to the stacked (B, 3, H, W) camera batch gave every image in the batch the same jitter. Wrap train_image_augs in a new PerSampleAugs module so each image draws its own parameters, and move the inline aug block of _robomimic_to_hpt_data into _apply_image_augs with the same gating. Deterministic eval augs stay batched. _apply_image_augs also checks train_image_augs before calling it: __init__ accepts None and train mode would have raised TypeError. The loop is not free: measured on an A40 at batch 64 with the shipped ColorJitter + Normalize, 13.8 -> 92.2 ms at 224x224 and 62.2 -> 105.5 ms at the camera's 360x640, per camera per training step. Correctness first here; the cost is removed further up the stack, where the jitter itself is vectorised over the batch. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VBmhpVy7ZcMtsBJWh38cJp
… LR group Two defects that would confound any backbone comparison: * a frozen `ResNet` backbone kept its BatchNorm layers in train mode, so it normalized with batch statistics and kept updating its running stats. `requires_grad = False` does not touch either. `train()` now puts a frozen backbone in eval mode. Latent today -- no shipped config freezes the ResNet -- so no run to date was affected. * the pretrained backbone and the randomly initialised trunk shared one LR. The new `model.backbone_lr_scale` key gives the encoders' backbones (`backbone_parameters()`) their own param group at `lr * scale`; at the default 1.0 the optimizer is built exactly as before, as a single group over `trainer.model.parameters()`. Nothing shipped sets it to anything else, so the two-group path is exercised only by its tests for now. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VBmhpVy7ZcMtsBJWh38cJp
ZarrDataset emits a real action_pad_mask (1.0 per real frame, 0.0 per repeat-last pad, honouring the annotation-cutoff EOS clamp) next to every horizoned action chunk. A new InterpolatePadMask transform carries it through the 30 -> 100 upsampling on the interpolators' own grid: an output step is real only if both source frames it interpolates between are real. HPT gains a use_pad_mask algo key (default false, declared in the two flagship model yamls; every other HPT config takes the __init__ default). With it on, process_batch_for_training builds pad_mask (B, S, 1) from the dataset mask instead of ones; DenoisingPolicy.loss_fn takes the mask and means the squared error over the valid steps only. With it off nothing changes: the mask is dropped from the batch and pad_mask stays all ones, which is exactly F.mse_loss. It ships off to match openpi, which asks LeRobotDataset for action_is_pad and then ignores it (Pi0.compute_loss means over the whole horizon). LeRobot's own policies do mask, but reduce with .mean() over all elements including the zeroed ones, so their loss shrinks as padding grows; this divides by the valid count instead, which keeps the gradient magnitude constant. Measured on the three pinned flagship episodes, 6.4-7.6 % of supervised action steps are repeat-last padding and 12.8-15.2 % of samples contain some, so turning it on changes the loss: a run before and a run after are not comparable, and the layout-aware eval metrics would have to be masked in the same change or train and val would be measuring different things. Padding is still repeat-last and no tail samples are dropped. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VBmhpVy7ZcMtsBJWh38cJp
…mbedding) The proprio keys the model consumes read K steps instead of one. Every change is gated on K > 1 (data) or history_len > 1 (model), and K = 1 / stride 1 is bit-for-bit today's pipeline. Data: Human.get_keymap gains proprio_history and history_stride; for K > 1 it marks the proprio entries with "history": K plus the stride, and ZarrDataset.__getitem__ reads the window idx - s*(K-1) ... idx in steps of s, clamped at the episode start and front-padded by repeating the first real frame, so the key is always (K, D) with the current frame last. A proprio_history_mask (1 = real, 0 = front padding) travels with it. The stride is why this ships on rather than at K = 1. Measured on a real flagship episode (30 fps, fold_clothes), the median wrist translation is 6.8 mm over 1 frame, 25.7 mm over 3, 42.9 mm over 5 and 80.8 mm over 10: an unstrided K = 3 window is 67 ms of past, close enough to a duplicate of the current pose to be mostly sensor noise, while the action chunk it conditions spans a full second. The flagship recipe therefore ships proprio_history: 3 / history_stride: 5 -- 0.33 s of past -- with the stem at history_len: 3. That changes the observation, so runs before and after this commit are not comparable, and history_dropout: 0.2 stops being inert: it is now a real hyperparameter, and the mitigation for the copycat/causal-confusion risk that proprio history classically brings. Transforms: the cartesian builders already treat a leading axis as batch; ActionChunkCoordinateFrameTransform / PoseCoordinateFrameTransform gain an opt-in target_history so a proprio target frame collapses to the CURRENT step (the action chunk must not move when history is added), and Reshape / QuaternionPoseToYPR tolerate the extra axis. observations.state.ee_pose is (K, 20). Eval: every human revert transform list starts with a SelectCurrentStep on its proprio key, so the val metrics, the cam-frame revert and the overlay video see the single current pose they always saw. No-op at K = 1 and idempotent. Norm stats are the baseline's: per-channel (D,) stats broadcast over (K, D), and both the training and the precompute stats pass force proprio_history=1 so K > 1 and K = 1 runs share one set. Model: MLPPolicyStem gains history_len / history_dropout. For history_len > 1 it owns a zero-init learned time embedding added after the projection, and HPT.stem_process skips the input-side sinusoid for such a stem (a constant [0, 1, 0, 1, ...] on normalized values is amplitude-1 noise on [-1, 1] data). history_dropout replaces the whole past by the current step per sample in training, exactly what an episode-start sample looks like after front padding. The K = 1 baseline stays one override away (proprio_history=1, history_stride=1, history_len=1) and the tests pin that it is bit-identical. Every other shipped config is untouched at K = 1, including the keypoint arm, whose wrist-frame proprio would need per-step framing before it can carry history. A compose test pins every train_*.yaml recipe's data K against its stem's history_len, because a recipe that inherits one model's data config and overrides the model is exactly how the two drift apart, and nothing else composes both halves. _stem_history_len takes the domain rather than matching a stem name by suffix: a cotrain run registers one state_ee_pose stem per domain, and a suffix match returns whichever comes first, so the human side was checked against eva's history_len. Latent while everything shipped K = 1; a blocker now. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VBmhpVy7ZcMtsBJWh38cJp
… padding `HPTModel.finalize_modules` ran `self.apply(self._init_weights)` over the whole model, which xavier-randomised the pretrained HF encoders inside the text stems (the frozen Qwen3-Embedding weights were overwritten in place and their biases zeroed). Modules that own pretrained weights now declare them through the `PretrainedWeights` mixin, and the init pass walks the tree without ever entering such a subtree, so the ResNet backbone, `T5Encoder` and any future pretrained stem are covered by the same mechanism. `HPT.__init__` also called `finalize_modules` twice; the first call is gone. Once the model is built, moved and upcast, `verify_pretrained_weights` hashes every declared pretrained submodule and compares it with the checkpoint read straight off the local HF snapshot, failing the run on a mismatch. It costs a second full copy of the encoder (~1.1 GB fp16 at 0.6B) and ~3 s of hashing per rank, so `verify_pretrained: false` turns it off; it is on by default, because a silently re-initialised encoder trains to a plausible wrong result. Only stems that own an HF snapshot are checked -- `ResNet` returns no reference, so it is protected from the init pass but not hash-verified, which the mixin docstring now says. Finally, `QwenPerTokenEncoder.compute_latent` now passes the tokenizer attention mask into its cross-attention. Padded positions are zeroed before the biased `proj`, so they were the learned constant `proj.bias`, not zero, and got real attention weight: text-conditioned results from before this commit are not comparable with results after it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VBmhpVy7ZcMtsBJWh38cJp
The annotation vocabulary is tiny next to the number of training steps, so a frozen text encoder re-runs the same 600M-parameter forward every step. `_Qwen3BaseEncoder` now keeps a lazy in-memory cache of the per-token hidden states keyed by the exact prompt string: misses go through the tokenizer + encoder in ONE sub-batch of unique strings, their un-padded (L_i, D) rows are stored detached, and the batch is re-assembled with the same LEFT padding the tokenizer produces, so `forward_with_mask` / the pooled path are unchanged. Only legal when the encoder is frozen - a trainable encoder must stay online. Knobs on the stem: `cache_text_features` (default true) and `text_cache_max_entries` (default 8192, LRU); both added to the model yamls with a Qwen stem. The cap counts entries, and an entry is the prompt's un-padded (L, D) states -- 4-25 KB for real annotations, 262 KB at max_length 128 -- so the default is ~100-200 MB of device memory in practice and 2 GB in the worst case; the docstring says so next to the knob. A batch carrying more unique prompts than the cap reads its rows from the miss pass rather than from the cache, because the LRU trim runs first and would otherwise evict entries the very next line looks up. `_apply` (to/cuda/float) clears the cache, and `clear_text_feature_cache()` / `text_feature_cache_stats()` are exposed. Padded positions come back as zeros instead of the encoder's pad-token output; every consumer masks them out. Cached-vs-online on 50 real prompts (fp16, each warmed alone: worst-case padding mismatch): max |diff| 3.6e-3 on the latents and 9.4e-2 on raw hidden states, exactly equal to the online path's own batch-padding noise (9.4e-2); in fp32 the same comparison is 1.1e-6 / 1.6e-4. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VBmhpVy7ZcMtsBJWh38cJp
HPT.__init__ defaulted self.device to torch.device('cuda') and moved the
freshly built policy there. Under submitit every rank sees all 8 GPUs, so all
8 ranks resolved that to cuda:0 and built their policy on the same device:
8 x 3.34B, which OOMs during construction. The 1B arm fit and survived, which
made the failure look config-specific.
PI already builds on CPU and leaves self.device unset for Lightning to fill in
through ModelWrapper.on_fit_start / on_validation_start. HPT now does the same:
no device is chosen in __init__ and nothing leaves CPU there.
Lightning's write then has to reach the HPTModel the algo holds. That model
reads self.device to build the loss accumulators in compute_loss, while
HPT.compute_losses accumulates on the algo's, so a write that stopped at the
algo would leave the two on different devices -- 'Expected all tensors to be
on the same device, but found at least two devices, cuda:0 and cpu!'.
HPT.device is therefore a property whose setter also writes
nets['policy'].device, so the two cannot diverge through the path the trainer
uses.
Verified on 8xH100 with the 3.34B hpt_3b arch-comparison arm: every rank
reports CUDA_VISIBLE_DEVICES [0..7], all 8 reach fit start and train an epoch
without an OOM.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EAGuZKPVfr2J9L7W9gtxET
The eval-time seeding (eval_base_seed + rank * EVAL_RANK_STRIDE + pass counter, the prompt sampling and the seeded generator) was 96 lines inside HPT, though nothing in it is HPT-specific: it reads only nets, device, eval_base_seed, annotation_key, annotation_sampling_mode and default_prompt. Moving it to egomimic/algo/eval_determinism.py as a mixin leaves hpt.py with the algo and gives the seeding contract one documented home. The attributes it reads are annotated, not defaulted: a class-level default would turn "the algo forgot to set this" from an AttributeError into a silent wrong value. The module docstring says plainly that this is HPT-only today -- PI mixes in nothing and has no eval seeding at all, so pi0.5 validation is still not reproducible -- and what wiring PI up would take. Pure code motion -- no behaviour change, the eight method bodies are byte-identical to the originals, and the existing determinism tests cover it unchanged. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VBmhpVy7ZcMtsBJWh38cJp
_video_fps divided 30 fps by world_size to undo a DistributedSampler stride that never applies: the val heads are CombinedLoaders Lightning does not shard (every rank runs every val batch; 1 rank = 223 val steps, 8 ranks = 8 x 223 in the 2026-09-16 DDP check), so rank 0 renders every frame and the 8-GPU videos came out as 8x slow motion (416 frames at 4 fps). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01THTuWdgk6M6pZ5Ya7ihZEF
On the lab's 8 x A40 nodes -- no NVLink, two NUMA islands joined by cross-socket links -- the per-step gradient all-reduce, not the math, sets the step time once a model is large enough. Measured 2026-09-16 on a ~1.2 B -parameter policy: 6.4 s/step at 8 x batch 16 against 1.4 s/step on a single GPU, and 3.6 s/step with the reduction in bf16. Only the reduction is bf16; master weights and the optimizer stay fp32. It is plain DDP otherwise -- no unused-parameter detection -- so a model that leaves a trainable tensor without a gradient needs ddp_pi's `ddp_find_unused_parameters_true` instead and will crash under this one. The config says so, because nothing else would stop you selecting it for Pi. The hook is a zero-argument factory rather than a Hydra `_partial_` because Lightning logs the hook's __qualname__, which a functools.partial does not carry. `make_bf16_compress_hook` is named for what it is: torch's own hook is `bf16_compress_hook`, and sharing that name made the factory look like the hook. test_eval_determinism's trainer parametrization now enumerates the config group instead of a hardcoded list, so a new trainer config cannot land without the gradient-clip knob being checked. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01THTuWdgk6M6pZ5Ya7ihZEF
…plit
data/mecka_fold_flagship_{opsplit,top3}_hpt_keypoints: the 144-D wrist-frame
MANO keypoint twins of the 6D flagship configs -- keymap_mode keypoints,
mode keypoints_wristframe_6d, no observations.state.ee_pose (so
pad_proprio_gripper is off); everything else, including the operator split,
the val loaders and the video episodes, is inherited.
train_zarr_mecka_flagship_kp_hpt pairs them with hpt_bc_keypoints_wrist_300M
and eval_hpt_keypoints_wrist, for both the val evaluator and the train-time
overlay.
hpt_bc_keypoints_wrist_300M gains the history_len / history_dropout stem knobs
and backbone_lr_scale, so its lanes match hpt_bc_mecka_6d_300M's; both are
inert at their defaults.
The tests check one thing composition cannot: `history_len` and the keymap's
`proprio_history` are a single setting living in two files, and HPTModel
raises when they disagree, so each data config this model pairs with is
checked against the stem that consumes it.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01THTuWdgk6M6pZ5Ya7ihZEF
_viz_keypoints flattened a (T, D) chunk to (T * 21, 3) and then drew slots 0..20 only -- `for k in range(21)` for the dots, and edge_ranges over the same 21 -- so an action chunk collapsed to its first timestep and the overlay never moved, however long the horizon. Every projected point past frame 0 was computed and thrown away. That is what a whole training campaign's validation videos turned out to be (2026-09-18). This replaces it with _viz_keypoints_horizon_trace, which draws the chunk as a trajectory and degenerates to per-slot start markers for a lone timestep (it promotes (D,) to (1, D) itself). The old renderer is deleted rather than left reachable: Human.viz(mode="keypoints") has one path, and a renderer you can reach by accident is one you will reach. Consequences, all intended: the dataset inspector, the Aria tutorial and mecka_keypoints_viz render through the trace now, so the single-frame callers show per-slot start markers instead of a posed skeleton, and Aria's raw 21-keypoint edge tables go with the renderer that consumed them -- they had no other caller. tests/unit/test_chunk_renderer.py pins both halves: that the trace follows the chunk, and that no second renderer is reachable from Human. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01THTuWdgk6M6pZ5Ya7ihZEF
PoseCoordinateFrameTransform gains per_step_target: with target_history K, step k's keypoints are expressed in step k's OWN wrist frame instead of the current step's. Re-expressing older steps in the current frame smears wrist motion into the articulation channels, and the palm-rigid MANO knuckles (1/5/13/17) have a very tight legitimate range, so they land far outside their per-cell bounds and worsen with age: measured 2026-09-17, 10.3 / 6.9 / 3.3 out-of-range cells per sample at history steps 0/1/2 against 0.0 at the current step. That cost 22,500 loader retries on one run and starved rank 0 into an NCCL watchdog kill. Per-step framing keeps every step a genuine state -- keypoint 0 is 0 at every step, so the norm stats and the bounds check describe all of them. The ACTION chunk keeps the current-step frame: its leading axis is the forward horizon, not history. It is the default, and it is inert unless the target itself was read with history and the pose carries the matching axis -- a keymap without proprio_history never emits that shape, so no existing call site changes. For anything that does, per-step is not a tuning choice: under the shipped K = 3 / stride 5 the oldest step is 10 frames back, where the measured wrist translation is 80.8 mm rather than the ~17 mm that already pushed 67 of 126 channels out of band. The flag remains as an escape hatch. Both halves are tested: the origin property at K=4, and bit-identical output at K=1 either way. Nothing shipped exercises this yet: no data config pairs proprio_history with a keypoint keymap, and the flagship keypoint configs are pinned at K = 1 because they sit below this commit. It is enablement for the keypoint history arm, not a fix to a live run. The per-step loop is also a Python loop over K (measured 59.5 -> 165.4 us per call at K=3, twice per sample); worth vectorising if that arm is ever configured, not before. CartesianYPRToRot6D also picks up the note about applying it to the proprio ee_pose, which the embodiment comment used to carry. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VBmhpVy7ZcMtsBJWh38cJp
Three rungs, identical in everything but the train operator set, so an arm's only variable is how many operators its data came from: k=1 9 eps / 25633 f / 0.237 h 1 operator k=2 24 eps / 60374 f / 0.559 h 2 operators (nested) k=4 48 eps / 118388 f / 1.096 h 4 operators (nested) Operators are ranked by ANNOTATED hours (not raw hours) and nested, so k=2's set contains k=1's. Shared by all three: the 3-episode seen-op val (drawn from operator #1, which every rung trains on) and the 35-episode unseen-op val. The train split is an operator FILTER, not an episode pin, so the counts above are hand-measured documentation -- test_opscale_ladder.py drives the real DatasetFilter with synthetic rows and asserts the nesting and the exclusions, which are the parts that live in the config. The metric budget is 'auto': trainHydra derives K = floor(limit_val_batches * batch_size / episodes) per head from the resolved split, so K follows the live filters instead of a hand count that goes stale when SQL grows (or when limit_val_batches or the batch size change); limit_val_batches=0, which the bench scripts use to turn validation off, subsamples nothing. Explicit ints keep their meaning, so the flagship recipes that pin K are untouched. At the 2026-09-17 counts auto reproduces the old numbers (1706 / 146 / 106, 213, 568). Each rung has its own train distribution, so norm stats are per-arm; the 6D and keypoint variants of a rung share filters, loader params and budgets and differ only in keymap and transform. The ladder itself is edited in one place, mecka_fold_freeform_opscale4_6d, which the other five inherit from. No proprio_history: it is half a setting (the model's stem history_len is the other half, and HPTModel raises when they disagree), and every model in the repo is history_len 1, so a rung asking for more is a rung nothing can train. A test pins that, since composition cannot see it. The annotation guard is written once in the parent (a YAML anchor covers its three splits). mecka_fold_freeform_opsplit_6d, whose held-out operator list the ladder reuses, has no such guard and stays frozen; its header now says so, and HPT warns once when a sample falls back to an empty default_prompt, as Pi does. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01THTuWdgk6M6pZ5Ya7ihZEF
ModelCheckpoint with `save_last: true` calls _save_checkpoint a second time for last.ckpt, so every checkpoint event writes the same tensors twice -- ~21 GB a time on Pi 0.5 (the size tests/integration/test_train_step_gpu.py records). It was measured at 9 GB and ~1.3 min on the now-retired ABC-DiT-L, inside an event that blocked the run for 2.7-3.4 min of a 42.3-min 1000-step cycle. "link" writes a relative symlink to the epoch checkpoint just saved instead. Safe here because save_top_k is -1, so the target is never collected, and _should_remove_checkpoint returns False when previous == current, which is always the case for last.ckpt. Resume is unaffected: _requeue_resume_path gates on os.path.isfile, which follows symlinks, and the opscale launcher already resumes through a hand-made symlink. A link can dangle where a file could not: the target deleted by hand, last.ckpt dropped by an rsync without -l, or a preemption between Lightning's unlink and relink. isfile is False in all three, and the requeue path used to warn and silently restart the run from scratch. It now raises when last.ckpt dangles or is missing next to other checkpoints; only an empty checkpoint dir falls back. One behaviour difference worth knowing: with "link" the callback relinks on EVERY train epoch once any checkpoint exists, not only on checkpoint epochs, so there is an extra os.remove + os.symlink + strategy.barrier per epoch. The test drives a real two-epoch fit and asserts both halves: "link" leaves a relative symlink to the newest epoch checkpoint, and `true` leaves a regular file the same size as it -- the copy this removes. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01M5dTGNydAZbRP1EsXgbp4W
Two problems on the <head>_video loaders, one of them a correctness bug. EvalVideo._write_video had no rank guard. The val heads come back from val_dataloader() inside CombinedLoaders, which Lightning does not wrap in a DistributedSampler, so every rank iterates every val batch and renders the same frames -- and every rank then wrote the same mp4 path. The file was whichever rank finished last, and could be torn. Separately, the video loaders were iterated in full on every validation pass. ModelWrapper.validation_step already returned before the forward on a non-viz pass, but Lightning had built the batch by then, so limit_val_batches x 64 pinned frames were decoded and discarded on every rank on the ~90% of passes that render nothing: ~1.7 min a pass, ~4% of a 1000-step cycle. _VizGatedLoader yields nothing when its gate is closed, which is on a non-viz pass and on every rank but 0. The val loaders are built once (_EvaluationLoop.setup_data returns early once _combined_loader is set), so the gate cannot be a length; it is an empty iterator, which _Sequential already reads as "move to the next loader". __len__ answers WITHOUT creating an iterator, because creating one forks that loader's workers and starts them prefetching -- doing the skipped work at setup instead, and never freeing it. It asks a fresh _MaxSizeCycle for the length, which stores its loaders and limits and builds no iterator until __iter__ (lightning 2.6.1). Ranks therefore iterate different numbers of val batches. That is safe because what they LOG is not: the metric loaders run everywhere and log the same keys, the video loader logs nothing, so the sync_dist reduction stays symmetric. It does mean the other ranks sit in that reduction while rank 0 renders -- 8.4 min for 76 batches back when all four ranks rendered at once, so well inside the 30-minute process-group timeout, but that timeout is now the ceiling on evaluator.viz_max_batches. trainHydra therefore refuses a head that has pinned video episodes and an evaluator with viz_max_batches=None. Every recipe that pins video episodes already caps it at 10; the evaluators that leave it unset (eval_hpt, eval_hpt_cartesian, eval_hpt_wrist) pair with configs that pin nothing, so they keep the ungated path where every rank renders together, as before. A desync here would hang a campaign rather than fail it, so it is tested against real DDP rather than argued: tests/fixtures/ddp_video_gate.py runs a 2-process gloo fit with the same collective pattern, at num_workers > 0 so the prefetch is visible. Both ranks finish; rank 0 steps the video loader on the viz epoch, rank 1 never touches the pinned episode. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01M5dTGNydAZbRP1EsXgbp4W
SaveAanchal
force-pushed
the
aanchal/mcap-to-lance
branch
from
October 21, 2026 06:27
eaa8bb1 to
47ebba9
Compare
RyanPCo
changed the base branch from
aanchal/mcap-validator
to
graphite-base/719
October 22, 2026 02:03
RyanPCo
force-pushed
the
aanchal/mcap-to-lance
branch
from
October 22, 2026 02:04
47ebba9 to
4815a4f
Compare
RyanPCo
changed the base branch from
graphite-base/719
to
ryanco/lance-annotations-stage
October 22, 2026 02:05
Leaves are keyed by recording_id. A recording is cut into segments, one per tasks row plus (with include_taskless_frames) the untasked gaps; filter lambdas run per segment, so a recording with several tasks or aliases is one leaf instead of duplicate catalog rows where the last one won. Pins and excludes take a recording_id or an old clip episode_hash, which narrows or subtracts that clip's time range. pinned_episode_subset renders a pinned clip alone. Claude-Session: https://claude.ai/code/session_01Kt1E8T2x8kdV9akAvbJyVd
Action chunks end at the anchor's segment end by default (repeat-last padding, flagged in action_pad_mask); action_clamp: recording runs them on. History reaches back into earlier segments by default; history_floor: task pads and masks it at the segment start, through a new EpisodeLeaf._history_floor hook. Claude-Session: https://claude.ai/code/session_01Kt1E8T2x8kdV9akAvbJyVd
require_hands_in_frame (none | either | both | left | right, default both) picks the anchors: the named hands of the embodiment valid over the whole action chunk, plus the head pose at the anchor unless none. Samples carry action_hand_mask ([left, right] valid over the real chunk) and proprio_hand_valid ([left, right] per proprio frame) for the losses, stats and bounds check. Claude-Session: https://claude.ai/code/session_01Kt1E8T2x8kdV9akAvbJyVd
…unds Samples with action_hand_mask / proprio_hand_valid (Lance leaves) now: - HPT/RDT (DenoisingPolicy) multiply the pad mask by the hand's channels of the bimanual layout, whatever use_pad_mask says; aux heads see the same flags. - Pi takes a masked mean over the converter's packed slots; slots the converter inserts (gripper zeros, padding) stay in, so an all-tracked batch gives today's loss. - Norm stats set untracked cells to NaN and use nan-aware stats per cell, in both the DataLoader and the per-episode cache paths; the bounds check skips those cells. - Proprio of an untracked hand reads as the dataset mean pose: the normalized mean, so 0 on normalized channels and the mean on rot6d pass-through ones. - HPT and Pi eval take an untracked hand's ground truth from the prediction, so it adds no error and the viz revert never sees the zero fill rotation. Claude-Session: https://claude.ai/code/session_01Kt1E8T2x8kdV9akAvbJyVd
Every run writes split.json (train/valid dataset keys per train dataset) and, with Lance datasets, lance_versions.json (table versions and their tags) next to git_sha.txt; a requeue or resume that differs keeps the first file and writes a timestamped one. Resolving an untagged Lance version warns that the run is reproducible only until cleanup_old_versions drops it. Resuming from a checkpoint whose run validated on episodes this run trains on warns. Claude-Session: https://claude.ai/code/session_01Kt1E8T2x8kdV9akAvbJyVd
t_end_ns was the last frame's timestamp, so under the half-open interval convention the last frame of every recording fell outside its task and alias, and the loader never sampled it (48 frames on the pilot tables). Claude-Session: https://claude.ai/code/session_01Kt1E8T2x8kdV9akAvbJyVd
steps/eva stores each arm's measured and commanded EE pose, joints and gripper under the zarr key names, plus left_valid/right_valid for forward-filled dropout frames. The loader maps zarr keys to columns per lab, treats a validity flag the table lacks (robots have no head) as always true, and knows the eva_* arm sets. tag, compact and cleanup now cover every lab's steps table. Claude-Session: https://claude.ai/code/session_01Kt1E8T2x8kdV9akAvbJyVd
One HDF5 demo becomes one recording with one task over it. Numerics go through EvaHD5Extractor and eva_to_zarr's per-arm split, so bimanual samples match zarr to float32 precision; all-zero pose frames stay forward-filled and are flagged invalid per arm. Frame times are the demo start plus i/30 s, since the HDF5 has none. Cameras are encoded at 480x360 AV1. A one-arm robot's data sits in its own slot of the (T, 14) arrays; eva_to_zarr reads a lone right arm from the left slot (zeros), so this splits both slots and keeps the robot's arms. The pilot committer takes --lab eva and reads catalog rows from meta.json. check_eva_samples compares Lance and zarr samples. Claude-Session: https://claude.ai/code/session_01Kt1E8T2x8kdV9akAvbJyVd
LanceStreamingDataset now reads frames straight from the MP4s: each process reads an MP4's sample table once (one range GET of its moov), then a batch fetches only the packets its frames need (keyframe through frame; adjacent with GOP 2) in one concurrent round of range GETs and decodes them in memory with PyAV. Frames match torchcodec bit for bit. On R2 with 8 workers this ties the K=8 chunk tables on raw reads (277 vs 276 samples/s steady state) and runs 5-15% behind through the full loader, with video stored once and nothing for the committer to build. So video_chunks tables, chunk_videos.py and --chunk-k go. The chunk path had been broken since leaves were keyed by recording_id (row.episode_hash). Claude-Session: https://claude.ai/code/session_01Kt1E8T2x8kdV9akAvbJyVd
The committer (egomimic/scripts/lance_commit.py, moved from lance_pilot/commit.py) is the only writer of the Lance tables. Converters stage deliveries under <staging>/<lab>/<vendor id>/ with meta.json uploaded last (commit.stage; --stage-to on aria_to_lance, eva_to_lance and the Ray driver, which no longer commits). Each run validates complete deliveries, commits them in batches, records each batch in a new changes table and tags every table <lab>-<UTC time>, then deletes their staged files. A delivery that fails validation gets REJECTED.json with its errors and a digest of its files and is skipped until they change; one already committed waits for --replace. commit.py works on local paths and s3:// roots alike (R2 credentials from the environment). Frame counts come from the MP4 index, so the committer no longer needs torchcodec. Six concurrent appends on R2 all land with the expected version count (tests/integration/test_lance_r2). Claude-Session: https://claude.ai/code/session_01Kt1E8T2x8kdV9akAvbJyVd
The format bench still wrote annotations into a tasks.lance column that the annotations table replaced, so its Lance tests failed on import. Claude-Session: https://claude.ai/code/session_01Kt1E8T2x8kdV9akAvbJyVd
lance_pilot/annotations.py becomes egomimic/scripts/lance_annotations.py and works on s3:// roots. from-zarr (was migrate-zarr) maps the zarr backlog through each episode's frame timestamps; from-bucket maps scale_to_bucket span lists through aliases.lance. Commits go through commit.commit_annotations, which records a changes row (lab "annotations") and tags every table like a recording batch. Claude-Session: https://claude.ai/code/session_01Kt1E8T2x8kdV9akAvbJyVd
lance_annotations.py stages a reviewed change file under <staging>/annotations/<sha256 prefix>/ (meta.json last) instead of committing it, and lance_commit --lab annotations commits each staged change, oldest first, after reviewing it again against the current tables. One that no longer applies gets REJECTED.json and waits until it is restaged. lance_commit is now the only writer of the Lance tables. changes gains a provenance column (JSON): a delivery batch records each recording's meta.json provenance, an annotation commit its approver and source file. A changes table written before the column existed gets it added in place (old rows read null). Claude-Session: https://claude.ai/code/session_01Kt1E8T2x8kdV9akAvbJyVd
What vendors deliver (one raw episode_<id>/episode.mcap per recording, plus optional annotation.mcap) and the check every delivery goes through before conversion to Lance: - SPEC.md: the vendor-facing spec; spec.py: the same rules as data (human / robot topic profiles, episode-info keys, frames, thresholds); schemas.py: the egoverse.* message types, rendered to proto/egoverse.proto for vendors. - validate.py: `python -m egomimic.rldb.mcap.validate episode_<id>/...` checks presence, video, clocks, calibration, IMU, poses and task/label spans, and prints the Lance recording_id each episode will get. - tests: synthetic episodes with one switch per mistake (each must report exactly its problem), proto/enum sync, and an integration run on real episodes. - pyproject: mcap, mcap-protobuf-support, foxglove-schemas-protobuf; ships the .proto. Checked on two real Aria recordings converted to the spec: both pass, and every column of aria_to_lance's steps rebuilds from the MCAP alone. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…er trips on Found in review; each passed validation and then failed in conversion: - a camera whose -info / -info-extra name another frame than its video frames (conversion multiplied through the wrong frame). The video, -info and -info-extra must now name the same frame. - a head-worn rig's steps camera not rigidly mounted on the head (e.g. its calibration in `world`): the converter places the steps through a /tf_static path from `head` to that camera, which is now required. (A frame placed nowhere is still reported once, as having no path to world.) - a NaN / Inf /tf_static translation (conversion: "SVD did not converge"); only the rotations were checked. Translations must now be finite. SPEC.md states the three rules. Both real Aria test recordings still validate clean. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`python -m egomimic.scripts.mcap_process.mcap_to_lance --lance <root> episode_<id>/...` validates each episode, checks its labels against the tasks it will have (Lance's own review rule, so nothing is half-committed), builds one steps row per frame of the steps-clock camera, writes one video per camera, and commits the recording, its tasks, alias and labels. Everything comes from the MCAP; the episode info replaces the SQL lookup of the Aria pilot. `convert_and_commit` is the same per episode, returning a Result (status, problems, frames, videos, valid fractions) for a pipeline to record. Re-runs skip committed recordings but still bring their labels up to date; --replace swaps a re-export in and drops its vendor's removed labels. Per-frame math follows aria_to_lance: filler rows for dropped frames, head pose as the camera pose turned upright (-info-extra), hands placed only with a usable head pose, MANO keypoints from the Aria fit (vendor joints mapped by hand-joints name), validity from tracking, timing and pose jumps. Matching takes the true nearest sample: projectaria's search, used by aria_to_lance, lands one sample off on about half the frames. Times stay Unix ns. The steps camera gives the front_1 / front_1_480 pinhole outputs (Aria keeps ARIA_INTRINSICS, other sources their field of view); every other camera keeps its own video, intrinsics and extrinsics. Human embodiments only; Lance has no robot steps schema yet. commit.Delivery.task becomes tasks: a recording can hold several tasks. Checked on two real Aria recordings against aria_to_lance (--no-mano): same rows on the same clock, and equal values wherever both picked the same samples. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
lance_commit is the only writer of the Lance tables, so the converter now stages: convert_and_stage writes each episode with commit.stage_delivery under <staging>/<lab>/<vendor recording id>/, carrying its catalog rows (recordings/tasks/aliases.parquet) and labels (labels.parquet) with it, meta.json last. lance_commit loads such "format: delivery" stages without the Aria SQL lookup and, after each batch, syncs their labels in one annotation commit (the label vendors' committed labels of that recording not among them are deleted, the rule convert_and_commit applied directly). Labels delivered for a recording already committed are staged as an annotation change. The CLI takes --stage-to; --lance is only read, to skip committed recordings; --tag goes, since the committer tags every batch. The alias now ends one period after the last frame (half-open), as aria_to_lance does since aria-task-end. Claude-Session: https://claude.ai/code/session_01Kt1E8T2x8kdV9akAvbJyVd
RyanPCo
force-pushed
the
ryanco/lance-annotations-stage
branch
from
October 22, 2026 14:06
8760d97 to
e80a7f6
Compare
RyanPCo
force-pushed
the
aanchal/mcap-to-lance
branch
from
October 22, 2026 14:06
4815a4f to
dca8ff9
Compare
RyanPCo
force-pushed
the
ryanco/lance-annotations-stage
branch
3 times, most recently
from
October 23, 2026 20:03
6c62f33 to
5c415cf
Compare
This was referenced Oct 27, 2026
RyanPCo
force-pushed
the
ryanco/lance-annotations-stage
branch
from
October 30, 2026 18:27
5c415cf to
8dced58
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Update (Ryan): stages instead of committing
Rebased onto #734 (
ryanco/lance-annotations-stage), solance_commitis the only Lance writer:convert_and_commit→convert_and_stage: each episode is staged withcommit.stage_delivery(steps, videos, recordings/tasks/aliases/labels parquet,meta.jsonlast) under<staging>/<lab>/<vendor recording id>/.lance_commitloads these without the Aria SQL lookup and syncs their labels after each batch (same rule as before: the label vendors' labels not in the delivery are deleted).--stage-to(required);--lanceis read-only;--tagremoved (the committer tags each batch).ts[-1] + period), as inaria-task-end.Delivery.task→tasksmerged with our committer (scripts/lance_commit.py, moved fromlance_pilot/commit.py).Your original commit is unchanged; the rework is the commit on top.
Stacked on #718, which is on #717.
What
Per episode, the converter:
Everything comes from the MCAP, with no SQL lookup.
convert_and_commit()does the same for one episode and returns aResult(status, problems, frames, videos, valid fractions). The ingestion pipeline in ingest: vendor buckets → conversion jobs → staff review → lance_commit (store, watcher, AWS deploy) #720 builds on the same pieces, but stages the converted result and commits it only after staff approval.--replaceswaps in a re-export.aria_to_lance:front_1/front_1_480; every other camera keeps its own aligned video, intrinsics and extrinsics.Change to #717:
commit.Delivery.task→tasks: list[dict], so a recording can hold several tasks. The callers are updated.Tests
tests/unit/test_mcap_to_lance.py: synthetic episodes end to end into a temporary Lance root.tests/integration/test_mcap_to_lance_aria.py: two real Aria recordings, compared witharia_to_lance --no-manooutput:CI note: CI has no
pylanceinstalled, so these tests and #717's Lance tests skip there; they pass locally. Unit failures already on #717 (theformat_bench, Piinfra_port_wiringand JPEG coalescing tests) are not from this stack.For @RyanPCo
get_nearest_*binary search lands one sample off the true nearest on about half the frames. That's up to 33 ms for hands at 30 Hz. This converter takes the true nearest. Switchingaria_to_lanceto a true nearest search would make both paths produce identical rows.aria_to_lancestores device time.ARIA_INTRINSICSputs the principal point at (320, 240), while the linear camera used for undistortion centres at (319.5, 239.5). I kept your values here so both paths agree.--mano-model-dir), so a deployment doesn't need the license-gated files inside the code checkout.🤖 Generated with Claude Code