Skip to content

Return one frame per requested timestamp in the torchvision_av backend - #67

Open
rakhimovv wants to merge 1 commit into
BeingBeyond:mainfrom
rakhimovv:fix/torchvision-av-frame-count
Open

rakhimovv wants to merge 1 commit into
BeingBeyond:mainfrom
rakhimovv:fix/torchvision-av-frame-count

Conversation

@rakhimovv

Copy link
Copy Markdown

Thanks for releasing this — a note on one difference between the video backends.

Summary

  • get_frames_by_timestamps documents its return as "Frames as numpy array of shape (N, H, W, C)"
    for N requested timestamps. The decord and opencv backends return exactly that; the
    torchvision_av backend seeks to the nearest preceding keyframe and returns every frame it
    decodes up to the last requested timestamp.
  • configs/posttrain/libero/libero_all.yaml and configs/posttrain/so101/so101_example.yaml both
    set video_backend: "torchvision_av", so this is the default path for the shipped configs.
  • This PR brings the function in line with the copy in
    tutorials/Being-H-EDU/BeingH/utils/video_utils.py, which already carries per-timestamp matching
    and a length check.

Root Cause

The extra frames reach the model because BeingH/dataset/datasets/vla_dataset.py turns each returned
frame into its own ViT image:

for frame in view_group:
    image_tensor = self.vit_transform(Image.fromarray(frame)).unsqueeze(0)
    ...
    packet['num_tokens'] += self.num_image_tokens + 2

With the default obs_indices=[0], the dataset asks for exactly one frame per view per sample, but
the number it gets back — and therefore the number of image tokens packed into the sequence —
depends on how far the sample is from a keyframe.

Measured on two LeRobot datasets, requesting a single timestamp at frame 0, 5, 20, 50 and 100:

dataset (codec) frames returned before after
Being-H-EDU_SO101/pick_cube_plate_trimmed (h264) 1, 6, 21, 51, 101 1, 1, 1, 1, 1
IPEC-COMMUNITY/libero_spatial_no_noops_1.0.0_lerobot (av1) 1, 2, 1, 1, 1 1, 1, 1, 1, 1

The size of the effect tracks keyframe spacing in each encode, so it is pronounced on the SO101
videos and mostly invisible on these LIBERO ones.

Test plan

  • Single-timestamp requests return exactly one frame on both datasets above (table).
  • 10-step LIBERO post-training run on libero_spatial (Being-H05-2B + InternVL3_5-2B +
    Qwen3-0.6B, 1×H100) before and after the change: both complete, checkpoint and
    libero_posttrain_metadata.json written as expected. This is a no-regression check — the
    frame-count table above is what demonstrates the fix.

The torchvision_av branch of get_frames_by_timestamps seeked to the
nearest preceding keyframe and returned every frame it decoded up to the
last requested timestamp, while the decord and opencv branches return one
frame per requested timestamp as documented.

vla_dataset turns each returned frame into its own ViT image, so the
number of image tokens packed per sample varied with distance from a
keyframe. Apply the per-timestamp matching and length validation already
present in the Being-H-EDU copy of this function.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant