fix(sft): load_video call sites omit the required use_audio_in_video and keep the raw 4-tuple - #295
Open
Anai-Guo wants to merge 1 commit into
Open
fix(sft): load_video call sites omit the required use_audio_in_video and keep the raw 4-tuple#295Anai-Guo wants to merge 1 commit into
Anai-Guo wants to merge 1 commit into
Conversation
…loaders `load_video(video, use_audio_in_video, **kwargs)` takes `use_audio_in_video` as a required parameter and returns a 4-tuple `(video, video_meta, audio, audio_meta)`. Both SFT media loaders called it as `load_video(p)`, which raises `TypeError: load_video() missing 1 required positional argument: 'use_audio_in_video'` and, once that is fixed, would still store the whole 4-tuple where downstream expects a `torch.Tensor`. Follow the canonical usage in `relax/utils/multimodal/process.py`, which unpacks the tuple and takes the video tensor, and use `use_audio_in_video=False` to match the default every other definition in the repo already uses.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Fixes the two SFT multimodal media loaders, which cannot load a video at all today:
relax/engine/sft/dataset/multimodal.py::_fetch_media(train path)relax/engine/sft/predict/loop.py(predict path)Both called
load_video(p).Why
load_videorequiresuse_audio_in_videoand returns a 4-tuple:So both call sites are wrong in two ways:
TypeErrorimmediately. Any SFT sample with a non-emptysample.videosfails, so SFT video training/predict is unreachable.would land in
mm_inputs["videos"], where downstream code expects a tensor:relax/utils/data/processor_pool.py:69→vid.contiguous().share_memory_()relax/engine/rollout/sglang_rollout.py:277→async_encode_video_tensor_for_rollout_engine(video)The two sibling loaders on the adjacent lines are both fine, which is why this
only affects videos —
load_image(image, **kwargs)andload_audio(audio, config=None, **kwargs)are happy with a single positional argument.Reproduced against the current
mainsources by extracting the three realsignatures and replaying the calls, with the two working loaders as a control group:
How
relax/utils/multimodal/process.py:79already contains the correct usage of thesame loader, and it is the spec this PR follows — unpack, keep the video tensor:
so both call sites become:
use_audio_in_video=Falseis chosen deliberately rather than invented:agentic/pipeline/runtime.py:3448,models/qwen_omni/.../model.py:47,models/qwen_omni/.../utils.py:69,models/qwen_omni/qwen3_omni_provider.py:56,utils/data/data.py:44,utils/data/data_utils.py:328;args/configin scope to thread a real value from, andaudiosseparately fromsample.audios, so pullingthe audio track out of the video here would double-count it.
Happy to thread
args.use_audio_in_videodown into the SFT dataset instead if you'dprefer that shape — it is a larger change, so I kept this PR to the bug.
Testing
ruff checkandruff format --diffclean on both files with the pinnedv0.15.9from.pre-commit-config.yaml, run from the repo root so thepyproject.tomlconfig applies (All checks passed!/2 files already formatted).available to run a full SFT step, so this verifies the arity and the unpack,
not the downstream tensor semantics.
Type of Change
🤖 Generated with Claude Code