Skip to content

fix(sft): load_video call sites omit the required use_audio_in_video and keep the raw 4-tuple - #295

Open
Anai-Guo wants to merge 1 commit into
redai-studio:mainfrom
Anai-Guo:fix/sft-load-video-arity
Open

fix(sft): load_video call sites omit the required use_audio_in_video and keep the raw 4-tuple#295
Anai-Guo wants to merge 1 commit into
redai-studio:mainfrom
Anai-Guo:fix/sft-load-video-arity

Conversation

@Anai-Guo

Copy link
Copy Markdown

What

Fixes the two SFT multimodal media loaders, which cannot load a video at all today:

  • relax/engine/sft/dataset/multimodal.py::_fetch_media (train path)
  • relax/engine/sft/predict/loop.py (predict path)

Both called load_video(p).

Why

load_video requires use_audio_in_video and returns a 4-tuple:

# relax/utils/multimodal/video_utils.py:361
def load_video(
    video: VideoInput,
    use_audio_in_video: bool,          # <- required, no default
    **kwargs: Any,
) -> Tuple[torch.Tensor, Dict[str, int], Optional[np.ndarray], Optional[Dict[str, int]]]:

So both call sites are wrong in two ways:

  1. They raise TypeError immediately. Any SFT sample with a non-empty
    sample.videos fails, so SFT video training/predict is unreachable.
  2. Even with the argument added, the shape is wrong — the whole 4-tuple
    would land in mm_inputs["videos"], where downstream code expects a tensor:
    • relax/utils/data/processor_pool.py:69vid.contiguous().share_memory_()
    • relax/engine/rollout/sglang_rollout.py:277async_encode_video_tensor_for_rollout_engine(video)

The two sibling loaders on the adjacent lines are both fine, which is why this
only affects videos — load_image(image, **kwargs) and
load_audio(audio, config=None, **kwargs) are happy with a single positional argument.

Reproduced against the current main sources by extracting the three real
signatures and replaying the calls, with the two working loaders as a control group:

load_video  signature: (video, use_audio_in_video, **kwargs)
load_image  signature: (image, **kwargs)
load_audio  signature: (audio, config=None, **kwargs)

  OK        load_image(p)   [control]
  OK        load_audio(p)   [control]
  TypeError load_video(p)   -> missing a required argument: 'use_audio_in_video'

  OK        load_video(p, use_audio_in_video=False)   [this PR]

How

relax/utils/multimodal/process.py:79 already contains the correct usage of the
same loader, and it is the spec this PR follows — unpack, keep the video tensor:

video_input, _, video_audio, _ = fetch_video(
    vision_info, image_patch_size=..., use_audio_in_video=use_audio_in_video, config=config,
)
video_inputs.append(video_input)

so both call sites become:

[load_video(p, use_audio_in_video=False)[0] for p in sample.videos]

use_audio_in_video=False is chosen deliberately rather than invented:

  • it is the default in every definition that declares it —
    agentic/pipeline/runtime.py:3448, models/qwen_omni/.../model.py:47,
    models/qwen_omni/.../utils.py:69, models/qwen_omni/qwen3_omni_provider.py:56,
    utils/data/data.py:44, utils/data/data_utils.py:328;
  • neither call site has args/config in scope to thread a real value from, and
  • both dicts already populate audios separately from sample.audios, so pulling
    the audio track out of the video here would double-count it.

Happy to thread args.use_audio_in_video down into the SFT dataset instead if you'd
prefer that shape — it is a larger change, so I kept this PR to the bug.

Testing

  • ruff check and ruff format --diff clean on both files with the pinned
    v0.15.9 from .pre-commit-config.yaml, run from the repo root so the
    pyproject.toml config applies (All checks passed! / 2 files already formatted).
  • Signature replay above (control group included). No GPU/video fixtures were
    available to run a full SFT step, so this verifies the arity and the unpack,
    not the downstream tensor semantics.
  • New tests added — there is no existing SFT multimodal test module to extend; happy to add one if you'd like.

Type of Change

  • Bug fix (non-breaking change that fixes an issue)

🤖 Generated with Claude Code

…loaders

`load_video(video, use_audio_in_video, **kwargs)` takes `use_audio_in_video`
as a required parameter and returns a 4-tuple
`(video, video_meta, audio, audio_meta)`.

Both SFT media loaders called it as `load_video(p)`, which raises
`TypeError: load_video() missing 1 required positional argument:
'use_audio_in_video'` and, once that is fixed, would still store the whole
4-tuple where downstream expects a `torch.Tensor`.

Follow the canonical usage in `relax/utils/multimodal/process.py`, which
unpacks the tuple and takes the video tensor, and use `use_audio_in_video=False`
to match the default every other definition in the repo already uses.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant