Skip to content

[RL] Forward multimodal inputs through Omni rollouts - #3

Closed
JingwenGu0829 wants to merge 1 commit into
hayden/omni-rl-clean-20260708from
agent/multimodal-generate-contract
Closed

[RL] Forward multimodal inputs through Omni rollouts#3
JingwenGu0829 wants to merge 1 commit into
hayden/omni-rl-clean-20260708from
agent/multimodal-generate-contract

Conversation

@JingwenGu0829

@JingwenGu0829 JingwenGu0829 commented Jul 11, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • make OmniGenerateFn send canonical images, audios, and videos fields
  • reuse Miles' established VLM image serializer instead of introducing another image path
  • keep the existing WAV input encoder for audio
  • forward transport-safe video URLs/data URIs and reject unsupported in-memory video shapes instead of silently dropping them
  • add payload and hook-level regression coverage

Why

The standard Miles VLM path already serializes images, but the custom Omni rollout hook bypasses that request builder and only sends audio. That prevents the stacked sglang-omni /generate contract from receiving the full multimodal input.

This draft is stacked on hayden/omni-rl-clean-20260708 from PR #2. Its companion change is sgl-project/sglang-omni#1032, based on hayden/rl-higgs-qwen3-clean-20260708.

Validation

python -m pytest \
  tests/fast/test_omni_generate_fn.py \
  tests/fast/test_omni_rollout_contract.py -q

33 passed, 1 xfailed

Draft follow-up

Run image/audio rollout-to-training parity against the companion sglang-omni branch. A general serializer for in-memory video tensors remains separate work; this draft fails those inputs explicitly.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant