Ex-Omni-2D

Expressive Omni-Modal Dialogue Models with Native Visual Presence

Haoyu Zhang^*1, Zhipeng Li², Xiaoying Tang¹, Tianshu Yu^†1, Yiwen Guo^†3

¹The Chinese University of Hong Kong, Shenzhen    ²LIGHTSPEED    ³Independent Researcher

  

^*Work done during an internship at LIGHTSPEED.   ^†Corresponding authors.

Dialogue that can listen, speak, and appear.

News

• [2026-08] Released the paper, project page,code, and online demo.
• Future project updates will be posted here.

Overview

Given a multimodal query, reference image, and reference audio, Ex-Omni-2D produces a structured Visual Thought Plan (VTP), response text, personalized speech, and synchronized avatar video. Native multi-codebook speech units form a shared acoustic-temporal interface between speech and video generation.

The video pathway provides two complementary operating points:

• Full-sequence Teacher: bidirectional, quality-oriented video generation.
• Prefix-Streaming Student: few-step block-causal generation for efficient incremental deployment.

Highlights

• Dialogue-native video responses. Visual behavior is planned from the current multimodal conversation instead of being supplied as a separate manual prompt.
• Visual Thought Plan. A structured VTP explicitly represents scene, emotion, movement style, and motion details.
• Shared speech-unit interface. The same generated units drive personalized speech synthesis and frame-aligned avatar motion.
• Quality and efficiency modes. The Teacher targets visual quality, while the distilled Student supports 2-, 4-, and 8-step streaming inference.
• Portable deployment. The same code supports personal workstations, standard GPU servers, and Hugging Face Spaces.

Model Zoo

Model	Video backbone	Denoising steps	Availability
Ex-Omni-2D	Wan-2.1-T2V-1.3B	50	Hugging Face
Ex-Omni-2D Prefix-Streaming Student	Wan-2.1-T2V-1.3B	2 / 4 / 8	Coming soon

Installation

Requirements

• Linux and Python 3.10
• NVIDIA GPU with BF16 support
• PyTorch 2.7 and CUDA 12.6
• FFmpeg and SoX available on PATH

The reproducible Conda environment is recommended:

git clone https://github.com/LOGO-CUHKSZ/Ex-Omni-2D-Code.git
cd Ex-Omni-2D-Code
conda env create -f environment.yml
conda activate ex-omni-2d-release

Alternatively, install into an existing Python 3.10 environment:

sudo apt-get install -y ffmpeg libsndfile1 sox
pip install -r requirements.txt
pip install -e . --no-deps

FlashAttention is optional. The runtime falls back to PyTorch SDPA when the requested backend is unavailable. Compatible kernels can be installed after PyTorch:

pip install --no-build-isolation -r requirements-attention.txt

The optional vLLM-Omni backend uses a newer PyTorch/CUDA stack and should be installed in a separate Python 3.12 environment:

conda create -n ex-omni-2d-vllm python=3.12 -y
conda activate ex-omni-2d-vllm
pip install -r requirements-vllm-omni.txt
pip install -e . --no-deps

Checkpoints

The release YAML uses lazy hf:// references. On first use, checkpoints are downloaded from lemonade666/Ex-Omni-2D and the canonical Wan2.1, Qwen3-VL, Qwen3-TTS, and OmniAvatar repositories into the standard Hugging Face cache.

For a private or gated model repository, authenticate first:

huggingface-cli login
# Or on a non-interactive machine:
export HF_TOKEN=hf_your_read_token

The source repository may be public while the model repository remains private; in that case, only authorized Hugging Face accounts can run the released weights. Local checkpoints are also supported. Copy an example YAML to configs/*.local.yaml and replace its hf://... values with absolute paths.

To create an explicit offline mirror:

python scripts/download_weights.py \
  --output-dir weights \
  --ex-omni-repo lemonade666/Ex-Omni-2D

Quick Start

Validate the configuration

This command validates the release schema without loading model weights:

python -m ex_omni validate \
  --config configs/inference.teacher.example.yaml

Add --check-weights to verify that every local or remote checkpoint is accessible.

Web demo

Launch the single-GPU Teacher on GPU 0:

python scripts/demo/gradio_app.py \
  --mode teacher \
  --teacher-gpu 0 \
  --server-name 127.0.0.1 \
  --port 7860

Open http://127.0.0.1:7860. For access from another machine on a trusted network, bind to 0.0.0.0 and use a firewall rule or SSH tunnel. Do not expose an unauthenticated Gradio process directly to the public internet.

Command line

Run one complete text-to-text/speech/video request without Gradio:

python -m ex_omni chat-to-video \
  --config configs/inference.teacher.example.yaml \
  --text "Please introduce yourself briefly." \
  --ref-img asset/ref_img.jpg \
  --ref-audio asset/ref_audio.wav \
  --output outputs/example.mp4

Prefix-Streaming Student

For a local Student checkpoint, create configs/inference.student.local.yaml, set video.lora_checkpoint, and run:

python scripts/demo/gradio_app.py \
  --mode student \
  --config configs/inference.student.local.yaml \
  --student-deployment auto

The launcher prefers the configured four-GPU split and falls back to one GPU when those resources are unavailable. The release configuration also describes the supported eight-GPU replica topology.

Configuration

The Teacher release configuration enables the validated acceleration stack by default: resident models, merged LoRA, KV caches, speech compilation, TeaCache, and video compilation with CUDA Graphs. Unsupported kernels fall back gracefully.

• Set video.tea_cache_l1_thresh: 0 for strict full-compute quality.
• Set video.resident_models: false to reduce peak VRAM at the cost of speed.
• Disable individual compile sections when diagnosing platform-specific compiler issues.
• Set runtime.ffmpeg_path when FFmpeg is not available on PATH.

The default Teacher configuration targets a validated 80--96 GB GPU deployment class. Peak memory depends on resolution, response duration, compilation, and resident-model settings. Smaller GPUs require reduced resolution or duration and more aggressive model offloading.

Repository Structure

Ex-Omni-2D/
├── asset/                       # Default reference media and README assets
├── configs/                     # Portable Teacher and Student YAML files
├── ex_omni/                     # Pipeline, model, runtime, and distributed code
├── scripts/demo/                # Gradio application
├── scripts/download_weights.py  # Reproducible checkpoint downloader
├── environment.yml              # Tested Conda environment
├── requirements.txt             # Native runtime dependencies
└── README.md

Evaluation code, paper sources, generated media, and checkpoints are kept out of this release repository.

Acknowledgements

This project builds on Qwen3, Qwen3-VL, Qwen3-TTS, Wan2.1, and OmniAvatar. We thank the authors and open-source communities behind these projects.

Citation

If you find this project useful, please cite:

@article{zhang2026exomni2d,
  title  = {Ex-Omni-2D: Expressive Omni-Modal Dialogue
            Models with Native Visual Presence},
  author = {Zhang, Haoyu and Li, Zhipeng and Tang, Xiaoying
            and Yu, Tianshu and Guo, Yiwen},
  year   = {2026}
}

@article{zhang2026ex,
  title   = {Ex-Omni: Enabling 3D Facial Animation Generation
             for Omni-modal Large Language Models},
  author  = {Zhang, Haoyu and Li, Zhipeng and Guo, Yiwen
             and Yu, Tianshu},
  journal = {arXiv preprint arXiv:2602.07106},
  year    = {2026}
}

License

The code is released under the Apache License 2.0. Model weights and third-party dependencies are subject to their respective licenses and terms of use.