OpenSora Data Augmentation Lab

A reproducible study of whether OpenSora-generated videos can improve visual recognition under limited labeled data. The project generates class-conditioned flower videos, extracts frames, filters synthetic samples, and compares several ResNet-18 training strategies on a 10-class subset of Oxford Flowers-102.

Highlights

• Built an end-to-end synthetic-data pipeline from text prompts to evaluation.
• Compared real-only, traditional, raw synthetic, and confidence-filtered training.
• Evaluated both real-domain accuracy and held-out generated-domain robustness.
• Raw OpenSora augmentation raised held-out generated frame accuracy from 53.3% to 79.2% and video accuracy from 53.3% to 80.0%.
• The gain came with a small real-test tradeoff, from 94% to 92%.

Main Results

Method	Real test	Generated frame	Generated video
Real only	94%	53.3%	53.3%
Real + OpenSora raw	92%	79.2%	80.0%
Real + filtered-correct	92%	66.7%	66.7%
Real + confidence >= 0.5	92%	56.7%	56.7%
Real + confidence >= 0.7	94%	53.3%	53.3%

The experiment suggests that synthetic video frames are useful here as robustness-oriented augmentation, while aggressive filtering removes diversity and weakens the benefit.

Pipeline

1. Build balanced train, validation, and test splits from 10 Flowers-102 classes.
2. Generate class-conditioned videos with OpenSora v1.3.
3. Extract four frames per generated video with FFmpeg.
4. Train an ImageNet-pretrained ResNet-18 linear probe.
5. Filter generated frames using baseline predictions and confidence thresholds.
6. Evaluate real images plus a separate set of generated prompts at frame and video level.

Repository Layout

data/prompts/       Prompt metadata
data/splits/        Reproducible split and generated-frame metadata
paper/              CVPR-style project report
results/tables/     Final metrics and per-class results
results/figures/    Selected qualitative comparisons
scripts/            Generation, training, filtering, and evaluation pipeline

Large datasets, generated videos, checkpoints, and raw run logs are intentionally excluded from Git.

Setup

git clone https://github.com/chaohanzzz/opensora-data-augmentation-lab.git
cd opensora-data-augmentation-lab

python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

Install FFmpeg separately for frame extraction. Clone and install Open-Sora, then expose its location:

export OPENSORA_ROOT=/path/to/Open-Sora

OpenSora has its own CUDA and package requirements. This repository does not vendor its source code or model weights.

Reproducing the Study

Prepare the Flowers-102 subset:

python scripts/prepare_flowers102_subset.py

Create prompts and run OpenSora generation:

python scripts/make_prompts.py
python scripts/run_opensora_t2v_prompts.py --limit 20

Extract frames and run classifier experiments:

python scripts/extract_generated_frames.py
bash scripts/run_baselines.sh
bash scripts/run_opensora_aug_experiments.sh

Every script supports command-line options for paths and experiment settings. The checked-in result tables document the completed server-side run.

Scope

This is a controlled pilot rather than a claim of broad state-of-the-art performance. It uses 10 flower classes, 49-frame 360p generation, four sampled frames per video, and a ResNet-18 linear probe. See paper/ for methodology, failure analysis, and limitations.

Acknowledgements

This project builds on Open-Sora and the Oxford 102 Flowers Dataset. OpenSora source code and dataset images retain their respective licenses.