Learning to Group and Order: Cross-Instance Self-Supervised RL for Vision-Centric MLLMs

NeurIPS 2026

GO-SSL is a cross-instance self-supervised RL framework for vision-centric multimodal large language models. It trains models to group shuffled visual elements by their source and recover their spatial, temporal, or depth order, using automatically verifiable rewards.

1. Framework

2. Environment

conda create -n go-ssl python=3.12 -y
conda activate go-ssl
pip install torch==2.3.0 torchvision==0.18.0 --index-url https://download.pytorch.org/whl/cu121
pip install -r requirements.txt

See environment notes for the original dependency snapshot and compatibility details.

3. Dataset

We use the COCO 2014 training images to construct cross-image jigsaw puzzles. No category annotations are required.

Download the images from COCO and set the image path:

export GO_SSL_IMAGE_DIR=/path/to/coco/train2014
bash scripts/prepare_pairs.sh

The CLIP–DINO preprocessing pipeline builds the image pairs for the hard-pair curriculum. See preprocessing instructions.

4. Training

Two training recipes are provided:

Setting	Curriculum
GO-SSL-CB-600	400 steps Random 2×2 → 600 steps CLIP-Blind 2×2
GO-SSL-CB-2×3	400 steps Random 2×2 → 400 steps CLIP-Blind 2×2 → 200 steps CLIP-Blind 2×3

export GO_SSL_SEED=YOUR_SEED  # Replace YOUR_SEED with an integer

# Check the configuration and input data
python train.py --config configs/random_2x2.json --dry-run
python train.py --config configs/random_2x2.json --check-data

# GO-SSL-CB-600
bash scripts/train_cb600.sh

# GO-SSL-CB-2×3
bash scripts/train_cb2x3.sh

# Optional: distributed data parallel (e.g. 2 GPUs, global batch size unchanged)
GO_SSL_NPROC_PER_NODE=2 bash scripts/train_cb600_ddp.sh

The implementation uses Qwen2.5-VL-7B-Instruct with LoRA. For an example of the group-and-order task and adapter export, see demo.py and merge_lora.py, respectively.

5. Evaluation

Most benchmarks use VLMEvalKit. For VSR, DA-2K, OmniSpatial, SugarCrepe++, and related tasks, we refer to the evaluation procedures from Visual Jigsaw.

Details and references: Eval/README.md.

6. Citation

If you find GO-SSL useful in your research, please cite:

@inproceedings{yang2026gossl,
  title     = {Learning to Group and Order: Cross-Instance Self-Supervised RL for Vision-Centric MLLMs},
  author    = {Songming Yang and Hong-Tao Yu and Xiu-Shen Wei},
  booktitle = {Advances in Neural Information Processing Systems},
  year      = {2026},
  note      = {To appear}
}