🎞️ xfactor-torch

Official PyTorch reimplementation of

XFactor: True Self-Supervised Novel View Synthesis is Transferable

Thomas W. Mitchel* · Hyunwoo Ryu* · Vincent Sitzmann

---

🛠️ Installation

conda create -n xfactor python=3.12
conda activate xfactor
pip install -r requirements.txt

🎬 Test: Control-by-Transfer NVS

Render the cornell-box sim-to-real visualization with a trained checkpoint. The output video is saved to ./_output/eval/sim_to_real.mp4 by default.

python eval.py --weights ./weights/5v_full.pt

This should reproduce a sim-to-real result close to the one shown on our project page:

sim_to_real.mp4

💾 Pretrained Weights

Download the checkpoints below and place them under ./weights/:

Checkpoint	Ctx Views	Iterations	Status
1v_pretrain.pt	1	100k	✅ Available
3v_light.pt	3	100k	✅ Available
5v_full.pt	5	200k	✅ Available

🏋️ Train

📂 Datasets

Set --re10k_path, --co3dv2_path, --dl3dv_path, and --mvimgnet_path to your local dataset directories.

Note

Each dataset is described by a BaseSceneIndex subclass under src/data/scene_index/ that abstracts away its directory layout. To train on a new dataset, implement your own BaseSceneIndex and wire it into train.py.

🚀 Train Script

All training outputs are saved under ./_output/$JOB_ID.

Monocular-Stereo Pretraining

⏱️ ~26 hours (0.95 iters/sec) on 8× L40S, which should roughly translate to ~8.5 hours on 8× H200.

# 2-view pretraining
bash ./train.sh \
    --use_wandb --compile \
    --base_seed 0 \
    --re10k_path <...> \
    --co3dv2_path <...> \
    --dl3dv_path <...> \
    --mvimgnet_path <...>

Multiview Finetuning (Light Setting)

⏱️ ~52 hours (1.85 iters/sec) on 8× L40S, which should roughly translate to ~16.5 hours on 8× H200.

bash ./train.sh \
    --pretrained_weights ./weights/1v_pretrain.pt \
    --use_wandb --compile \
    --base_seed 1 \
    --n_ctxt_views 3 \
    --self_mask_prob 0.02 \
    --upweight \
    --gap_sigma 0.25 \
    --re10k_weight 3 \
    --co3dv2_weight 2 \
    --dl3dv_weight 5 \
    --mvimgnet_weight 2 \
    --re10k_window 40 200 \
    --co3dv2_window 9 40 \
    --dl3dv_window 6 20 \
    --mvimgnet_window 6 20 \
    --re10k_path <...> \
    --co3dv2_path <...> \
    --dl3dv_path <...> \
    --mvimgnet_path <...>

Multiview Finetuning (Original Paper Setting)

⏱️ ~56 hours (1 iters/sec) on 8× H200.

bash ./train.sh \
    --pretrained_weights ./weights/1v_pretrain.pt \
    --use_wandb --compile \
    --base_seed 1 \
    --total_steps 200000 \
    --n_ctxt_views 5 \
    --self_mask_prob 0.02 \
    --upweight \
    --gap_sigma 0.25 \
    --re10k_weight 3 \
    --co3dv2_weight 2 \
    --dl3dv_weight 5 \
    --mvimgnet_weight 2 \
    --re10k_window 40 200 \
    --co3dv2_window 9 40 \
    --dl3dv_window 6 20 \
    --mvimgnet_window 6 20 \
    --re10k_path <...> \
    --co3dv2_path <...> \
    --dl3dv_path <...> \
    --mvimgnet_path <...>

📈 Training Larger Models

If you have the compute, scale the model up by appending:

bash ./train.sh \
    <...>
    --n_enc_layers 12 \
    --n_dec_layers 24

Scaling up the model improves geometric reasoning and greatly mitigates the warping and discontinuity artifacts that arise when XFactor cannot reliably estimate depth and falls back on shortcuts in its latent representation.

---

🔬 Minor Differences from the Original JAX Implementation

🎯 Attention masking

The JAX version uses id-based masking (a.k.a. document masking) to run two disjoint patch groups in a single forward pass, exploiting the block-sparsity of the mask with BLOCK_SIZE=64 (the number of tokens per quadrant of a 256×256 image). Unfortunately, PyTorch lacks an equivalent: FlexAttention only supports BLOCK_SIZE=128 in compile mode, and BLOCK_SIZE=64 with max-autotune is unstable and offers no speedup over eager mode due to frequent graph cuts. Since compile is up to ~2.5× faster than eager, compiling is essential. We instead run the two patch groups as two separate forward passes. This is empirically faster than dense attention masking or more complicated sparse/block-diagonal masking schemes.

🎲 Self-mask ratio

The original JAX implementation unmasks 5% of samples; we unmask 1 sample per every 16 (6.25%) to keep input shapes static and avoid recompilation.

🧱 Minor Architectural Differences

A handful of small details differ — for instance, global and frame attention run sequentially rather than in parallel, and the encoder and decoder use separate input projections instead of a shared one — but these are negligible.