Stable Derolling: First Stage Derolling Pipeline using Marigold Framework

🛠️ Setup

The inference code was tested on:

• Ubuntu 22.04 LTS, Python 3.10.12, CUDA 11.7, GeForce RTX 3090 (pip, Mamba)
• CentOS Linux 7, Python 3.10.4, CUDA 11.7, GeForce RTX 4090 (pip)
• Windows 11 22H2, Python 3.10.12, CUDA 12.3, GeForce RTX 3080 (Mamba)
• MacOS 14.2, Python 3.10.12, M1 16G (pip)

🪧 A Note for Windows users

We recommend running the code in WSL2:

1. Install WSL following installation guide.
2. Install CUDA support for WSL following installation guide.
3. Find your drives in /mnt/<drive letter>/; check WSL FAQ for more details. Navigate to the working directory of choice.

📦 Repository

Clone the repository (requires git):

git clone https://github.com/JacobACarter/Stable-Derolling.git
cd Marigold

💻 Dependencies

We provide several ways to install the dependencies.

1. Using Mamba, which can installed together with Miniforge3.

   Windows users: Install the Linux version into the WSL.

   After the installation, Miniforge needs to be activated first: source /home/$USER/miniforge3/bin/activate.

   Create the environment and install dependencies into it:

   mamba env create -n marigold --file environment.yaml
   conda activate marigold

2. Using pip: Alternatively, create a Python native virtual environment and install dependencies into it:

   python -m venv venv/marigold
   source venv/marigold/bin/activate
   pip install -r requirements.txt

Keep the environment activated before running the inference script. Activate the environment again after restarting the terminal session.

At inference, specify the checkpoint path: Use the test_run.py file!

python test_run.py \
    --checkpoint checkpoint/marigold-v1-0 \
    --denoise_steps 50 \
    --ensemble_size 1 \
    --input_rgb_dir input/in-the-wild_example\
    --output_dir output/in-the-wild_example

🏋️ Training

Based on the previously created environment, install extended requirements:

pip install -r requirements++.txt -r requirements+.txt -r requirements.txt

Set environment parameters for the data directory:

export BASE_DATA_DIR=YOUR_DATA_DIR  # directory of training data
export BASE_CKPT_DIR=YOUR_CHECKPOINT_DIR  # directory of pretrained checkpoint

Download Stable Diffusion v2 checkpoint into ${BASE_CKPT_DIR} Download the datasets into '${BASE_DATA_DIR}': 10-12 10-20

OUR PRETRAINED CHECKPOINT Run training script

python test_train.py --config config/train_collocated_10_20.yaml

Resume from a checkpoint, e.g.

python test_train.py --resume_run output/train_collocated_10_20/checkpoint/latest