Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 30 additions & 0 deletions job_bundles/gsplat_pipeline/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,36 @@ on your Deadline Cloud CUDA farm.
Now you just need a video that captures many viewpoints of a subject to reconstruct in 3D,
and then you can run the Gaussian Splatting pipeline on your farm.

### Known issues

#### PyTorch 2.6 compatibility with NeRF Studio export (workaround applied)

The `nerfstudio` package on conda-forge (v1.1.5) requires PyTorch 2.6+, which changed `torch.load` to default
to `weights_only=True`. This breaks NeRF Studio's `ns-export` command when loading checkpoints. A workaround
is applied in `train_nerfstudio.sh` that patches `torch.load` to restore the previous behavior. This workaround
can be removed once the upstream fix is merged:
https://github.com/nerfstudio-project/nerfstudio/pull/3711

#### GSPLAT_SIMPLE_TRAINER and NERFSTUDIO_SPLATFACTOW require the custom nerfstudio package

The `GSPLAT_SIMPLE_TRAINER` and `NERFSTUDIO_SPLATFACTOW` trainer options depend on commands
(`gsplat_simple_trainer`, `splatfactow_export`) that are only available in the custom-built nerfstudio
conda package from the [NeRF Studio conda recipe](../../conda_recipes/nerfstudio/). The conda-forge version
of nerfstudio does not include these. If you want to use these trainers, follow the
[NeRF Studio sample conda package recipe README](../../conda_recipes/nerfstudio/README.md) to build the
package into your S3 conda channel.

#### Simplified prerequisites for the default NERFSTUDIO trainer

If you only need the default `NERFSTUDIO` (splatfacto) trainer, you do not need to deploy the CUDA farm
CloudFormation template or build the custom nerfstudio conda package. The minimum requirements are:

- A Deadline Cloud farm with a GPU fleet
- A queue environment with `conda-forge` included in the Conda Channels (e.g. `deadline-cloud conda-forge`)
- Conda packages: `ffmpeg colmap glomap nerfstudio cuda` (edit in GUI submitter)

The full CUDA farm setup and custom package build are only required for the `GSPLAT_SIMPLE_TRAINER` and `NERFSTUDIO_SPLATFACTOW` options.

## Capture a video of a subject

You can use a video-capable camera like your smartphone to capture a video of a subject for your Gaussian Splatting.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -42,9 +42,30 @@ if [[ "$1" == splatfacto-w* ]]; then
--output_dir ./nerfstudio_workspace \
--camera_idx 0
mv ./nerfstudio_workspace/splat.ply "$OUTPUT_PLY_FILE"

# WORKAROUND: nerfstudio's ns-export is incompatible with PyTorch 2.6+
# which defaults torch.load to weights_only=True. The checkpoint contains
# numpy globals that are safe but not allowlisted. We patch torch.load
# to restore the pre-2.6 behavior. Remove this when nerfstudio is fixed.
# See: https://github.com/nerfstudio-project/nerfstudio/pull/3711

# TODO: Once the upstream fix is merged, replace the code below with:
# else
# ns-export gaussian-splat \
# --load-config ./nerfstudio_workspace/splatfacto/*/config.yml \
# --output-dir "$(dirname "$OUTPUT_PLY_FILE")" \
# --output-filename "$(basename "$OUTPUT_PLY_FILE")"
# fi

else
ns-export gaussian-splat \
--load-config ./nerfstudio_workspace/splatfacto/*/config.yml \
--output-dir "$(dirname "$OUTPUT_PLY_FILE")" \
--output-filename "$(basename "$OUTPUT_PLY_FILE")"
fi
python -c "
import torch, glob
_orig = torch.load
torch.load = lambda *a, **kw: _orig(*a, **{**kw, 'weights_only': False})
import sys
cfg = glob.glob('./nerfstudio_workspace/splatfacto/*/config.yml')[0]
sys.argv = ['ns-export', 'gaussian-splat', '--load-config', cfg, '--output-dir', '$(dirname "$OUTPUT_PLY_FILE")', '--output-filename', '$(basename "$OUTPUT_PLY_FILE")']
from nerfstudio.scripts.exporter import entrypoint
entrypoint()
"
fi
Loading