Predictive Visual Coding for Continuum Robots

Research code for predictive visual coding in continuum robots.

Environment

The reference environment uses Python 3.7.16, PyTorch 1.13.1, CUDA 11.7, and tiny-cuda-nn 1.7. An NVIDIA GPU with CUDA support is required.

conda env create -f environment.yml
conda activate predictive-visual-coding

Data

Set data_root in the configuration file to a directory with the following layout:

data_root/
  YYYY_MMDD/
    colmap/
      images/
      sparse/0/
    commands/
      <path>.json
      <path>/<world_time>/images/*.jpg

Each record associates a six-dimensional command with synchronized multi-view RGB images.

Usage

Train with a JSON configuration:

python main.py train --config configs/example.json

Render RGB and depth outputs from a checkpoint:

python main.py render \
  --config configs/example.json \
  --checkpoint outputs/example/checkpoints/stage3.pt \
  --split test --path 181 --output-dir outputs/rendered

License and acknowledgements

This project includes code derived from GraphDECO 3D Gaussian Splatting and uses a modified vendored copy of gsplat. It also uses tiny-cuda-nn and pytorch-ssim.

See LICENSE and docs/third_party.md for license and attribution details.