S³O: Selective Spatial-Spectral Operator for Cross-Scale Fusion

Official PyTorch implementation of S³O, a selective spatial-spectral operator for cross-scale hyperspectral image (HSI) and multispectral image (MSI) fusion, accepted to Findings of CVPR 2026.

Authors. Jieyuan Pei¹*, Wei Li¹*, Zhuoxuan Li², Junwei Zhu¹, Meiyi Lu¹, Jiawei Jiang¹, Chenyu Wang¹, Jianwei Zheng¹†

¹ Zhejiang University of Technology    ² Tongji University    * Equal contribution    † Corresponding author (zjw@zjut.edu.cn)

Figure 1. Overview of the proposed S³O Backbone and the S³O Block with the hard-location selection mechanism.

Overview

S³O fuses a low-resolution hyperspectral image (LR-HSI) with a high-resolution multispectral / RGB image (HR-MSI) to recover a high-resolution hyperspectral image (HR-HSI). It is built around two ideas:

• A factorized spatial-spectral state-space backbone that achieves continuous-discrete equivalence and therefore generalizes across spatial scales unseen during training.
• A selective hard-location enhancement mechanism that uses an intrinsic spatial-spectral non-separability score to route only the difficult tokens through a heavier refinement branch, keeping the average compute close to a separable operator.

S³O is evaluated on the CAVE, Harvard, and Chikusei benchmarks at multiple scaling factors, with a strong emphasis on cross-scale generalization (training at one scale, evaluating at others).

Figure 2. (Left) PSNR versus Params / GFLOPs trade-off on CAVE x4. (Right) Relative log-amplitudes of spectral feature maps between the reconstructed image and the ground truth.

Pull requests, bug reports, and reproduction notes are welcome.

---

Project Layout

S3O-release/
├── configs/
│   └── datasets.yaml          # Dataset path configuration
├── data/
│   ├── CAVE_Dataset.py        # CAVE dataset loader
│   ├── Harvard_Dataset.py     # Harvard dataset loader
│   ├── Chikusei_Dataset.py    # Chikusei dataset loader (HDF5)
│   └── Pypher.py              # Embedded PyPHER (BSD), used for psf2otf
├── models/
│   ├── s3o_backbone.py        # Spatial-Spectral SSM backbone (Block, S3OBackbone)
│   └── s3o_models.py          # Dataset-specific wrappers (S3OCaveModel, ...)
├── scripts/
│   ├── setup_datasets.py      # Write/update configs/datasets.yaml from CLI args
│   └── run_all_s3o.sh         # Convenience launcher for sequential training
├── train/
│   ├── train_cave.py          # Training entry point for CAVE
│   ├── train_harvard.py       # Training entry point for Harvard
│   └── train_chikusei.py      # Training entry point for Chikusei
├── test/
│   ├── test_cave.py           # Evaluation entry point for CAVE
│   ├── test_harvard.py        # Evaluation entry point for Harvard
│   └── test_chikusei.py       # Evaluation entry point for Chikusei
├── utils/
│   ├── config.py              # YAML / path helpers
│   ├── data_utils.py          # Dataset loading helpers
│   ├── metrics.py             # PSNR / SSIM / SAM / ERGAS and FFT helpers
│   └── SSIM.py                # Differentiable SSIM module
├── demo.py                    # Single-image inference demo
├── requirements.txt           # Minimal pip requirements
├── requirements_conda.txt     # Frozen conda environment for full reproducibility
├── LICENSE                    # MIT
└── CITATION.cff               # Machine-readable citation metadata

---

Environment Setup

S³O depends on PyTorch, Mamba SSM (mamba-ssm), causal-conv1d, timm, and a few standard image / data libraries. The minimal install is:

conda create -n s3o python=3.10 -y
conda activate s3o
pip install -r requirements.txt

requirements_conda.txt is a frozen export of the exact conda environment used by the authors. Use it only if you want byte-for-byte reproducibility:

conda create -n s3o --file requirements_conda.txt

Note. mamba-ssm and causal-conv1d require a CUDA toolchain at install time. If your machine cannot build them locally, install the matching prebuilt wheels from the upstream releases.

---

Datasets

S³O is evaluated on three standard HSI super-resolution benchmarks. Download each dataset from its official source and arrange it as below (paths can be customized via configs/datasets.yaml).

Dataset	Bands	Image size	Layout
CAVE	31	512 x 512	<root>/HSI/{name}.mat, <root>/RGB/{name}.mat, Train.txt / Test.txt
Harvard	31	1040 x 1392	<root>/{i}.mat with keys HS (HSI) and HRMS (RGB)
Chikusei	128	512 x 512	train_Chikusei.h5 / test_Chikusei.h5 with GT and RGB datasets

Configure dataset paths

You can either edit configs/datasets.yaml directly or run the helper:

python scripts/setup_datasets.py \
  --cave-train     /path/to/CAVE/Train \
  --cave-test      /path/to/CAVE/Test \
  --harvard-train  /path/to/Harvard/Train \
  --harvard-test   /path/to/Harvard/Test \
  --chikusei-train /path/to/train_Chikusei.h5 \
  --chikusei-test  /path/to/test_Chikusei.h5

Every train/test entry point accepts --dataset_config <path> to override the default YAML location.

---

Training

CAVE (x4)

python train/train_cave.py \
  --dataset_config configs/datasets.yaml \
  --log_root ./experiments \
  --sf 4 \
  --epochs 1000

Harvard (x4)

python train/train_harvard.py \
  --dataset_config configs/datasets.yaml \
  --log_root ./experiments \
  --sf 4 \
  --epochs 1000

Chikusei (x8)

python train/train_chikusei.py \
  --dataset_config configs/datasets.yaml \
  --log_root ./experiments \
  --sf 8 \
  --epochs 200

You can run all three sequentially with:

bash scripts/run_all_s3o.sh configs/datasets.yaml ./experiments

Each training run writes:

• experiments/S3O_<DATASET>/<sf>/checkpoint/ - per-epoch and best checkpoint
• experiments/S3O_<DATASET>/<sf>/tensorboard/ - TensorBoard scalars
• experiments/S3O_<DATASET>/<sf>/logs/ - plain-text training log

The CAVE and Harvard pipelines automatically run a cross-scale evaluation after training (set --no_cross_scale to disable).

---

Evaluation

Each test script loads a checkpoint, runs inference at the requested scale, and prints PSNR / SSIM / SAM / ERGAS averages. Pass --save_results to also save per-image .mat files plus a summary.txt under the chosen output directory.

# CAVE x4
python test/test_cave.py \
  --checkpoint experiments/S3O_CAVE/4/checkpoint/BestModel.pth \
  --dataset_config configs/datasets.yaml \
  --sf 4

# Harvard x4
python test/test_harvard.py \
  --checkpoint experiments/S3O_Harvard/4/checkpoint/BestModel.pth \
  --dataset_config configs/datasets.yaml \
  --sf 4

# Chikusei x8
python test/test_chikusei.py \
  --checkpoint experiments/S3O_Chikusei/8/checkpoint/BestModel.pth \
  --dataset_config configs/datasets.yaml \
  --sf 8

---

Single-Image Demo

demo.py loads a trained CAVE checkpoint and runs S³O on a single HSI / RGB pair. The script applies the same Gaussian blur + downsampling degradation used during training to produce the LR-HSI input.

python demo.py \
  --checkpoint /path/to/BestModel.pth \
  --hsi_path   /path/to/sample_hsi.mat \
  --rgb_path   /path/to/sample_rgb.mat \
  --sf 4 \
  --compute_metrics \
  --output sr_result.mat

---

Repository Notes

• data/Pypher.py is an embedded copy of PyPHER by Alexandre Boucaud (BSD license). The original copyright header is preserved in the file. Only psf2otf and a few helpers are used.
• utils/SSIM.py is a standard differentiable SSIM implementation.
• utils/metrics.py keeps several legacy commented-out variants of PSNR / SSIM helpers from earlier development; they are not used by the released pipeline but are kept for reference.

License

Released under the MIT License. See LICENSE for the full text.

The embedded PyPHER source in data/Pypher.py retains its original BSD license. The S³O code does not incorporate any other third-party source beyond the dependencies listed in requirements.txt.

Citation

If you find this work useful, please cite our paper. Machine-readable citation metadata is also provided in CITATION.cff.

@inproceedings{pei2026s3o,
  title     = {{S$^3$O}: Selective Spatial-Spectral Operator for Cross-Scale Fusion},
  author    = {Pei, Jieyuan and Li, Wei and Li, Zhuoxuan and Zhu, Junwei and
               Lu, Meiyi and Jiang, Jiawei and Wang, Chenyu and Zheng, Jianwei},
  booktitle = {Findings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
  year      = {2026}
}

Contact

For questions about the code or paper, please open a GitHub issue or contact the corresponding author Jianwei Zheng (zjw@zjut.edu.cn).