ACA: Anatomy Contextualized Adaption of CT Foundation Models

ACA learns organ-level contrastive alignment on top of frozen CT foundation model encoders, enabling zero-shot findings classification from abdominal CT scans. Two training tracks are provided:

• Merlin backbone: Merlin I3ResNet image encoder with Clinical-Longformer text encoder
• CT-CLIP backbone: CT-CLIP CTViT image encoder with BiomedVLP-CXR-BERT text encoder

Supported model variants: mlp, aca_wo_scan, aca, aca_wo_anatomy, fvlm, spatial_transformer, visd_boost.

---

Installation

uv sync

PyTorch is indexed against CUDA 12.8. A GPU with at least 24 GB VRAM is recommended for training.

The CT-CLIP backbone additionally requires two subpackages; see CT-CLIP/README.md for details.

---

Pretrained Models and Datasets

CT-CLIP checkpoint

Download CT-CLIP_v2.pt from the CT-RATE HuggingFace repository and place it at CT-CLIP/models/CT-CLIP-Related/CT-CLIP_v2.pt.

Merlin checkpoint

The Merlin model weights are downloaded automatically on first use. If automatic download fails, download them from HuggingFace and place the checkpoint files under Merlin/merlin/models/checkpoints/.

Datasets

• CT-RATE: download from HuggingFace
• Merlin abdominal CT dataset: download from Stanford AIMI

---

Repository Structure

Public_ECCV2026W/
├── CT-CLIP/              # training + eval with CT-CLIP (CTViT) encoder
├── Merlin/               # training + eval with Merlin (I3ResNet) encoder
└── Preprocess/           # preprocessing pipelines (4 encoder and dataset combinations)

---

Data Preprocessing

Before training, run the preprocessing pipelines in Preprocess/ to extract organ embeddings from the raw CT volumes. See Preprocess/README.md for the full pipeline.

---

Path Configuration

Before training or evaluation, set the dataset paths in two places:

1. Training configs: edit packed_dir and report_text_dir in the YAML files under Merlin/train_merlin/configs/ or CT-CLIP/train_ct_clip/configs/.

2. Evaluation scripts: each script has a dataset root constant (DATA_ROOT, CT_RATE_ROOT, or MERLIN_DATA_ROOT) near the top of the file that must point to your local dataset.

---

Training

Both backbones use the same interface: --config selects the model variant and dataset, --gpu selects the device. For example:

# Merlin backbone
cd Merlin
python train_merlin/train.py --config train_merlin/configs/config_aca.yaml --gpu 0

# CT-CLIP backbone
cd CT-CLIP
python train_ct_clip/train.py --config train_ct_clip/configs/config_aca.yaml --gpu 0

For all model variants and the visd_boost multi-stage training procedure, see Merlin/README.md and CT-CLIP/README.md.

---

Evaluation

Baselines (pretrained models, no trained head)

# Merlin on the Merlin dataset
cd Merlin
python eval/eval_zero_shot_cls.py

# CT-CLIP on CT-RATE
cd CT-CLIP
python eval/run_zero_shot_ct_clip_ct_rate.py --gpu 0

ACA-trained models

# Merlin encoder on the Merlin dataset
cd Merlin
python eval/eval_trained_zero_shot.py \
    --checkpoint runs/aca/checkpoints/best.pt \
    --model_type aca \
    --gpu 0

# CT-CLIP encoder on CT-RATE
cd CT-CLIP
python eval/run_zero_shot_trained.py \
    --config train_ct_clip/configs/config_aca.yaml \
    --ckpt   runs/CT_RATE/aca/checkpoints/best.pt \
    --gpu 0

For all evaluation scripts and flags, see Merlin/README.md and CT-CLIP/README.md.