EndoCLIP

EndoCLIP is a contrastive vision–language framework for gastrointestinal endoscopy. It adapts open-vocabulary image–text pretraining to endoscopic video frames and their paired clinical reports, so that a single model can be used for zero-shot lesion recognition, frame–report retrieval, and transferable endoscopic representation learning.

This repository contains the full training and evaluation codebase (vendored from OpenCLIP v3.3.0, see Acknowledgements) together with the release information for the EndoReport100 dataset.

---

Table of Contents

• Features
• Repository Layout
• Installation
• Data: EndoReport100
• Usage
  • Loading a model
  • Training
  • Zero-shot evaluation
• Upstream OpenCLIP Documentation
• Citation
• License
• Acknowledgements

---

Features

• Contrastive image–text pretraining over endoscopic frames and de-identified report text.
• Wide model coverage — ResNet, ViT, ConvNeXt, CoCa, SigLIP and timm-backed towers, all configurable through JSON model configs in src/open_clip/model_configs/.
• Distributed training with PyTorch DDP / FSDP, mixed precision (amp, amp_bf16), gradient checkpointing, and gradient accumulation.
• Webdataset and CSV pipelines, so both sharded video-frame archives and simple image_path,caption tables can be used without code changes.
• Zero-shot classification and retrieval evaluation built into the training loop.

---

Repository Layout

EndoCLIP/
├── src/
│   ├── open_clip/            # model definitions, tokenizer, losses, pretrained registry
│   │   └── model_configs/    # JSON architecture configs
│   └── open_clip_train/      # training entry point, data pipelines, schedulers, zero-shot eval
├── scripts/                  # helper launch scripts
├── docs/                     # upstream documentation, pretrained model tables, notebooks
├── tests/                    # unit / regression tests
├── tutorials/                # notebook tutorials
├── requirements.txt          # inference dependencies
├── requirements-training.txt # additional training dependencies
├── requirements-test.txt     # test dependencies
├── OPENCLIP_README.md        # unmodified upstream OpenCLIP README
├── CITATION_openclip.cff     # unmodified upstream OpenCLIP citation metadata
└── LICENSE                   # MIT license inherited from OpenCLIP

---

Installation

# 1. Clone
git clone https://github.com/Jia7878/EndoCLIP.git
cd EndoCLIP

# 2. Create an environment
conda create -n endoclip python=3.10 -y
conda activate endoclip

# 3. Install PyTorch matching your CUDA version (see https://pytorch.org)
pip install torch torchvision

# 4. Install EndoCLIP in editable mode
pip install -e .

# 5. Extra dependencies for training
pip install -r requirements-training.txt

Verify the installation:

python -c "import open_clip; print(open_clip.__version__)"

---

Data: EndoReport100

If EndoCLIP or EndoReport100 is useful for your research, please cite our paper: A report-grounded vision-language foundation model for colonoscopy from 280,000 routine reports.

The de-identified EndoReport100 release contains 100 cases and 7,003 frames, together with frame-level annotations and instance metadata.

Download: https://doi.org/10.6084/m9.figshare.33085637

Access is restricted and subject to approval by the dataset owners. All frames and reports are de-identified; any attempt at re-identification is prohibited.

Expected format

The training scripts consume either a CSV table or webdataset shards.

CSV — one row per frame:

filepath,caption
frames/case001/000123.jpg,"Sessile polyp in the sigmoid colon, approximately 8 mm."
frames/case001/000124.jpg,"Normal mucosa with clear vascular pattern."

Webdataset — .tar shards where each sample pairs an image (.jpg) with its text (.txt) under a shared key.

---

Usage

Loading a model

import torch
from PIL import Image
import open_clip

model, _, preprocess = open_clip.create_model_and_transforms(
    "ViT-B-16", pretrained="/path/to/endoclip_checkpoint.pt"
)
model.eval()
tokenizer = open_clip.get_tokenizer("ViT-B-16")

image = preprocess(Image.open("frames/case001/000123.jpg")).unsqueeze(0)
text = tokenizer(["a colonoscopy frame of a polyp",
                  "a colonoscopy frame of normal mucosa"])

with torch.no_grad(), torch.autocast("cuda"):
    image_features = model.encode_image(image)
    text_features = model.encode_text(text)
    image_features /= image_features.norm(dim=-1, keepdim=True)
    text_features /= text_features.norm(dim=-1, keepdim=True)
    probs = (100.0 * image_features @ text_features.T).softmax(dim=-1)

print("label probs:", probs)

Training

Single node, 4 GPUs, CSV input:

torchrun --nproc_per_node 4 -m open_clip_train.main \
    --model ViT-B-16 \
    --pretrained openai \
    --train-data /path/to/endoreport100/train.csv \
    --val-data /path/to/endoreport100/val.csv \
    --csv-img-key filepath \
    --csv-caption-key caption \
    --dataset-type csv \
    --batch-size 128 \
    --lr 1e-5 \
    --wd 0.1 \
    --epochs 30 \
    --warmup 500 \
    --workers 8 \
    --precision amp \
    --logs ./logs \
    --name endoclip-vitb16 \
    --report-to tensorboard

Webdataset shards:

torchrun --nproc_per_node 4 -m open_clip_train.main \
    --model ViT-B-16 \
    --dataset-type webdataset \
    --train-data "/path/to/endoreport100/shards/train-{000000..000031}.tar" \
    --train-num-samples 7003 \
    --batch-size 128 --epochs 30 --precision amp \
    --logs ./logs --name endoclip-vitb16-wds

All available flags are documented in src/open_clip_train/params.py.

Zero-shot evaluation

python -m open_clip_train.main \
    --model ViT-B-16 \
    --resume ./logs/endoclip-vitb16/checkpoints/epoch_30.pt \
    --imagenet-val /path/to/endoreport100/zeroshot_val \
    --batch-size 128 \
    --precision amp

For broader benchmarking we recommend the CLIP Benchmark suite, which consumes OpenCLIP-format checkpoints directly.

---

Upstream OpenCLIP Documentation

Because this repository vendors the OpenCLIP source tree, the upstream documentation applies verbatim:

Topic	Document
Full upstream README	OPENCLIP_README.md
Pretrained model zoo	docs/PRETRAINED.md
DataComp models	docs/datacomp_models.md
CLIPA training	docs/clipa.md
Conceptual Captions walkthrough	docs/clip_conceptual_captions.md
Interactive notebook	docs/Interacting_with_open_clip.ipynb
Release history	HISTORY.md

---

Citation

If EndoCLIP or EndoReport100 is useful for your research, please cite our paper: A report-grounded vision-language foundation model for colonoscopy from 280,000 routine reports.

Please also cite OpenCLIP, whose implementation this work is built on. The upstream citation metadata is preserved in CITATION_openclip.cff:

@software{ilharco_gabriel_2021_5143773,
  author    = {Ilharco, Gabriel and Wortsman, Mitchell and Wightman, Ross and
               Gordon, Cade and Carlini, Nicholas and Taori, Rohan and
               Dave, Achal and Shankar, Vaishaal and Namkoong, Hongseok and
               Miller, John and Hajishirzi, Hannaneh and Farhadi, Ali and
               Schmidt, Ludwig},
  title     = {OpenCLIP},
  month     = jul,
  year      = 2021,
  publisher = {Zenodo},
  doi       = {10.5281/zenodo.5143773},
  url       = {https://doi.org/10.5281/zenodo.5143773}
}

---

License

The code in this repository is distributed under the MIT license inherited from OpenCLIP — see LICENSE. The original copyright notice of the OpenCLIP authors is retained unmodified.

---

Acknowledgements

This project is built on OpenCLIP.

The entire src/, docs/, scripts/, tests/ and tutorials/ trees in this repository are copied from OpenCLIP release v3.3.0 (mlfoundations/open_clip@v3.3.0), with EndoCLIP-specific configurations and endoscopy adaptations layered on top. The upstream README and citation file are preserved unmodified as OPENCLIP_README.md and CITATION_openclip.cff, and the upstream MIT LICENSE is retained in full.

We thank the OpenCLIP authors and the wider LAION / mlfoundations community for releasing their work openly. Please follow the upstream OpenCLIP citation guidance when using this code.