【NeurCom'2026 🔥】PHA-Net: Prototype-based Hierarchical Alignment Network for Text-Video Retrieval

The implementation of NeurCom 2026 paper PHA-Net: Prototype-based Hierarchical Alignment Network for Text-Video Retrieval

📌 Citation

If you find our method useful in your work, please cite:

@article{JING2026134693,,
  title={PHA-Net: Prototype-based Hierarchical Alignment  Network for Text-Video Retrieva},
  author={Jing, Xiaolun and Yin, Kezhao and Yang Xinxing and Yang, Genke and Chu, Jian},
  journal={Neurocomputing},
  pages={134693},
  year={2026},
  publisher={Elsevier}
  }

💡 I also have other text-video retrieval projects that may interest you ✨.

An Empirical Study of Excitation and Aggregation Design Adaptions in CLIP4Clip for Video–Text Retrieval
Accepted by NeurCom 2024 | EA-CLIP4Clip Code
Xiaolun Jing, Genke Yang, Jian Chu

TC-MGC: Text-Conditioned Multi-Grained Contrastive Learning for Text-Video Retrieval
Accepted by INFFUS 2025 | TC-MGC Code
Xiaolun Jing, Genke Yang, Jian Chu

Text-Video Retrieval With Global-Local Contrastive Consistency Learning
Accepted by CAC 2025 | GLCCL Code
Xiaolun Jing, Genke Yang, Jian Chu

📕 Overview

With the emergence of large-scale image-text pre-training models, e.g., CLIP, text-video retrieval has experienced substantial advances in recent years. Existing best-performing methods involve aligning cross-modal semantics at individual, local, and global levels simultaneously, raising concerns about the intrinsic semantic mismatch between concise texts and rich videos. A canonical approach is to integrate language-video attention mechanism into the hierarchical framework while this paradigm only optimizes visual representations with prohibitive computational costs. In this paper, we propose a new prototype-based hierarchical alignment network (PHA-Net) to align individual/local/global level representations across modalities. Concretely, we introduce multiple modality-shared prototypes as the bridge to efficiently optimize text and video representations for cross-modal alignment. Then, we argue that the imbalanced semantic distribution in clustered tokens may undermine retrieval performance, as tokens with weak semantics are of little interest. To reduce the impact of these tokens, a proposed prototype-supported token merge module is responsible for enhancing tokens with strong semantics and suppressing others with weak semantics via the prototype semantics guidance. Moreover, we devise a prototype contrastive loss to encourage textual and visual prototypes to focus on different semantic information. The idea of this auxiliary loss is to ensure higher similarity between textual and visual prototypes from the same prototype than those from different prototypes. Extensive experiments on four benchmarks confirm the effectiveness of our PHA-Net, which achieves significant improvements in the sum of all recalls on MSR-VTT (8.8%), ActivityNet (19.2%), VATEX (0.4%), and Charades (4.9%).

📚 Method

🚀 Quick Start

Setup code environment

pip install -r requirements.txt

Datasets

We train our model on MSR-VTT, ActivityNet, VATEX and Charades datasets respectively. Please refer to this repo for data preparation.

Datasets	Google Cloud	Baidu Yun	Peking University Yun
MSR-VTT	Download	Download	Download
ActivityNet	TODO	Download	Download
VATEX	TODO	TODO	TODO
Charades	TODO	TODO	TODO

Model Zoo

Checkpoint	Google Cloud
MSR-VTT	Download
ActivityNet	Download
VATEX	Download
Charades	Download

Download CLIP Model

wget -P ./modules https://openaipublic.azureedge.net/clip/models/40d365715913c9da98579312b702a82c18be219cc2a73407c4526f58eba950af/ViT-B-32.pt
wget -P ./modules https://openaipublic.azureedge.net/clip/models/5806e77cd80f8b59890b7e101eabd078d9fb84e6937f9e85e4ecb61988df416f/ViT-B-16.pt

Compress Video

python preprocess/compress_video.py --input_root [raw_video_path] --output_root [compressed_video_path]

Train on MSR-VTT

python -m torch.distributed.launch --nproc_per_node=4 --master_port='30500' \
main_phanet_retrieval.py --do_train --num_thread_reader=8 \
--epochs=5 --batch_size=128 --batch_size_val 64 --n_display=50 \
--train_csv ../DataSet/MSRVTT/data/file/MSRVTT_train.9k.csv \
--val_csv ../DataSet/MSRVTT/data/file/MSRVTT_JSFUSION_test.csv \
--data_path ../DataSet/MSRVTT/data/file/MSRVTT_data.json \
--features_path ../DataSet/MSRVTT/data/compressed/clip4clip_video_frame_input \
--output_dir ../Model/Experience/phanet_msrvtt_vit32 \
--log_dir ../Log/Experience/phanet_msrvtt_vit32 \
--visualize_dir ../Visualize/Experience/phanet_msrvtt_vit32 \
--lr 1e-4 --max_words 32 --max_frames 12 \
--datatype msrvtt --expand_msrvtt_sentences \
--feature_framerate 1 --coef_lr 1e-3 \
--freeze_layer_num 0  --slice_framepos 2 \
--t_indiv_protos_num 3 --v_indiv_protos_num 3 \
--t_local_protos_num 1 --v_local_protos_num 1 \
--kl_loss_flag --kl_loss_weight 0.2 \
--protos_contrastive_loss_flag --protos_contrastive_loss_weight 0.2 \
--loose_type --linear_patch 2d \
--pretrained_clip_name ViT-B/32

Train on ActivityNet

python -m torch.distributed.launch --nproc_per_node=8 --master_port='30501' \
main_phanet_retrieval.py --do_train --num_thread_reader=8 \
--epochs=20 --batch_size=64 --batch_size_val 64 --n_display=50 \
--data_path ../DataSet/ActivityNet/data/compressed/split_file \
--features_path ../DataSet/ActivityNet/data/compressed/clip4clip_video_frame_input \
--output_dir ../Model/phanet_activity_vit32 \
--log_dir ../Log/phanet_activity_vit32 \
--visualize_dir ../Visualize/phanet_activity_vit32 \
--lr 1e-4 --max_words 64 --max_frames 64 \
--datatype activity \
--feature_framerate 1 --coef_lr 1e-3 \
--freeze_layer_num 0  --slice_framepos 2 \
--t_indiv_protos_num 3 --v_indiv_protos_num 3 \
--t_local_protos_num 1 --v_local_protos_num 1 \
--kl_loss_flag --kl_loss_weight 0.2 \
--protos_contrastive_loss_flag --protos_contrastive_loss_weight 0.2 \
--loose_type --linear_patch 2d --sim_header seqTransf \
--pretrained_clip_name ViT-B/32

Train on VATEX

python -m torch.distributed.launch --nproc_per_node=4 --master_port='30502' \
main_phanet_retrieval.py --do_train --num_thread_reader=8 \
--epochs=5 --batch_size=128 --batch_size_val 128 --n_display=50 \
--data_path ../DataSet/VATEX/data/compressed/split_file \
--features_path ../DataSet/VATEX/data/compressed/clip4clip_video_frame_input \
--output_dir ../Model/phanet_vatex_vit32 \
--log_dir ../Log/phanet_vatex_vit32 \
--visualize_dir ../Visualize/phanet_vatex_vit32 \
--lr 1e-4 --max_words 32 --max_frames 12 \
--datatype vatex \
--feature_framerate 1 --coef_lr 1e-3 \
--freeze_layer_num 0  --slice_framepos 2 \
--t_indiv_protos_num 3 --v_indiv_protos_num 3 \
--t_local_protos_num 1 --v_local_protos_num 1 \
--kl_loss_flag --kl_loss_weight 0.2 \
--protos_contrastive_loss_flag --protos_contrastive_loss_weight 0.2 \
--loose_type --linear_patch 2d --sim_header seqTransf \
--pretrained_clip_name ViT-B/32

Train on Charades**

python -m torch.distributed.launch --nproc_per_node=4 --master_port='30503' \
main_phanet_retrieval.py --do_train --num_thread_reader=8 \
--epochs=5 --batch_size=128 --batch_size_val 64 --n_display=50 \
--data_path ../DataSet/Charades/data/compressed/split_file \
--features_path ../DataSet/Charades/data/compressed/clip4clip_video_frame_input \
--output_dir ../Model/phanet_charades_vit32 \
--log_dir ../Log/phanet_charades_vit32 \
--visualize_dir ../Visualize/phanet_charades_vit32 \
--lr 1e-4 --max_words 32 --max_frames 12 \
--datatype charades \
--feature_framerate 1 --coef_lr 1e-3 \
--freeze_layer_num 0  --slice_framepos 2 \
--t_indiv_protos_num 3 --v_indiv_protos_num 3 \
--t_local_protos_num 1 --v_local_protos_num 1 \
--kl_loss_flag --kl_loss_weight 0.2 \
--protos_contrastive_loss_flag --protos_contrastive_loss_weight 0.2 \
--loose_type --linear_patch 2d --sim_header seqTransf \
--pretrained_clip_name ViT-B/32

🎗️ Acknowledgments

Our code is based on HBI and CLIP4Clip. We thank the original authors for their open-sourcing.