Adapter-X: A Novel General Parameter-Efficient Fine-Tuning Framework for Vision

Arxiv Paper • Citation

🏃 Intro Adapter-X

Parameter-efficient fine-tuning (PEFT) has become increasingly important as foundation models continue to grow in both popularity and size. Adapter has been particularly well-received due to their potential for parameter reduction and adaptability across diverse tasks. However, striking a balance between high efficiency and robust generalization across tasks remains a challenge for adapter-based methods. We analyze existing methods and find that: 1) parameter sharing is the key to reducing redundancy; 2) more tunable parameters, dynamic allocation, and block-specific design are keys to improving performance.

Inspired by this insight, we introduce a novel framework named Adapter-X. First, a Sharing Mixture of Adapters (SMoA) module is proposed to fulfill token-level dynamic allocation, increased tunable parameters, and inter-block sharing at the same time. Second, some block-specific designs like Prompt Generator (PG) are introduced to further enhance the ability of adaptation.

Extensive experiments across 2D image and 3D point cloud modalities demonstrate that Adapter-X represents a significant milestone as it is the first to outperform full fine-tuning in both 2D image and 3D point cloud modalities with significantly fewer parameters, i.e., only 0.20% and 1.88% of original trainable parameters for 2D and 3D classification tasks.

🚩 News

• [2024/06/05] Upload paper and init project.

⚡ Quick Start

Build Environment

💻 Training and Evaluation

Training guidance

📖 Citation

If you find our code or paper helps, please consider citing:

@article{li2024adapter,
  title={Adapter-X: A Novel General Parameter-Efficient Fine-Tuning Framework for Vision},
  author={Li, Minglei and Ye, Peng and Huang, Yongqi and Zhang, Lin and Chen, Tao and He, Tong and Fan, Jiayuan and Ouyang, Wanli},
  journal={arXiv preprint arXiv:2406.03051},
  year={2024}
}

Acknowledgments

Thanks to these amazing works: NOAH, Bi-adapter, DAPT, Switch Transformer and X-MoE.