ARMOR: Empowering Multimodal Understanding Model with Interleaved Multimodal Generation Capability

Jianwen Sun^1,2* · Yukang Feng^1,2* · Chuanhao Li⁵· Fanrui Zhang^2,3
Zizhen Li^1,2 · Jiaxin Ai^2,4 · Sizhuo Zhou^2,3 · Pengfei Zhou⁵
Yu Dai¹ · Shenglin Zhang¹ · Kaipeng Zhang^2,5†

¹Nankai University   ²Shanghai Innovation Institute
   ³University of Science and Technology of China   ⁴WUHAN University   
⁵Shanghai AI Laboratory
   
*equal contribution   †corresponding author

💡 News

• 2025/03/09: The technical report of ARMOR is released! Our code will be released soon.

📖 Introduction

This repo implements ARMOR, a unified visual tokenizer well-suited for both generation and understanding tasks. It operates within a single autoregressive framework to enable interleaved image-text inputs and outputs, autonomously selecting the most appropriate response modality depending on the query.

📖 Overview

Armor is a unified understanding and generation model improved based on a multimodal large language model (MLLM). It operates within a single autoregressive framework to enable interleaved image-text inputs and outputs, autonomously selecting the most appropriate response modality depending on the query. Building upon a pretrained MLLM, Armor employs a unified embedding space to represent both textual and visual information, thus reducing model complexity, and introduces an asymmetric encoder-decoder to unify generation and understanding. Through training on a meticulously curated, high-quality dataset of interleaved text and images with our proposed What or How to Generate (WoHG) method, Armor not only preserves much of the original model’s capabilities but also achieves impressive image generation performance. Coupled with a forward-switching mechanism, Armor enables highly natural text-image interleaved output, all while requiring minimal computational resources. The research findings indicate that enhancing a pretrained MLLM with an autoregressive architecture and an asymmetric encoder-decoder demonstrates substantial potential and research value for developing unified understanding and generation models. Furthermore, the results also reaffirm that a fully autoregressive approach remains a promising foundation for building unified large-scale model architectures.

🏆 Expirement

Understanding Performance

Generation Performance

🚀 Usage

1. Download VAE ckpt

huggingface-cli download --resume-download HerzogFL/chameleon_vae --local-dir code/VAE --local-dir-use-symlinks False

2. coming soon...

📞 Contact

• Jianwen Sun: sunjianwen@mail.nankai.edu.cn
• Yukang Feng: yukangfeng@mail.nankai.edu.cn
• Kaipeng Zhang: zhangkaipeng@pjlab.org.cn

🖊️ Citation

If you feel ARMOR useful in your project or research, please kindly use the following BibTeX entry to cite our paper. Thanks!

@misc{sun2025armorempoweringmultimodalunderstanding,
      title={ARMOR: Empowering Multimodal Understanding Model with Interleaved Multimodal Generation Capability}, 
      author={Jianwen Sun and Yukang Feng and Chuanhao Li and Fanrui Zhang and Zizhen Li and Jiaxin Ai and Sizhuo Zhou and Yu Dai and Shenglin Zhang and Kaipeng Zhang},
      year={2025},
      eprint={2503.06542},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2503.06542}, 
}