Reproduce “UniMS: A Unified Framework for Multimodal Summarization with Knowledge Distillation” on OpenResearch

About this paper

Zhengkun Zhang, Xiaojun Meng, Yasheng Wang, Xin Jiang, Qun Liu, Zhenglu Yang

UniMS, developed by researchers from Nankai University and Huawei Technologies, introduces a unified framework for multimodal summarization that simultaneously performs abstractive and extractive text summarization, alongside image selection. It leverages knowledge distillation from a pre-trained vision-language model for image relevance and incorporates a visual-guided decoder, achieving new state-of-the-art results across these tasks.

Key insights

• A unified framework that simultaneously performs abstractive and extractive text summarization, alongside image selection, can achieve superior overall performance in multimodal summarization.
• Knowledge distillation from powerful pre-trained vision-language models like CLIP offers an effective and robust alternative for learning text-image relevance, overcoming the long-standing dependency on image captions.
• Simple linear projection for visual embeddings can yield comparable performance to complex pre-trained visual backbones, highlighting an efficiency advantage without sacrificing accuracy in multimodal summarization.

Summary by alphaXiv.