Reproduce “Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Tasks” on OpenResearch

About this paper

Wenhui Wang, Hangbo Bao, Li Dong, Johan Bjorck, Zhiliang Peng, Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed and 3 more

Microsoft's BEIT-3 model introduces a general-purpose multimodal foundation model leveraging a Multiway Transformer architecture and a unified masked data modeling objective for pretraining. It achieves state-of-the-art results across a wide range of vision and vision-language tasks, including 84.03% VQA accuracy on VQA v2.0 and 62.8 mIoU on ADE20K semantic segmentation, while exclusively utilizing publicly available datasets.

Key insights

• Unified masked data modeling (generative pretraining) is highly effective for multimodal learning, implicitly establishing cross-modal alignment and reducing the need for complex multi-objective pretraining.
• The Multiway Transformer architecture provides a robust, unified backbone capable of adapting to diverse task formats (vision, language, fusion, retrieval, generation) within a single model.
• State-of-the-art performance can be achieved across a broad range of vision and vision-language tasks solely using publicly available datasets, promoting accessibility and open research.

Summary by alphaXiv.