Reproduce “4M-21: An Any-to-Any Vision Model for Tens of Tasks and Modalities” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Roman Bachmann, Oğuzhan Fatih Kar, David Mizrahi, Ali Garjani, Mingfei Gao, David Griffiths, Jiaming Hu, Afshin Dehghan and 1 more
Researchers at EPFL and Apple introduce 4M-21, a generalist vision model that processes and generates 21 diverse visual modalities using a unified Transformer architecture. The model achieves competitive out-of-the-box performance across various tasks, often matching or surpassing specialized pseudo-labeling networks, and enables fine-grained, steerable multimodal generation.
Key insights
- A single generalist vision model can scale effectively to encompass tens of diverse modalities (21 in 4M-21) without sacrificing out-of-the-box performance relative to specialist models.
- Unifying diverse visual inputs and outputs through discrete tokenization and a masked modeling objective within a Transformer architecture allows for robust 'any-to-any' prediction and generation.
- Massive pseudo-labeling combined with strategic multi-dataset co-training is a viable method for distilling knowledge from numerous specialized models into a single, comprehensive foundation model.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.