Reproduce “Beyond Language Modeling: An Exploration of Multimodal Pretraining” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Shengbang Tong, David Fan, John Nguyen, Ellis Brown, Gaoyue Zhou, Shengyi Qian, Boyang Zheng, Théophane Vallaeys and 13 more
The unified model trains language and visual prediction from scratch, using next-token prediction for text and flow matching to generate visual representations. It applies the same framework to video and text-formatted navigation actions, without an action-specific adapter. In navigation tests, 1% in-domain data gives competitive performance, while MoE scaling narrows the fitted language–vision token-allocation gap, a trend limited to studied scales.
Key insights
- Representation Autoencoders (RAE) using semantic encoders provide an optimal unified visual representation, excelling in both visual understanding and generation, simplifying multimodal architectures.
- Diverse visual and language data are complementary, with multimodal co-training yielding synergy that enhances downstream capabilities like VQA and naturally leads to world modeling.
- Mixture-of-Experts (MoE) architectures enable efficient and effective multimodal scaling, naturally inducing modality specialization and harmonizing the scaling asymmetry between vision and language.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.