Reproduce “Unbundling Multimodal Vision Encoding” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Ellis Brown, Chris Hoang, Shengyi Qian, Shang-Wen Li, Mike Lewis, Marjan Ghazvininejad, Luke Zettlemoyer, Lucas Beyer and 4 more
The authors compare multimodal model designs by separating visual pretraining, dedicated vision capacity, and where image processing occurs, while controlling data and compute. At the longest budget for a 1.7B backbone, in-fusion reaches 49.2% Avg. VQA, versus 48.9% for a scratch encoder and 44.8% without dedicated vision processing. These dense-backbone experiments do not rule out benefits from external data.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.