Reproduce “Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Models” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Matt Deitke, Christopher Clark, Sangho Lee, Rohun Tripathi, Yue Yang, Jae Sung Park, Mohammadreza Salehi, Niklas Muennighoff and 42 more

This work introduces Molmo, a family of state-of-the-art open-weight Vision-Language Models (VLMs), and PixMo, a comprehensive suite of VLM-independent training datasets. Molmo-72B achieves leading performance on academic benchmarks and human preference evaluations, demonstrating that high-performing VLMs can be built entirely from truly open data and models without relying on proprietary systems for data generation.

Key insights

  • High-quality human-annotated data, collected without reliance on proprietary VLMs, performs comparably to data generated by powerful commercial models for VLM training.
  • Architectural choices like overlapping multi-crops for vision input and text-only dropout during pre-training significantly improve VLM performance.
  • Targeted, VLM-independent synthetic data can effectively imbue models with robust, specialized skills, as demonstrated by leading performance in clock reading and counting.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.