Reproduce “InfiMM-WebMath-40B: Advancing Multimodal Pre-Training for Enhanced Mathematical Reasoning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Xiaotian Han, Yiren Jian, Xuefeng Hu, Haogeng Liu, Yiqi Wang, Qihang Fan, Yuang Ai, Huaibo Huang and 3 more
Researchers from ByteDance and the Chinese Academy of Sciences introduce InfiMM-WebMath-40B, the first publicly available large-scale multimodal pre-training dataset for mathematics, which significantly enhances mathematical reasoning in MLLMs. Models trained on this dataset achieve state-of-the-art performance among open-source models on benchmarks like MathVerse and We-Math.
Key insights
- High-quality, interleaved image-text pre-training data is crucial for developing robust mathematical reasoning in MLLMs, providing deeper knowledge acquisition than instruction tuning alone.
- A meticulously curated domain-specific dataset, even with fewer tokens, can yield competitive or superior mathematical reasoning performance compared to models trained on much larger, less specialized datasets.
- Continuous pre-training on domain-specific multimodal data, such as InfiMM-WebMath-40B, consistently and substantially boosts MLLM performance on complex math benchmarks, validating its efficacy as a foundational resource.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.