Reproduce “TextSquare: Scaling up Text-Centric Visual Instruction Tuning” on OpenResearch

About this paper

Jingqun Tang, Chunhui Lin, Zhen Zhao, Shu Wei, Binghong Wu, Qi Liu, Yangfan He, Kuan Lu and 10 more

Researchers from ByteDance Inc., East China Normal University, Huazhong University of Science and Technology, University of Minnesota, and Cornell University developed TextSquare, an open-source Multimodal Large Language Model. They introduced the "Square" pipeline to generate a 9.1 million-pair high-quality text-centric Visual Question Answering (VQA) dataset, which enabled TextSquare to achieve performance comparable to or surpassing leading closed-source models like GPT4V and Gemini Pro on text-intensive benchmarks.

Key insights

• High-quality and large-scale instruction tuning data, particularly when enriched with explicit reasoning pathways, is a critical factor for boosting MLLM performance and effectively mitigating hallucinations in text-centric VQA.
• Rigorous data filtering mechanisms, such as MLLM self-evaluation and consistency checks during synthetic data generation, are crucial for producing reliable and effective training datasets that significantly enhance downstream model performance.
• An empirical scaling law was observed in text-centric VQA, demonstrating that model performance and training convergence loss improve logarithmically with increasing instruction tuning data volume, validating the necessity of massive datasets like Square-10M.

Summary by alphaXiv.