Reproduce “VLMine: Long-Tail Data Mining with Vision Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Mao Ye, Gregory P. Meyer, Zaiwei Zhang, Dennis Park, Siva Karthik Mustikovela, Yuning Chai, Eric M Wolff
VLMine leverages Vision Language Models to semantically identify rare, long-tail examples within large unlabeled datasets, demonstrating improved model robustness on these under-represented instances in both image classification and 3D object detection tasks. The method systematically outperforms uncertainty-based baselines, particularly for tail classes and objects, by combining VLM-derived semantic novelty with existing data mining signals.
Key insights
- Vision Language Models (VLMs) offer a unique and complementary semantic signal for identifying long-tail examples, distinct from the uncertainty-based signals typically used by task-specific models.
- The extensive "world knowledge" of VLMs can be effectively leveraged for data curation, enabling the identification of subtle intra-class long-tail variations crucial for robust 3D perception in autonomous systems.
- Combining multiple, diverse data mining signals through a Pareto mining strategy proves more effective for discovering long-tail data than relying on single signals or simple score aggregations.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.