Reproduce “See, Think, Learn: A Self-Taught Multimodal Reasoner” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Sourabh Sharma, Sonam Gupta, Sadbhawna
See-Think-Learn (STL) is a self-training framework designed to enhance multimodal reasoning in Vision-Language Models (VLMs) by explicitly integrating visual perception and discriminative learning. The framework, which utilizes structured reasoning prompts and negative rationales, surpassed previous self-training methods like STaR by an average of 3% and R3V by 7% on LLaVA-v1.5-7B, demonstrating competitive performance with models fine-tuned on human-annotated rationales on stronger VLMs.
Key insights
- Structuring rationale generation to first describe visual observations ("see") and then infer ("think") is crucial for grounding VLM reasoning in perceptual details.
- Incorporating discriminative negative rationales, explaining why incorrect options are wrong, significantly enhances a VLM's ability to distinguish valid reasoning paths.
- High-quality, self-generated rationales can be competitive with or surpass human-annotated rationales, offering a scalable and cost-effective method for VLM improvement without external supervision.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.