Reproduce “Improving Reasoning in Vision-Language Models via Perception Verified Self-Training” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Sourabh Sharma, Sonam Gupta, Sadbhawna
A framework called Perception Verified Self-Training enhances reasoning in Vision-Language Models by explicitly verifying visual perception within the self-training process. It achieved up to 16% improvement over existing self-training baselines on M3CoT benchmarks and reduced visual hallucinations, demonstrating robust, perceptually grounded multimodal reasoning.
Key insights
- Explicitly disentangling perception (via captions) from reasoning allows for targeted verification, effectively mitigating visual hallucinations and language shortcuts in VLM self-training.
- Unsupervised evaluation of self-generated captions using combined textual and visual similarity metrics (PerceptEval) is feasible and critical for ensuring perceptual grounding.
- A two-stage curriculum learning approach, guided by perception verification, systematically enhances robust and visually faithful reasoning in Vision-Language Models by progressively integrating perceptually verified rationales.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.