Reproduce “Reconstructive Visual Instruction Tuning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Haochen Wang, Anlin Zheng, Yucheng Zhao, Tiancai Wang, Zheng Ge, Xiangyu Zhang, Zhaoxiang Zhang
Reconstructive Visual Instruction Tuning (ROSS) enhances Large Multimodal Models (LMMs) by introducing a vision-centric denoising objective, improving fine-grained visual comprehension and significantly reducing hallucinations. The approach achieves state-of-the-art performance with a single visual encoder, often surpassing models that rely on multiple aggregated visual experts and even commercial LMMs.
Key insights
- Denoising as a reconstructive objective is superior to direct regression for handling spatial redundancy in visual signals, providing more meaningful supervision and reducing overfitting.
- Reconstruction-based teacher tokenizers, such as a continuous VAE, are more effective for the ROSS framework than discriminative ones (e.g., DINOv2) because they preserve sufficient image details for the reconstructive task.
- Intrinsic activation via vision-centric reconstruction (ROSS) can outperform or match extrinsic assistance (aggregating multiple experts) in enhancing LMM visual comprehension and reducing hallucinations, offering a more efficient and lightweight solution.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.