Reproduce “Machine Vision Therapy: Multimodal Large Language Models Can Enhance Visual Robustness via Denoising In-Context Learning” on OpenResearch

About this paper

Zhuo Huang, Chang Liu, Yinpeng Dong, Hang Su, Shibao Zheng, Tongliang Liu

Machine Vision Therapy leverages multimodal large language models to generate denoised supervision, effectively improving the out-of-distribution robustness of conventional computer vision models while significantly reducing the need for extensive human annotation. The framework demonstrated substantial accuracy gains across various OOD benchmarks and robustness tasks.

Key insights

• MLLMs can effectively enhance vision model robustness indirectly by acting as supervisors, rather than direct task solvers, thereby overcoming their inherent task incompatibility for precise visual recognition.
• Reframing complex classification problems into denoising True/False questions via Denoising In-Context Learning (DICL) is a powerful strategy to align MLLM reasoning with specific vision tasks.
• A small, confidence-based human-annotated support set is sufficient to estimate a transition matrix, enabling MLLMs to generate high-quality, rectified supervision for entire out-of-distribution datasets with minimal human effort.

Summary by alphaXiv.