Reproduce “Continual SFT Matches Multimodal RLHF with Negative Supervision” on OpenResearch

About this paper

Ke Zhu, Yu Wang, Yanpeng Sun, Qiang Chen, Jiangjiang Liu, Gang Zhang, Jingdong Wang

Researchers from Nanjing University and Baidu VIS developed Negative Supervised Fine-Tuning (nSFT), an efficient approach for aligning Vision-Language Models that integrates explicit negative supervision, achieving performance comparable to or surpassing computationally intensive multimodal Reinforcement Learning from Human Feedback (RLHF) methods while significantly reducing training costs.

Key insights

• The primary advantage of multimodal RLHF, particularly DPO, over standard continual SFT stems from its ability to leverage negative supervision, rather than its complex reinforcement learning structure.
• Explicitly extracting and integrating negative supervision via an LLM-driven conversation reconstruction process enables a simpler SFT method to match or exceed RLHF performance in VLM alignment.
• A structured Vision Error Codebook (VEC) is crucial for guiding the LLM in precisely identifying and correcting fine-grained multimodal hallucinations and errors within rejected responses.

Summary by alphaXiv.