Reproduce “Enhancing VLM Reward Models Through Structure-Aware Fine-Tuning” on OpenResearch

About this paper

Pyrros Koussios, Chenhao Li, Xin Chen, Andreas Krause

Researchers at ETH Zürich developed Structure-Aware Fine-Tuning (SAFT), a self-supervised method that refines Vision-Language Model (VLM) reward models by leveraging intrinsic structural priors of the environment. This approach significantly enhances policy learning efficiency and reward model alignment with ground-truth objectives, while substantially reducing human labeling requirements compared to preference-based methods.

Key insights

• VLM reward failures often stem from a 'structural brittleness'—a lack of adherence to fundamental environmental symmetries and smoothness—rather than solely semantic misunderstanding.
• Task-inherent structural priors can serve as a potent, label-free source of regularization, effectively improving the stability and accuracy of VLM reward models.
• Self-supervised injection of structural inductive biases into VLM reward models can substantially reduce the human labeling effort typically associated with preference-based learning methods.

Summary by alphaXiv.