Reproduce “EQA-RM: A Generative Embodied Reward Model with Test-time Scaling” on OpenResearch

About this paper

Yuhang Chen, Zhen Tan, Tianlong Chen

A generative embodied reward model, EQA-RM, and the first dedicated benchmark for EQA reward models, EQAREWARDBENCH, are presented. EQA-RM demonstrates superior evaluation accuracy for Embodied Question Answering trajectories, particularly when utilizing its novel test-time scaling, which enables it to surpass leading VLM-as-a-Judge baselines.

Key insights

• Generative reward models, particularly when combined with test-time scaling, can significantly enhance evaluation precision and robustness in complex multimodal tasks like Embodied Question Answering.
• A contrastive reinforcement learning strategy, utilizing systematically perturbed data (e.g., shuffled frames, jumbled reasoning), effectively teaches a reward model to discern fine-grained spatio-temporal and logical distinctions in embodied agent trajectories.
• A two-stage training process involving rejective fine-tuning for initial format learning and subsequent contrastive policy optimization is effective for developing specialized multimodal reward models.

Summary by alphaXiv.