Reproduce “Rethinking Reward Signals in Video GRPO: When Scores Become Targets” on OpenResearch

About this paper

Rui Li, Yuanzhi Liang, Ziqi Ni, Haibin Huang, Chi Zhang, Xuelong Li

TaRoS reframes reward design in video GRPO as a scheduling problem: reward components should be weighted according to both their achieved quality and their ability to distinguish samples within a generation group. By adapting mixture weights with thresholded performance and Hoyer-based score separation, the method reduces reward saturation and improves VBench results across Wan2.1 and HunyuanVideo models, with the largest reported gain reaching 80.56 from a 79.58 untrained baseline on Wan2.1-T2V-1.3B.

Key insights

• A reward used in GRPO must be evaluated not only by its absolute quality signal but also by its current ability to rank competing samples generated for the same prompt.
• Reward-model scale or parameter count is not a substitute for reward-signal management: the 72B evaluator can still exhibit category-dependent behavior and eventual saturation.
• Adaptive scheduling can alter the optimization direction over training and across prompts, addressing multi-objective conflict without the additional cost of explicit gradient-surgery methods.

Summary by alphaXiv.