Reproduce “DocPO: Advancing Document Policy Optimization via Tailored Step-Aware Rewards” on OpenResearch

About this paper

Yunhao Wang, Binghong Wu, Zhenyu Huang, Jiacheng Shi, Shuo Huang, Tinghao Yu, Feng Zhang

DocPO, developed by Tencent Hunyuan, advances document parsing by employing tailored, step-aware rewards within a reinforcement learning framework. The approach achieves state-of-the-art performance with a Qwen2.5-VL-3B backbone, securing an overall score of 95.49 on OmniDocBench and 93.76 on DocElemHard for text, formulas, and tables, outperforming specialized vision language models without architectural modifications.

Key insights

• Dynamically sharpening reference-based rewards via Step-Aware Annealing significantly enhances the discriminability of signals in high-accuracy regimes, leading to more effective fine-grained policy optimization.
• Tailoring reward functions to the intrinsic properties of each document element (e.g., prioritizing structural correctness for tables, syntactic validity and semantic equivalence for formulas) is crucial for robust and accurate document parsing.
• Superior document parsing performance can be achieved through sophisticated reward engineering in RL, even when leveraging general-purpose Vision Language Models, reducing the need for specialized architectural designs.

Summary by alphaXiv.