Reproduce “GRACE: Gradient-aligned Reasoning Data Curation for Efficient Post-training” on OpenResearch

About this paper

Junjie Li, Ziao Wang, NingXuan Ma, Jianghong Ma, Xiaofeng Zhang

Researchers developed GRACE, a gradient-aligned reasoning data curation method that models reasoning traces as sequences of optimization events to efficiently post-train large language and vision-language models. Using only 20% of curated data, GRACE achieved 108.8% average relative performance compared to training on the full MMathCoT-1M dataset, significantly reducing computational overhead while improving performance.

Key insights

• Reasoning data can be viewed as dynamic sequences of optimization events, allowing for a more granular assessment of individual step contributions beyond sample-level evaluation.
• The optimization utility of a reasoning step is effectively captured by its gradient alignment with the final task objective and its coherence with the historical reasoning trajectory.
• An efficient representation-level proxy can accurately approximate parameter-space gradient directions, enabling scalable step-level data valuation for large datasets.

Summary by alphaXiv.