Reproduce “ThoughtFold: Folding Reasoning Chains via Introspective Preference Learning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Ziyan Liu, Xueda Shen, Yuzhe Gu, Songyang Gao, Kuikun Liu, Guangran Cheng, Chengqi Lyu, Dahua Lin and 2 more
A framework called ThoughtFold enhances Large Reasoning Models (LRMs) by reducing redundant explorations within their reasoning chains through introspective preference learning. This approach improves reasoning efficiency and accuracy, leading to an absolute accuracy increase of 0.98% to 2.82% and a token length reduction of 39.4% to 56.1% on various benchmarks.
Key insights
- Introspective redundancy identification via a prune-and-verify strategy is effective in precisely identifying superfluous reasoning steps within correct trajectories.
- A dynamic mask strategy, integrated into a preference optimization framework, provides crucial step-level credit assignment to encourage concise and accurate reasoning paths.
- A joint optimization objective, balancing outcome-based RLVR and fine-grained preference learning, successfully maintains reasoning accuracy while significantly enhancing efficiency.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.