Reproduce “Unleashing the Reasoning Potential of Pre-trained LLMs by Critique Fine-Tuning on One Problem” on OpenResearch

About this paper

Yubo Wang, Ping Nie, Kai Zou, Lijun Wu, Wenhu Chen

One-shot Critique Fine-Tuning (CFT) activates latent reasoning abilities in pre-trained large language models using a single problem, outperforming supervised fine-tuning and matching reinforcement learning with 1/15th to 1/20th the computational cost. This approach trains models to critique diverse incorrect solutions, fostering a deeper understanding of reasoning patterns.

Key insights

• Learning from critiques of diverse incorrect solutions provides a richer and more effective supervisory signal for reasoning than directly learning from correct solutions.
• A minimal, qualitatively rich dataset derived from a single problem can efficiently activate latent reasoning capabilities in LLMs, outperforming methods requiring large datasets or complex reinforcement learning.
• Critique fine-tuning offers a significantly more computationally efficient and stable approach compared to reinforcement learning for improving LLM reasoning performance.

Summary by alphaXiv.