Reproduce “Learning to Think: Information-Theoretic Reinforcement Fine-Tuning for LLMs” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jingyao Wang, Wenwen Qiang, Zeen Song, Changwen Zheng, Hui Xiong
The Learning to Think (L2T) framework fine-unes large language models to achieve higher reasoning accuracy with significantly fewer tokens. It introduces a universal, information-theoretic dense process reward that guides models to adaptively optimize reasoning depth.
Key insights
- Integrating a dense, internally derived information-theoretic reward enables LLMs to learn both effective and efficient reasoning, balancing accuracy with token consumption.
- Quantifying the information gain in model parameters and penalizing redundant parameter updates at each step effectively guides the model to learn parsimonious and relevant reasoning.
- Decomposing complex reasoning into a sequence of episodic decisions provides granular control and feedback, allowing LLMs to adapt their reasoning depth dynamically to task requirements.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.