Reproduce “Optimizing Test-Time Compute via Meta Reinforcement Fine-Tuning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yuxiao Qu, Matthew Y. R. Yang, Amrith Setlur, Lewis Tunstall, Edward Emanuel Beeching, Ruslan Salakhutdinov, Aviral Kumar
Meta Reinforcement Fine-Tuning (MRT) enables large language models to more efficiently utilize test-time compute for complex reasoning tasks. The approach yields 2-3x relative accuracy gains and 1.5x-5x token efficiency improvements on math reasoning benchmarks by optimizing for progress during the reasoning process.
Key insights
- Conventional LLM training with fixed budgets and sparse outcome rewards leads to inefficient token usage and poor "discovery" of solutions for complex problems.
- Framing test-time compute as a meta-RL problem allows for optimizing the reasoning process itself, encouraging an explore-exploit balance equivalent to minimizing cumulative regret.
- Introducing a dense "progress" reward, defined as the change in success likelihood after each reasoning episode, directly incentivizes LLMs to make meaningful, consistent advancements throughout their reasoning trajectory.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.