Reproduce “RAST: Reasoning Activation in LLMs via Small-model Transfer” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Siru Ouyang, Xinyu Zhu, Zilin Xiao, Minhao Jiang, Yu Meng, Jiawei Han
RAST introduces a method to activate latent reasoning capabilities in large language models by transferring logit-level adjustments from smaller, reinforcement learning-tuned models at inference time. This approach significantly enhances reasoning performance on mathematical and coding benchmarks while reducing GPU memory requirements by over 50% compared to direct RL training.
Key insights
- Reinforcement learning (RL) primarily activates latent reasoning capabilities within large language models by selectively adjusting the probabilities of a small, localized set of reasoning-critical tokens.
- The distributional shifts induced by RL, particularly for reasoning behaviors, are largely model-size invariant, enabling their transfer across different model scales.
- Injecting logit-level adjustments from small RL-tuned models into larger base models at inference time can effectively enhance complex reasoning without requiring full RL fine-tuning on the large model.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.