Reproduce “ReaLM: Reflection-Enhanced Autonomous Reasoning with Small Language Models” on OpenResearch

About this paper

Yuanfeng Xu, Zehui Dai, Jian Liang, Jiapeng Guan, Guangrun Wang, Liang Lin, Xiaohui Lv

ReaLM, a reinforcement learning framework, enhances Small Language Models (SLMs) to achieve robust, autonomous, and generalizable reasoning by learning from diverse reasoning paths. It demonstrates average accuracy gains of 2.5% to 4.2% on various complex tasks, outperforming existing methods and effectively distilling domain-specific knowledge for vertical applications.

Key insights

• Learning from a diverse mix of both correct and incorrect reasoning paths, rather than just correct ones, significantly improves an SLM's robustness and ability to identify and avoid failure modes.
• Gradually reducing external guidance during training allows SLMs to achieve high-quality autonomous reasoning without sacrificing much performance, making them practical for standalone deployment.
• Injecting domain-specific rules and expert rationale into the CoT generation process effectively distills specialized knowledge, enabling SLMs to generalize and perform well in constrained "vertical domains."

Summary by alphaXiv.