Reproduce “Can Large Models Teach Student Models to Solve Mathematical Problems Like Human Beings? A Reasoning Distillation Method via Multi-LoRA Interaction” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Xinhe Li, Jiajun Liu, Peng Wang
Researchers at Southeast University developed LoRID (Multi-LoRA Interaction for mathematical reasoning Distillation), a method that distills advanced mathematical reasoning into Small Language Models (SLMs) by mimicking human intuitive and deep knowledge-driven thinking. This approach uses multiple LoRA blocks for distinct reasoning modes, achieving notable accuracy improvements on benchmarks like GSM8K and approaching the performance of much larger, closed-source models.
Key insights
- Mimicking human dual-process thinking, with separate intuitive and knowledge-guided reasoning modules, is effective for distilling complex capabilities into SLMs.
- Explicitly teaching SLMs to generate and apply generalized mathematical knowledge, rather than just solving examples, enhances their reasoning abilities.
- An iterative, self-correcting interaction between these distinct reasoning modules allows SLMs to achieve more robust and accurate solutions with high parameter efficiency.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.