Reproduce “Magistral” on OpenResearch

About this paper

Mistral-AI, Abhinav Rastogi, Albert Q. Jiang, Andy Lo, Gabrielle Berrada, Guillaume Lample, Jason Rute, Joep Barmentlo and 92 more

Mistral AI introduces Magistral, its inaugural reasoning model, which leverages a custom Reinforcement Learning from Verifiable Rewards (RLVR) pipeline to significantly enhance the reasoning abilities of its base models. Magistral Medium, trained purely with RL, achieved a nearly 50% increase on AIME '24 (pass@1), while Magistral Small demonstrated that RL effectively boosts distilled models and both models showed surprising improvements in multimodal reasoning despite text-only training.

Key insights

• Pure Reinforcement Learning (RL) alone can achieve substantial reasoning performance gains on large models without initial distillation from other reasoning models.
• Text-only RL training not only preserves but surprisingly improves multimodal reasoning capabilities, suggesting deep generalization of learned reasoning patterns.
• Detailed ablations revealed that controlling exploration via the PPO-like clip-higher threshold was more stable than entropy bonuses, and specific batching strategies influenced training stability.

Summary by alphaXiv.