Reproduce “Mixture of Parrots: Experts improve memorization more than reasoning” on OpenResearch

About this paper

Samy Jelassi, Clara Mohri, David Brandfonbrener, Alex Gu, Nikhil Vyas, Nikhil Anand, David Alvarez-Melis, Yuanzhi Li and 2 more

The authors test whether expert capacity helps recall more than reasoning. In synthetic phone-book lookup, a mixture-of-experts model with 42 million active parameters exceeds a dense model with ten times as many parameters on learned-entry recall. Across benchmarks, experts help world knowledge more than commonsense and math; this does not establish performance on unseen facts or all reasoning tasks.

Key insights

• MoEs are theoretically and empirically more efficient for memorization tasks, leveraging their numerous experts to store vast amounts of information with fewer active parameters.
• For reasoning tasks, MoE performance is primarily determined by the number of active parameters (effective width) rather than the total number of experts, indicating a fundamental limitation when active computation is constrained.
• MoEs show a larger generalization gap on mathematical problems, suggesting they tend to memorize specific training examples instead of learning generalizable reasoning principles.

Summary by alphaXiv.