Reproduce “BatchEnsemble: An Alternative Approach to Efficient Ensemble and Lifelong Learning” on OpenResearch

About this paper

Yeming Wen, Dustin Tran, Jimmy Ba

BatchEnsemble introduces a parameter-efficient ensembling technique that reduces memory and computational costs while maintaining accuracy and improving predictive uncertainty in deep neural networks. The method also provides a scalable solution for lifelong learning, capable of handling a large sequence of tasks without catastrophic forgetting.

Key insights

• Efficient parameterization using shared weights and low-rank perturbations can achieve deep ensemble benefits with substantially reduced memory and computational costs.
• The method provides competitive predictive uncertainty estimates and calibration, demonstrating its effectiveness for out-of-distribution detection and exploration-exploitation tasks.
• By associating distinct low-rank components with tasks, BatchEnsemble offers a scalable and forgetting-resistant approach to lifelong learning, overcoming memory limitations of prior methods.

Summary by alphaXiv.