Reproduce “NeuroTrails: Training with Dynamic Sparse Heads as the Key to Effective Ensembling” on OpenResearch

About this paper

Bram Grooten, Farid Hasanov, Chenxiang Zhang, Qiao Xiao, Boqian Wu, Zahra Atashgahi, Ghada Sokar, Shiwei Liu and 4 more

NeuroTrails introduces a training paradigm that enables efficient deep learning ensembles by combining shared early layers with dynamic sparse training. This method creates distinct neural pathways within a shared network, achieving ensemble-level accuracy and robustness with computational costs approaching those of single models.

Key insights

• Dynamic sparse training enables different ensemble heads to carve out specialized neural pathways through a shared backbone, creating the necessary prediction diversity for effective ensembling.
• There exists a "Goldilocks zone" for prediction diversity, where NeuroTrails achieves an optimal balance between insufficient diversity (TreeNets) and excessive, costly diversity (full ensembles).
• An optimal backbone length of approximately one-third of the total network depth provides sufficient shared feature extraction while allowing enough independent layers for head specialization.

Summary by alphaXiv.