Reproduce “Multi-CLS BERT: An Efficient Alternative to Traditional Ensembling” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Haw-Shiuan Chang, Ruei-Yao Sun, Kathryn Ricci, Andrew McCallum
Multi-CLS BERT, developed by researchers at UMass Amherst and Amazon, introduces a method to achieve ensemble-like performance within a single BERT model by utilizing multiple CLS tokens. This approach yields accuracy improvements and enhanced confidence estimates, notably outperforming a larger BERT model in low-data scenarios and reducing inference time by nearly 5x compared to traditional ensembling.
Key insights
- Ensemble-like benefits can be intrinsically integrated into a single BERT model by employing multiple CLS tokens and carefully designed diversification mechanisms.
- Explicitly fostering diversity among these multiple internal CLS representations during both pre-training and fine-tuning is crucial for achieving performance gains similar to traditional ensembles.
- The proposed method effectively balances collaborative and independent learning among internal 'facets,' leading to better generalization and more reliable confidence estimates without significant overhead.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.