Reproduce “Entropy-Guided Token Dropout: Training Autoregressive Language Models with Limited Domain Data” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jiapeng Wang, Yiwen Hu, Yanzipeng Gao, Haoyu Wang, Shuo Wang, Hongyu Lu, Jiaxin Mao, Wayne Xin Zhao and 2 more
Researchers from Renmin University of China and collaborators developed Entropy-Guided Token Dropout (EntroDrop), an adaptive regularization method for autoregressive language models. EntroDrop addresses multi-epoch training degradation by selectively masking low-entropy tokens and dynamically adjusting regularization intensity, leading to improved performance and extended effective training on scarce domain data.
Key insights
- Multi-epoch training initially benefits from limited domain data but eventually causes performance degradation in autoregressive models due to overfitting.
- The observed performance degradation is non-uniform; it is primarily driven by models overfitting on predictable 'low-entropy' tokens, while their ability to generalize on 'high-entropy' tokens deteriorates.
- Intelligent, adaptive regularization targeting predictable tokens and dynamically adjusting strength can rebalance learning dynamics and make multi-epoch training on scarce data more effective.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.