Reproduce “Towards Understanding the Overfitting Phenomenon of Deep Click-Through Rate Prediction Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zhao-Yu Zhang, Xiang-Rong Sheng, Yujing Zhang, Biye Jiang, Shuguang Han, Hongbo Deng, Bo Zheng
This paper investigates the 'one-epoch phenomenon' in deep Click-Through Rate (CTR) prediction models, where performance abruptly degrades at the second training epoch. The research empirically demonstrates that this behavior stems from a rapid change in the joint distribution of embeddings for trained samples, particularly due to high-dimensional sparse features, causing the MLP layers to quickly overfit.
Key insights
- The 'one-epoch phenomenon' is driven by a rapid change in the joint distribution of embeddings for 'trained' versus 'untrained' samples, particularly for sparse fine-grained features.
- The MLP layers of deep CTR models quickly adapt to this suddenly separable distribution of 'trained' embeddings at the beginning of the second epoch, causing immediate overfitting.
- This specific overfitting behavior is fundamentally different from traditional deep learning overfitting due to the unique characteristics of high-dimensional, sparse data and Embedding-MLP architectures.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.