Reproduce “Exploring Feature-based Knowledge Distillation for Recommender System: A Frequency Perspective” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zhangchi Zhu, Wei Zhang
Researchers at East China Normal University introduce FreqD, a lightweight knowledge distillation method for recommender systems that reweights feature knowledge from a frequency perspective. It consistently surpasses state-of-the-art baselines, improving Recall@20 by up to 18.23% on CiteULike and enabling student models to achieve teacher-level performance with significantly fewer parameters.
Key insights
- Standard feature-based knowledge distillation loss implicitly assigns equal weights to all frequency components of knowledge, as rigorously shown through theoretical proof.
- Low-frequency knowledge components, which characterize common properties among nodes, are found to be more difficult for smaller student models to acquire but are critical for recommendation performance.
- By differentially weighting knowledge components based on their frequency, particularly by emphasizing low-frequency knowledge, student models can more effectively learn and improve recommendation accuracy.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.