Reproduce “Pre-trained Language Models for Keyphrase Generation: A Thorough Empirical Study” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Di Wu, Wasi Uddin Ahmad, Kai-Wei Chang
Researchers from UCLA conducted a thorough empirical study on pre-trained language models (PLMs) for keyphrase generation, systematically comparing architectural designs, pre-training domains, and parameter allocations. Their findings provide practical recommendations, demonstrating PLMs' data efficiency and the critical role of domain-specific pre-training in achieving high performance.
Key insights
- In-domain pre-training is a fundamental factor for KPG performance, consistently outperforming general-domain PLMs and providing benefits that are not diminished by task-specific pre-training.
- Encoder-only PLMs, when adapted with sequence-to-sequence attention masks, can achieve strong KPG performance, sometimes rivaling or exceeding general-domain encoder-decoder models.
- Optimal encoder-decoder PLM architecture for KPG prioritizes model depth over width, with a preference for deeper encoders and shallower decoders for improved keyphrase quality and inference efficiency.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.