Reproduce “LinkBERT: Pretraining Language Models with Document Links” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Michihiro Yasunaga, Jure Leskovec, Percy Liang
LinkBERT introduces a pretraining method for language models that integrates knowledge spanning across documents by explicitly leveraging document links like hyperlinks and citations. This approach substantially improves performance on multi-hop question answering and domain-specific tasks, achieving new state-of-the-art results on benchmarks such as BLURB and MedQA-USMLE.
Key insights
- Explicitly incorporating high-quality document links during pretraining significantly enhances an LM's ability to internalize salient, multi-document knowledge beyond mere lexical similarity.
- The Document Relation Prediction (DRP) objective is crucial for teaching language models to understand and utilize the semantic relationships between documents effectively.
- Fusing graph-based self-supervised learning with traditional language modeling allows smaller, specialized models to outperform much larger general-purpose models on domain-specific reasoning tasks.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.