Reproduce “SPECTER: Document-level Representation Learning using Citation-informed Transformers” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Arman Cohan, Sergey Feldman, Iz Beltagy, Doug Downey, Daniel S. Weld
SPECTER learns paper representations for scientific discovery using citation links as training signals and title and abstract text at inference. Starting from SciBERT, it trains on cited and uncited paper pairs, including hard negatives. Across seven SciDocs tasks, it averages 80.0 versus 76.9 for SGC; SGC performs better on direct-citation prediction and requires citation-graph access.
Key insights
- Integrating citation graph information with Transformer models through a triplet margin loss effectively learns strong inter-document relationships.
- The inclusion of 'hard negative' examples (papers cited by cited papers) is crucial for pushing the model to learn finer-grained distinctions between related and unrelated documents.
- General-purpose document embeddings derived from SPECTER can outperform task-specific fine-tuning of language models, indicating their versatility and efficiency for downstream applications.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.