Reproduce “SPECTER: Document-level Representation Learning using Citation-informed Transformers” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Arman Cohan, Sergey Feldman, Iz Beltagy, Doug Downey, Daniel S. Weld

SPECTER learns paper representations for scientific discovery using citation links as training signals and title and abstract text at inference. Starting from SciBERT, it trains on cited and uncited paper pairs, including hard negatives. Across seven SciDocs tasks, it averages 80.0 versus 76.9 for SGC; SGC performs better on direct-citation prediction and requires citation-graph access.

Key insights

  • Integrating citation graph information with Transformer models through a triplet margin loss effectively learns strong inter-document relationships.
  • The inclusion of 'hard negative' examples (papers cited by cited papers) is crucial for pushing the model to learn finer-grained distinctions between related and unrelated documents.
  • General-purpose document embeddings derived from SPECTER can outperform task-specific fine-tuning of language models, indicating their versatility and efficiency for downstream applications.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.