Reproduce “unarXive 2022: All arXiv Publications Pre-Processed for NLP, Including Structured Full-Text and Citation Network” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Tarek Saier, Johan Krause, Michael Färber
Researchers from the Karlsruhe Institute of Technology developed unarXive 2022, a large-scale scholarly dataset of nearly 1.9 million arXiv publications pre-processed for NLP, offering structured full-text, a highly complete citation network (44.4% of references linked), and specialized training data for citation recommendation and IMRaD classification tasks.
Key insights
- Leveraging LaTeX sources directly enables the creation of a much cleaner and richly structured dataset with precise content linking compared to PDF-derived datasets.
- Integrating a comprehensive, actively maintained target corpus like OpenAlex for reference matching is crucial for building a highly complete and up-to-date citation network.
- Providing pre-processed, task-specific training and test data (e.g., for citation recommendation and IMRaD classification) on platforms like Hugging Face Hub drastically lowers the barrier for NLP researchers to utilize large scholarly datasets.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.