Reproduce “ArxivDIGESTables: Synthesizing Scientific Literature into Tables using Language Models” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Benjamin Newman, Yoonjoo Lee, Aakanksha Naik, Pao Siangliulue, Raymond Fok, Juho Kim, Daniel S. Weld, Joseph Chee Chang and 1 more

A framework for automatically generating scientific literature review tables using language models is introduced, alongside ARXIVDIGESTABLES, a new large-scale dataset of 2,228 human-authored tables, and DECONTEXTEVAL, a robust metric for schema evaluation. The study demonstrates that LMs can generate useful comparative aspects, with novel generated aspects often matching or exceeding gold aspects in usefulness and specificity, and reveals the critical role of contextual information in improving schema reconstruction.

Key insights

  • Providing additional contextual information (e.g., gold captions, in-text references, few-shot examples) significantly improves language models' ability to reconstruct human-authored table schemas.
  • LM-generated comparative aspects that do not match reference tables are frequently useful and specific, sometimes even more so than human-authored ones, suggesting LMs can aid in discovering new comparative dimensions.
  • Automated value evaluation metrics show limitations, often underestimating improvements in value generation observed by human evaluators, indicating a need for more sensitive semantic evaluation methods.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.