Reproduce “How Data Scientists Review the Scholarly Literature” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Sheshera Mysore, Mahmood Jasim, Haoru Song, Sarah Akbar, Andre Kenneth Chase Randall, Narges Mahyar

Based on interviews and think-aloud sessions with 20 data scientists, the study presents literature review as an iterative process that combines search, credibility assessment, technical interpretation, cross-paper comparison, and social sensemaking. Participants’ main difficulties included translating ideas into disciplinary terminology, distinguishing substantive advances from incremental work, recovering omitted implementation details, and understanding unfamiliar mathematics, pointing to opportunities for more context-aware and collaborative scholarly tools.

Key insights

  • Literature review is not simply document retrieval: researchers repeatedly alternate between searching, learning terminology, evaluating papers, interpreting methods, and revising their understanding of the field.
  • Participants struggled to identify the precise contribution or “delta” of papers, especially when many studies addressed similar problems, and often needed surveys, citation networks, code, talks, blogs, or peer explanations to resolve uncertainty.
  • Social connections—including collaborators, peers, online communities, and authors—function as an important part of scholarly infrastructure by supplying recommendations, credibility judgments, technical explanations, and shared interpretations.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.