Reproduce “Large Language Models Penetration in Scholarly Writing and Peer Review” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Li Zhou, Ruijie Zhang, Xunlian Dai, Daniel Hershcovich, Haizhou Li

Researchers developed LLMetrica, a framework incorporating rule-based metrics and machine learning models, along with the ScholarLens dataset, to systematically detect and quantify the increasing use of large language models in scholarly writing (abstracts) and peer review (meta-reviews, reviews). The study found a clear increase in LLM penetration since 2023, observing distinct linguistic and semantic characteristics in LLM-generated content, such as higher word length and lower readability, and lower sentence specificity in reviews.

Key insights

  • Large Language Models leave distinct linguistic and semantic fingerprints on scholarly content, characterized by specific word usage patterns (e.g., higher average word length, lower readability) and semantic homogeneity.
  • Effective detection of LLM-generated content in academic contexts necessitates domain-specific models, which significantly outperform general-purpose detectors.
  • LLM penetration exhibits varying trends across different academic content types (e.g., abstracts vs. reviews) and by the nature of LLM assistance (e.g., refining vs. synthesizing content).

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.