Reproduce “NLP Reproducibility For All: Understanding Experiences of Beginners” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Shane Storks, Keunwoo Peter Yu, Ziqiao Ma, Joyce Chai

An investigation into NLP reproducibility for beginners, conducted by the University of Michigan, reveals that author efforts in documentation and code quality are more influential for successful reproduction (with students achieving an average relative error within 3%) than individual programming skill or paper comprehension. This highlights practical barriers faced by new entrants and offers guidance for enhancing research accessibility.

Key insights

  • Individual programming skill level and theoretical comprehension of a research paper have a limited influence on the time spent or perceived difficulty for beginners in reproducing NLP results.
  • Explicit and thorough reproducibility efforts by paper authors, such as clear code usage documentation and precise dependency specification, are more critical determinants of a beginner's success and experience.
  • Students frequently suggested improvements to reproducibility checklists, emphasizing the need for comprehensive dependency lists (including version numbers) and detailed code usage examples.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.