Reproduce “Re3: A Holistic Framework and Dataset for Modeling Collaborative Document Revision” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Qian Ruan, Ilia Kuznetsov, Iryna Gurevych
Researchers at the UKP Lab, Technical University of Darmstadt, introduce Re3, a holistic framework and the Re3-Sci dataset for modeling the review-revision-response cycle in scholarly documents. This comprehensive dataset features over 11,000 human-annotated sentence-level edits with qualitative labels and explicit linkages, enabling empirical insights into human revision behavior and the evaluation of LLM performance on tasks like edit intent classification (0.70 accuracy) and revision alignment (0.97 accuracy).
Key insights
- Empirical analysis showed that 'Modify' (54.54%) and 'Add' (28.93%) are the most common edit actions, with 'Fact/Evidence' (45.02%) and 'Clarity' (21.78%) being the predominant edit intents, often with specific action-intent relationships.
- Edits are not uniformly distributed but are concentrated in the initial and final sections of papers, exhibiting significant clustering tendencies at both paragraph and section levels.
- Explicit edit suggestions from reviewers are substantially more likely to be acted upon (over 50% implemented) compared to implicit suggestions or general weakness comments.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.