Reproduce “ParaRev: Building a dataset for Scientific Paragraph Revision annotated with revision instruction” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Léane Jourdan, Nicolas Hernandez, Richard Dufour, Florian Boudin, Akiko Aizawa
The research introduces ParaRev, the first dataset for scientific paragraph revision annotated with human-written, personalized instructions, shifting the focus from sentence-level to paragraph-level text refinement. Experiments demonstrate that providing detailed instructions to large language models like Llama3, Mistral, and GPT-4o leads to improved revision quality, with average gains of +4.07 ROUGE-L, +3.66 SARI, and +0.75 Bertscore compared to general label-based prompts.
Key insights
- Providing detailed, personalized instructions to language models significantly improves the quality and relevance of automated paragraph-level revisions.
- Current automatic evaluation metrics (ROUGE-L, SARI, Bertscore) may not fully capture the quality of complex, substantial text revisions, potentially penalizing models that make more creative or impactful changes.
- Shifting the focus from general labels to explicit, human-like revision instructions is essential for developing more effective and nuanced automated writing assistance tools.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.