Reproduce “LLMs can Perform Multi-Dimensional Analytic Writing Assessments: A Case Study of L2 Graduate-Level Academic English Writing” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zhengxiang Wang, Veronika Makarova, Zhi Li, Jordan Kodner, Owen Rambow
This study demonstrates that Large Language Models can perform multi-dimensional analytic writing assessments for L2 graduate-level academic English, producing scores often within one point of human ratings and generating comprehensive, specific feedback comments. The research introduces a new corpus of L2 literature reviews and ProEval, an LLM-based framework for evaluating feedback comment quality.
Key insights
- LLMs can generate multi-dimensional analytic scores that approximate human expert ratings and produce feedback comments exhibiting similar characteristics to human-provided feedback, particularly when assessment criteria are presented independently.
- The ProEval framework offers a reliable and scalable method for automatically evaluating feedback comment quality, extracting fine-grained problem characteristics that correlate well with human perceptions of specificity and helpfulness.
- LLMs consistently provide extensive and specific feedback comments, even for subjective assessment criteria, and demonstrate a negative correlation between assigned scores and feedback quantity, mirroring human assessor behavior.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.