Reproduce “DREsS: Dataset for Rubric-based Essay Scoring on EFL Writing” on OpenResearch

About this paper

Haneul Yoo, Jieun Han, So-Yeon Ahn, Alice Oh

KAIST researchers created DREsS, a large-scale dataset for automated essay scoring in English as a Foreign Language (EFL) writing, featuring expert-annotated rubric-based scores. This resource combines newly collected classroom essays, standardized existing datasets, and synthetic data generated via a novel corruption-based augmentation strategy, leading to a 45.44% improvement in QWK score for baseline models compared to using only new data.

Key insights

• Integrating newly collected, expert-annotated EFL classroom data with standardized existing datasets and targeted augmentation significantly enhances the performance of rubric-based AES models.
• The proposed Corruption-based Augmentation Strategy for Essays (CASE) is more effective at generating realistic, low-scoring synthetic EFL essays than large language models like gpt-4o, which tend to produce overly grammatically correct texts.
• AES models require specific data that reflects the unique linguistic characteristics and error patterns of EFL learners to provide accurate analytical feedback, as general-purpose LLMs without fine-tuning often fail to capture these nuances.

Summary by alphaXiv.