Reproduce “Beyond Agreement: Diagnosing the Rationale Alignment of Automated Essay Scoring Methods based on Linguistically-informed Counterfactuals” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yupei Wang, Renfen Hu, Zhe Zhao
The authors test whether automated essay scoring responds to rubric-relevant changes rather than matching human scores. They create counterfactuals that correct errors, alter language complexity, or disrupt organization, then compare responses. On ELLIPSE, fine-tuned BERT’s QWK was 0.680 versus 0.261 for zero-shot GPT-4. BERT-like models responded less to organization while LLMs responded more broadly; these measures do not reveal reasoning.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.