Reproduce “Unveiling Scoring Processes: Dissecting the Differences between LLMs and Human Graders in Automatic Scoring” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Xuansheng Wu, Padmaja Pravin Saraf, Gyeonggeon Lee, Ehsan Latif, Ninghao Liu, Xiaoming Zhai
The paper examines whether Mixtral can score middle-school science responses using criteria aligned with human rubrics. It prompts the model to generate item-level analytic rubrics and compares them with human-written criteria. With analytic-rubric examples plus a holistic rubric, rubric-alignment F1 was 0.752 and accuracy was 54.58%, versus 34.83% without analytic rubrics. These findings concern one model and twelve science items.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.