Reproduce “"I understand why I got this grade": Automatic Short Answer Grading with Feedback” on OpenResearch

About this paper

Dishank Aggarwal, Pritam Sil, Bhaskaran Raman, Pushpak Bhattacharyya

Researchers at IIT Bombay and Fujitsu Research of India developed the Engineering Short Answer Feedback (EngSAF) dataset, a multi-domain resource for automatic short answer grading with content-elaborated feedback. They introduced a Label-Aware Synthetic Feedback Generation (LASFG) strategy using LLMs and demonstrated a fine-tuned Mistral-7B model achieving 92.5% accuracy and 4.5/5 feedback correctness in a real-world examination deployment.

Key insights

• The EngSAF dataset, with its human-evaluated high quality (98% label correctness, 4.55/5 feedback correctness), provides a robust foundation for advancing ASAG systems beyond simple grading.
• Incorporating the original question (`with_question` setting) consistently and substantially improves the performance of fine-tuned LLMs for both grade prediction and feedback generation, leading to approximately 10-12% higher BERTScore for Llama-2 and Mistral-7B on unseen data.
• The deployed Mistral-7B model demonstrates practical viability and pedagogical value, with subject matter experts rating its output label accuracy at 92.5% and feedback correctness at 4.5/5 in a real-world examination setting.

Summary by alphaXiv.