Reproduce “SEFORA: Student Essays with Feedback Corpus and LLM Feedback Evaluation Framework” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Shayan Peyghambari Oskoui, Norah Almousa, Zhaoyi Joey Hou, Carolina Gustafson, Gayle Rogers, Raquel Coelho, Diane Litman, Xiang Lorraine Li

Researchers from the University of Pittsburgh introduce SEFORA, a public corpus of student essays with authentic, fine-grained instructor feedback, and UNIMATCH, a novel evaluation framework. Their study finds that current Large Language Models struggle to generate feedback aligned with human instructors, with increased verbosity significantly reducing feedback quality.

Key insights

  • Current Large Language Models consistently underperform in generating feedback that aligns with human instructor priorities, with no tested configuration achieving an F1 score above 0.4.
  • The most significant factor affecting feedback quality is verbosity; generating more feedback units strongly correlates with lower precision and overall F1 scores.
  • Providing explicit pedagogical feedback categories in the prompt (e.g., Task Constraints, Elaboration) helps LLMs produce more relevant feedback, improving F1 scores.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.