Reproduce “Evaluating LLM-Generated Q&A Test: a Student-Centered Study” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Anna Wróblewska, Bartosz Grabek, Jakub Świstak, Daniel Dan
Researchers from Warsaw University of Technology and MODUL University Vienna developed a student-centered pipeline to generate university-level Q&A tests using GPT-4o, combining automated question generation with psychometric analysis and student feedback. Their study demonstrated that these AI-generated questions exhibit psychometric quality and perceived quality comparable to human-authored tests.
Key insights
- LLMs like GPT-4o can generate multiple-choice questions for higher education that are psychometrically comparable to human-authored items in terms of discrimination and difficulty.
- Integrating a student-centered evaluation approach provides valuable insights into perceived question quality and fosters critical thinking about AI-generated content among students.
- Robust psychometric analysis, including Differential Item Functioning (DIF), is crucial for identifying and addressing potential biases or suboptimal performance in LLM-generated test items.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.