Reproduce “SMART: Simulated Students Aligned with Item Response Theory for Question Difficulty Prediction” on OpenResearch

About this paper

Alexander Scarlatos, Nigel Fernandez, Christopher Ormerod, Susan Lottridge, Andrew Lan

The SMART framework presents a method for predicting the difficulty of open-ended assessment items by simulating student responses aligned with Item Response Theory. It enhances the correlation between instructed student ability and generated response scores, yielding more accurate item difficulty estimations compared to direct prediction or unaligned simulation approaches.

Key insights

• Explicitly aligning simulated student response generation with Item Response Theory (IRT) likelihoods using Direct Preference Optimization (DPO) significantly enhances the accuracy of item difficulty prediction.
• Simulating student responses is a more effective strategy for predicting open-ended item difficulties than direct prediction methods that rely solely on item text.
• The DPO-aligned student simulator produces responses that more accurately reflect ability-specific linguistic and content patterns, leading to greater psychometric realism.

Summary by alphaXiv.