Reproduce “Rethinking LLM-Judged Helpfulness as a Pedagogy Signal: A Pre-Registered Audit Across Tutor Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Shuyi Fan, Boyuan Deng, Mengyu Xu, Jiale Liu, Hongyang Zhang, Qiaoxin Yang, Chongyang Gao
A pre-registered audit tests whether general LLM-judged helpfulness can serve as a proxy for effective tutoring. Across Claude Sonnet 4.6, GPT-5.5, and Gemini 3.1 Pro Preview tutor bases, answer-revealing turns received higher immediate helpfulness ratings but were followed by substantially less independent student reasoning, while helpfulness rankings varied across judges and policies.
Key insights
- General helpfulness did not reliably separate the two tutoring policies at the session level, despite a large difference on the pedagogy-specific rubric.
- At the turn level, answer-revealing responses were rated as more helpful but were followed by independent reasoning on only 33% of turns, versus 73% after non-leaky turns.
- Helpfulness orderings changed across judges and tutor bases, indicating that judge choice and score compression can materially affect evaluation conclusions.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.