Reproduce “Rethinking Scaffolding in LLM Tutors: The Interactional Mismatch Between Benchmarks and Real-World Deployments” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Alexandra Neagu, Jeffrey T. H. Wong, Marcus Messer, Rhodri Nelson, Peter B. Johnson
Researchers at Imperial College London developed a two-metric evaluation pipeline to assess both chatbot scaffolding quality and student uptake in LLM tutors. Their analysis revealed a mismatch between the consistent student engagement with scaffolding observed in AI tutor benchmarks and the diverse, often scaffolding-bypassing, student behaviors in real-world educational settings.
Key insights
- Student interaction patterns with LLM tutors in real-world settings differ substantially from those implicitly assumed by current AI tutor benchmarks.
- Students frequently bypass chatbot scaffolding, often treating LLMs as tools for efficient problem resolution rather than traditional conversational partners, due to low perceived social accountability.
- The relationship between a chatbot's scaffolding efforts and student uptake is weak and inconsistent, suggesting other student-driven situational factors strongly influence engagement.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.