Reproduce “Misalignment Between Educators’ Testing and End Users’ Behavior: What 17,000+ Chatbot Messages Reveal About AI Literacy Training for Teachers” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Ruiwei Xiao, Jiayi Wang, Miina Koyama, Yue Wang, Qing Xiao, Xinying Hou, Norman Bier, John Stamper

The authors examine how higher-education educators build and test chatbots, comparing builder conversations with later peer and end-user use. In 36 deployed bots, users produced 23 bot–intent pairs involving adversarial behavior, but builders had tested only 6 (26%). Answer dumping often occurred despite prompts prohibiting direct answers. The observational study does not show that testing reduces failures or training caused them.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.