Reproduce “Measuring "Success" In the Transition from Pre- to Post-LLM Conversational Agents through the Lens of Interaction & Explanations for Older Adults” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Tamara Zubatiy, Niharika Mathur, Reina Chan, Srijan Pokharel, Rosemarie Santa González, Agata Rozga, Xiang Zhi Tan, Elizabeth D. Mynatt
The authors examine how to assess conversational agents for older adults when interaction counts may not reflect success. They draw on more than eight years of research with over 100 participants, including pre-LLM deployments and an early-stage LLM system. In one medication-reminder case, about 38% of user interactions reflected conversational misalignments, not successful exchanges; this finding applies only to that pre-LLM deployment.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.