Reproduce “How to Align Large Language Models for Teaching English? Designing and Developing LLM based-Chatbot for Teaching English Conversation in EFL, Findings and Limitations” on OpenResearch

About this paper

Jaekwon Park, Jiyoung Bae, Unggi Lee, Taekyung Ahn, Sookbun Lee, Dohee Kim, Aram Choi, Yeil Jeong and 2 more

Researchers from Enuma, Inc., Seoul National University, The University of Alabama, and Korea University designed, developed, and evaluated an LLM-based chatbot for English as a Foreign Language (EFL) conversation teaching using a six-phase iterative Design and Development Research methodology. The study identified that a smaller, open-source LLM, "NeuralHermes-2.5-Mistral-7B-AWQ," effectively optimized with a specific prompt template, could achieve high performance for EFL tasks, while also developing a comprehensive evaluation framework and gathering critical teacher feedback on usability and pedagogical needs.

Key insights

• Smaller, open-source LLMs can be effectively aligned for specific educational tasks through sophisticated prompt engineering, offering a viable, cost-effective, and privacy-conscious alternative to larger proprietary models.
• Effective integration of LLMs in education necessitates a hybrid evaluation approach, combining automated quantitative metrics with essential qualitative feedback from human experts to ensure pedagogical validity and user acceptance.
• Pedagogical alignment for conversational AI in EFL requires careful management of conversational flow, adherence to grade-appropriate language, topic guardrails, and the provision of natural, contextually relevant feedback mechanisms.

Summary by alphaXiv.