Reproduce “X-TURING: Towards an Enhanced and Efficient Turing Test for Long-Term Dialogue Agents” on OpenResearch

About this paper

Weiqi Wu, Hongqiu Wu, Hai Zhao

Researchers at Shanghai Jiao Tong University developed X-TURING, an enhanced framework for the Turing Test, to efficiently evaluate Large Language Models' human-likeness in prolonged conversational settings. This framework revealed that while current LLMs like GPT-4 can achieve moderate human pass rates in short interactions (up to 51.9% for 3 turns), their ability to maintain consistency and persona significantly decreases over extended dialogues, with pass rates dropping as low as 13.3% for 110-turn interactions.

Key insights

• Current Large Language Models (LLMs) struggle with long-term conversational coherence, consistency, and persona maintenance over extended interactions.
• Human judges are more adept at distinguishing LLMs from humans than LLM judges are, indicating a gap in AI's understanding of subtle human social and conversational nuances.
• Providing more dialogue history generally improves an LLM's ability to simulate human conversational styles, though this effect exhibits diminishing returns, and the quality of the history is critical.

Summary by alphaXiv.