Reproduce “Catch Me If You Can? Not Yet: LLMs Still Struggle to Imitate the Implicit Writing Styles of Everyday Authors” on OpenResearch

About this paper

Zhengxiang Wang, Nafis Irtiza Tripto, Solha Park, Zhenzhen Li, Jiawei Zhou

Large Language Models (LLMs) consistently struggle to imitate the implicit writing styles of everyday authors, particularly in informal contexts, even with few-shot in-context learning. A multi-angle evaluation framework revealed that while few-shot prompting helps, LLM-generated texts are still overwhelmingly detected as AI-written.

Key insights

• LLMs exhibit a 'personalization gap,' consistently struggling to replicate the implicit, nuanced styles of everyday authors, especially in informal domains like blogs and online forums.
• Few-shot in-context learning generally improves stylistic alignment compared to zero-shot, but its benefits for implicit style imitation diminish quickly beyond a few examples.
• Despite varying success in statistical style alignment, LLM-generated texts are almost universally flagged as AI-written by detectors, indicating a failure to achieve human-like authenticity.

Summary by alphaXiv.