Reproduce “A Confederacy of Models: a Comprehensive Evaluation of LLMs on Creative Writing” on OpenResearch

About this paper

Carlos Gómez-Rodríguez, Paul Williams

A comprehensive study evaluated various large language models (LLMs) on zero-shot creative writing using a unique, complex prompt designed to prevent training data reuse. The research found that top commercial LLMs, particularly GPT-4, matched or slightly surpassed trained human writers in many creative writing aspects, while humans maintained a lead in originality.

Key insights

• The ability to integrate dark humor appears to be an emergent property of larger, more sophisticated LLMs, demonstrating a
• binary divide
• where only top commercial models and humans show competence.
• Evaluation outcomes in subjective domains like creative writing are profoundly sensitive to methodological choices, such as the human baseline and prompt design, explaining differences with concurrent studies.
• While top LLMs achieved high scores, human writers retained a crucial edge in originality/creativity, suggesting that unique human innovation remains a differentiator.

Summary by alphaXiv.