Reproduce “CLIN: an Objective Framework for Evaluating Creativity in Short Persian Literary Text” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Mohammad Reza Modarres, Armin Tourajmehr, Yadollah Yaghoobzadeh, Mohammad Taher Pilehvar
Modarres et al. introduce CLIN, an objective framework for evaluating creativity in short Persian literary texts across structured dimensions like originality, fluency, and elaboration. The framework's proxy metrics achieved human alignment comparable to or better than leading LLMs while demonstrating that LLM-based creativity judgments are inconsistent for subjective aspects and not reliably improved by complex prompting.
Key insights
- LLM-human agreement for creativity evaluations is dimension-dependent, stronger for structured dimensions (originality, fluency, elaboration) and weaker for subjective ones (emotion, attractiveness).
- Elaborate LLM judging protocols (few-shot, ensemble, multi-agent debate) do not consistently improve agreement with human judgments for creativity.
- Simple, objective proxy metrics can effectively approximate structured creativity dimensions with performance comparable to or better than LLMs at lower cost.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.