Reproduce “Measuring Psychological Depth in Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Fabrice Harel-Canada, Hanyu Zhou, Sreya Muppalla, Zeynep Yildiz, Miryung Kim, Amit Sahai, Nanyun Peng
Researchers at UCLA developed and validated the Psychological Depth Scale (PDS) for evaluating the subjective psychological impact of narratives, demonstrating its reliability with human evaluators. Their work found that state-of-the-art language models like GPT-4 can produce stories with psychological depth that are statistically indistinguishable from, or superior to, highly-rated human-written content.
Key insights
- Psychological depth in narratives can be reliably and consistently assessed by human evaluators using a theoretically grounded framework like the PDS.
- Large Language Models can effectively act as automated judges for psychological depth, particularly when employing sophisticated prompting strategies such as Mixture-of-Personas.
- State-of-the-art LLMs, specifically GPT-4, can generate narratives that achieve psychological depth on par with, or even surpass, human-authored stories from certain contexts.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.