Reproduce “Characterising the Creative Process in Humans and Large Language Models” on OpenResearch

About this paper

Surabhi S. Nath, Peter Dayan, Claire Stevenson

The authors develop a measure of semantic jumps in human and LLM response sequences, using sentence embeddings to estimate when successive ideas shift categories. On brick and paperclip alternative-use tasks, humans and LLMs had similar jump counts, but flexibility was associated with higher originality only in LLM responses. These findings concern the tested tasks and originality scores, not internal thought processes.

Key insights

• Large Language Models (LLMs) can match human distributions of creative process flexibility in divergent thinking, but their originality directly correlates with flexibility, a relationship not consistently observed in humans where both persistent and flexible approaches can lead to creative outcomes.
• LLMs exhibit significantly more persistent semantic exploration compared to humans in verbal fluency tasks, suggesting fundamental differences in how they navigate well-defined semantic spaces.
• Simple parameter adjustments like 'temperature' do not reliably shift an LLM's fundamental exploration bias in divergent thinking tasks, highlighting limitations in controlling their creative process.

Summary by alphaXiv.