Reproduce “Accurate in space, unreliable in time: how LLMs represent national cultural change” on OpenResearch

About this paper

Yalda Daryani, Miranda Bogen, Madeleine I. G. Daepp

Large language models (LLMs) accurately represent national cultural values at a given point in time but struggle to capture how these values evolve over extended periods. The study demonstrates that LLMs understate the magnitude of observed cultural shifts, frequently lag behind current cultural positions, and generally fail to reproduce reversals in cultural trajectories.

Key insights

• LLMs are "accurate in space" (good at placing countries culturally at a static moment) but "unreliable in time" (poor at representing how cultures change historically).
• This temporal flattening in LLM representations risks fostering psychological essentialism and perpetuating outdated views of dynamic cultural identities, contributing to representational harms.
• Current methods for evaluating LLM cultural awareness are incomplete and must be expanded to include "change alignment" to provide a more robust assessment of cultural competence.

Summary by alphaXiv.