Reproduce “Misalignment of LLM-Generated Personas with Human Perceptions in Low-Resource Settings” on OpenResearch

About this paper

Tabia Tanzin Prama, Christopher M. Danforth, Peter Sheridan Dodds

Researchers from the University of Vermont empirically assessed the ability of state-of-the-art large language models to generate culturally accurate and empathetic personas in Bangladesh, a low-resource environment. They found LLM-generated personas exhibited significant misalignment with human perceptions, demonstrating systematic biases across political, gender, and religious dimensions, alongside an unnatural positive sentiment bias.

Key insights

• LLMs struggle with deep contextual, cultural, and emotional understanding, resulting in a significant accuracy and qualitative perception gap compared to humans in low-resource settings.
• Systematic biases exist within LLMs, leading to inconsistent and unrepresentative persona generation across political, gender, and religious minority dimensions.
• LLMs exhibit a "Pollyanna Principle" bias, generating overly positive sentiment that lacks the balanced emotional expression found in authentic human dialogue.

Summary by alphaXiv.