Reproduce “Mind the Gap: Pitfalls of LLM Alignment with Asian Public Opinion” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Hari Shankar, Vedanta S P, Sriharini Margapuri, Debjani Mazumder, Ponnurangam Kumaraguru, Abhijnan Chakraborty
An audit of large language models (LLMs) reveals consistent misrepresentation of religious public opinion in Asian countries, particularly for minority groups, even while showing good alignment on general social issues. Prompting in local languages can partially improve alignment, but fundamental biases persist.
Key insights
- Large language models consistently struggle to accurately represent public opinion on sensitive religious topics in Asian contexts, performing significantly worse than on broader social issues.
- While prompting LLMs in native languages can partially reduce distributional gaps, it often fails to correct fundamental underlying probability shifts in model responses.
- LLMs exhibit specific representational harms by amplifying negative stereotypes for certain minority religious identities, while prompt disambiguation substantially improves calibration on culturally specific questions.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.