Reproduce “PALMs: Using Multi Construct-Grounded Rationales for Modeling Population Preferences in LLMs” on OpenResearch

About this paper

Priyanka Dey, Brihi Joshi, Preyashi Poddar, Jieyu Zhao, Emilio Ferrara

Researchers at the University of Southern California developed Population Aligned Language Models (PALMs) which leverage multi-construct-grounded rationales as latent supervision to train large language models to better align with diverse human population preferences. This approach led to an average relative improvement of 8.59% in population alignment over baselines and robust generalization to personalized reward modeling, population simulation, and social intelligence tasks.

Key insights

• Integrating established psychological and cultural constructs directly into preference learning acts as a robust inductive bias, enabling LLMs to internalize and reason about the underlying mechanisms of human judgment.
• Leveraging construct-grounded rationales as latent supervision, with masked loss, effectively shapes the model's internal representations of preferences without explicit prediction targets or overfitting.
• Surface-level optimization on behavioral data alone may reduce representational diversity, underscoring the benefits of structured social science knowledge for rich population alignment.

Summary by alphaXiv.