Reproduce “Heterogeneous Value Alignment Evaluation for Large Language Models” on OpenResearch

About this paper

Zhaowei Zhang, Ceyao Zhang, Nian Liu, Siyuan Qi, Ziqi Rong, Song-Chun Zhu, Shuguang Cui, Yaodong Yang

This research introduces the Heterogeneous Value Alignment Evaluation (HVAE) system, an automated framework to assess how large language models (LLMs) align with diverse human values using the Social Value Orientation (SVO) framework. The evaluation of mainstream LLMs revealed a general propensity towards neutral values and varying performance based on model architecture and fine-tuning methods.

Key insights

• Current LLMs exhibit a systemic bias towards neutral values, demonstrating lower value rationality for individualistic and competitive orientations, likely due to safety fine-tuning.
• The ability of LLMs to benefit from self-constructed goal prompting for value alignment is correlated with their conversational coherence and reasoning capabilities.
• Fine-tuning methodologies, such as Constitutional AI versus Reinforcement Learning from Human Feedback (RLHF), influence value alignment adaptability, potentially involving trade-offs between harmlessness and flexibility.

Summary by alphaXiv.