Reproduce “Measuring Human and AI Values Based on Generative Psychometrics with Large Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Haoran Ye, Yuhang Xie, Yuanyi Ren, Hanjun Fang, Xin Zhang, Guojie Song
This research introduces Generative Psychometrics for Values (GPV), an LLM-based framework for measuring values by dynamically analyzing text-revealed selective perceptions. It demonstrates improved construct validity and stability for both human and AI value assessment, effectively mitigating response biases in large language models and showing predictive power for LLM safety.
Key insights
- Value measurement can be made more robust and nuanced by dynamically extracting 'selective perceptions' from free-form text, overcoming the limitations of static questionnaires and lexicons.
- Prior methods for measuring LLM values are highly susceptible to response biases, leading to unreliable results, which GPV successfully mitigates by analyzing generated content rather than direct self-reports.
- LLM value measurements, particularly those derived from Hofstede's VSM framework, can serve as interpretable and predictive features for LLM safety scores, opening new avenues for AI alignment strategies.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.