Reproduce “When to Make Exceptions: Exploring Language Models as Accounts of Human Moral Judgment” on OpenResearch

About this paper

Zhijing Jin, Sydney Levine, Fernando Gonzalez, Ojasv Kamal, Maarten Sap, Mrinmaya Sachan, Rada Mihalcea, Josh Tenenbaum and 1 more

A study introduces a novel challenge set and a cognitively-inspired prompting strategy to improve large language models' understanding of when humans make exceptions to moral rules. This approach, which models human contractualist decision-making, enabled InstructGPT to achieve a 64.47% F1 score on a new benchmark, demonstrating a 10.53% improvement over the base model in predicting human moral flexibility.

Key insights

• Explicitly modeling multi-step human-like reasoning processes can significantly enhance LLMs' ability to predict human moral flexibility, outperforming models that rely solely on pattern recognition from training data.
• LLMs, even advanced ones like InstructGPT, demonstrate specific weaknesses in moral reasoning, particularly in understanding the underlying 'purpose of the rule' and accurately quantifying nuanced 'utility' and 'losses to others'.
• Many LLMs exhibit 'dogmatic reasoning,' adhering rigidly to literal rule formulations without adequately considering specific contextual conditions that might warrant an exception.

Summary by alphaXiv.