Reproduce “AI Alignment at Your Discretion” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Maarten Buyl, Hadi Khalaf, Claudio Mayrink Verdun, Lucas Monteiro Paes, Caio C. Vieira Machado, Flavio du Pin Calmon
Researchers from Ghent and Harvard Universities introduce a framework for "alignment discretion," a concept adapted from legal philosophy, to analyze how human and algorithmic annotators interpret and apply principles in AI alignment. Their empirical analysis, using novel metrics on safety datasets, reveals that human annotators exhibit substantial arbitrariness, and current large language models often fail to replicate human principle prioritization.
Key insights
- Only a minority of cases (15-20%) in safety alignment datasets involve genuine principle conflict, where annotators must exercise discretion; the majority show clear consensus or indifference.
- Human annotators frequently deviate from principle consensus, exhibiting substantial "arbitrariness" (up to 28.9%), challenging the notion of purely principled feedback.
- Current RLHF methods struggle to transfer human discretion, with LLM policies showing significant "discretion discrepancy" from human principle prioritization, despite reward models often mirroring it well.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.