Reproduce “Preference Learning for AI Alignment: a Causal Perspective” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Katarzyna Kobalczyk, Mihaela van der Schaar

Researchers from the University of Cambridge apply causal inference principles to address fundamental limitations in current reinforcement learning from human feedback (RLHF) approaches, demonstrating through empirical experiments on UltraFeedback and HH-RLHF datasets that standard reward models fail to generalize when trained under confounding conditions or limited latent overlap, with causally-inspired multihead and adversarial architectures achieving substantially better performance on out-of-distribution "inconsistent" examples (where user objectives differ from prompt types) by learning representations that isolate causal text features from spurious correlations, while revealing that even moderate correlations between latent factors (ρ=0.6) in training data lead to significant performance drops from 67.8% to 59.7% accuracy when these correlations break at test time, establishing the need for interventional data collection and causal assumptions like unconfoundedness and latent positivity to build robust reward models that can answer counterfactual questions about human preferences.

Key insights

  • Confounding between user objectives and prompt characteristics creates systematic bias that prevents generalization to interventional settings
  • Limited latent positivity (strong correlations between causal factors in training data) leads to poor out-of-distribution performance even when ground-truth factors are identifiable
  • Multihead architectures that condition on user context while learning shared representations significantly outperform standard approaches on inconsistent examples
  • Adversarial training can further improve robustness by encouraging representations to discard spurious correlations with confounding variables

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.