Reproduce “Inverse Constitutional AI: Compressing Preferences into Principles” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Arduin Findeis, Timo Kaufmann, Eyke Hüllermeier, Samuel Albanie, Robert Mullins
Inverse Constitutional AI (ICAI) extracts natural language principles from pairwise text preference data, compressing annotator choices into an interpretable constitution. This framework enables the identification of underlying reasons for preferences and the detection of biases, showing improved agreement with original annotations, especially for preferences unaligned with the base model's intrinsic biases.
Key insights
- LLMs, guided by ICAI-derived principles, can effectively reconstruct preference datasets, demonstrating the ability to follow instructions even when they contradict their inherent biases (e.g., GPT-4o on unaligned data).
- ICAI can successfully capture distinct individual user preferences and interpret the preferences of demographic subgroups, providing qualitative insights into their decision-making processes.
- Systematic biases, such as preferences for verbosity, specific stylistic formats (e.g., numbered lists), or assertiveness, are prevalent and detectable across diverse human feedback datasets using this framework.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.