Reproduce “Self-CTRL: Self-Consistency Training with Reinforcement Learning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Itamar Pres, Laura Ruis, Melat Ghebreselassie, Belinda Z. Li, Jacob Andreas
Self-CTRL is a reinforcement learning framework designed by MIT CSAIL that optimizes language models for consistency between their meta-level self-explanations and object-level behaviors. The method improved the faithfulness of explanations in probabilistic reasoning tasks, achieving an R^2 of 0.65, and enhanced safety in constitutional AI settings, reducing the HarmBench attack success rate to 0.5%.
Key insights
- Language models can recover and articulate latent probabilistic structures from their own sampled behaviors through self-supervised consistency training, even without explicit ground-truth explanations.
- Explicitly optimizing for consistency significantly improves both the simulatability (predictability from explanations) and the safety (adherence to principles) of language models in diverse domains.
- The self-supervised nature of Self-CTRL offers a scalable approach for AI alignment by leveraging the model itself to generate the necessary explanation-behavior evidence, reducing reliance on extensive human labeling.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.