Reproduce “Offline Safe Policy Optimization From Heterogeneous Feedback” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Ze Gong, Pradeep Varakantham, Akshat Kumar
The paper introduces Offline Safe Policy Optimization from Heterogeneous Feedback (Offline Safe POHF), a framework with an algorithm called PreSa, for learning safe and reward-maximizing policies in continuous control tasks from offline human feedback. PreSa consistently outperforms existing baselines in achieving a balance between reward and safety, often matching or exceeding the safety performance of methods with ground truth rewards and costs.
Key insights
- Directly optimizing a policy from heterogeneous human feedback (reward preferences and binary safety labels) in an offline setting is effective for simultaneously achieving high rewards and safety in continuous control tasks, bypassing the limitations of two-phase model-based approaches.
- Explicitly separating feedback signals for reward preferences and safety labels is more effective for learning safe policies than relying on a single, aggregated 'general preference' signal that implicitly combines both.
- The proposed utility function for safety alignment, combined with a feasibility generalization bound, provides a robust mechanism to incorporate safety constraints and ensure empirical safety performance reliably approximates true expected safety.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.