Reproduce “ACE : Off-Policy Actor-Critic with Causality-Aware Entropy Regularization” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Tianying Ji, Yongyuan Liang, Yan Zeng, Yu Luo, Guowei Xu, Jiawei Guo, Ruijie Zheng, Furong Huang and 2 more
An off-policy actor-critic algorithm, ACE, enhances sample efficiency and robustness in continuous control by incorporating causality-aware entropy regularization and a gradient-dormancy-guided reset mechanism. It consistently outperformed leading baselines across 29 diverse tasks, achieving over 70% improvement in some challenging tabletop manipulation scenarios and demonstrating significant gains in high-dimensional dexterous hand control.
Key insights
- The importance or "significance" of different primitive behaviors (individual action dimensions) varies over time during policy learning, and accounting for this can lead to more efficient exploration.
- Causal relationships between action dimensions and rewards can be learned from observational data and leveraged to guide exploration, prioritizing actions with higher causal influence.
- The "gradient dormancy phenomenon" in neural networks, where neurons cease active learning, can hinder performance and lead to local optima; dynamic perturbation based on dormancy can mitigate this.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.