Reproduce “No Representation, No Trust: Connecting Representation, Collapse, and Trust Issues in PPO” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Skander Moalla, Andrea Miele, Daniil Pyatko, Razvan Pascanu, Caglar Gulcehre
Researchers from EPFL and Google DeepMind empirically demonstrate that Proximal Policy Optimization (PPO) agents suffer from feature rank deterioration and capacity loss, which directly undermines the algorithm's trust region mechanism and leads to performance collapse. They introduce Proximal Feature Optimization (PFO), an auxiliary loss that regularizes feature dynamics and effectively mitigates these issues across Atari and MuJoCo environments.
Key insights
- PPO agents are susceptible to feature rank deterioration and capacity loss, similar to value-based methods, with increased non-stationarity accelerating this degradation.
- Representation collapse directly causes the breakdown of PPO's trust region, as probability ratios frequently diverge beyond clipping limits due to coupled gradients from highly colinear representations.
- The observed performance collapse in PPO is characterized by high but trivial policy entropy, resulting from neuron inactivation in the feature layer and the network's inability to distinguish between states.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.