Reproduce “A Method for Evaluating Hyperparameter Sensitivity in Reinforcement Learning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jacob Adkins, Michael Bowling, Adam White
An empirical methodology featuring metrics for hyperparameter sensitivity (Φ) and effective dimensionality (d(ω)), along with a performance-sensitivity plane, quantifies the tuning demands of reinforcement learning algorithms. Applying this to PPO variants, the study found that certain performance-enhancing normalization techniques also increased the reliance on environment-specific hyperparameter optimization.
Key insights
- No tested normalization variant simultaneously improved performance and reduced hyperparameter sensitivity, challenging a common assumption that such techniques universally simplify tuning.
- While some advantage normalization methods increased performance, this gain was consistently accompanied by an increased reliance on per-environment hyperparameter tuning.
- The relative performance ranking of algorithms can vary significantly based on how many hyperparameters are specifically tuned per environment.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.