Reproduce “A Robust Quantile Huber Loss With Interpretable Parameter Adjustment In Distributional Reinforcement Learning” on OpenResearch

About this paper

Parvin Malekzadeh, Konstantinos N. Plataniotis, Zissis Poulos, Zeyu Wang

Researchers at the University of Toronto propose a generalized quantile Huber loss function for distributional reinforcement learning by providing a probabilistic interpretation of the classical loss via Wasserstein distance between Gaussian-noised quantiles. This new loss incorporates an interpretable parameter 'b' that adaptively tunes the loss based on the uncertainty in predicted and target quantiles, effectively eliminating the need for heuristic parameter selection. The method achieves superior performance on Atari games and demonstrates faster, more stable convergence in financial hedging tasks.

Key insights

• The classical quantile Huber loss can be reinterpreted as minimizing the 1-Wasserstein distance between Dirac delta distributions, implying an assumption of zero noise.
• Introducing Gaussian noise into the predicted and target quantiles leads to a generalized quantile Huber loss that is more robust to outliers and offers smoother differentiability.
• The threshold parameter 'k' in the classical quantile Huber loss can be approximated by 'b', the absolute difference in standard deviations of the noise in the predicted and target quantiles, providing a data-driven, interpretable adjustment.

Summary by alphaXiv.