Reproduce “Statistical Context Detection for Deep Lifelong Reinforcement Learning” on OpenResearch

About this paper

Jeffery Dick, Saptarshi Nath, Christos Peridis, Eseoghene Benjamin, Soheil Kolouri, Andrea Soltoggio

The Sliced Wasserstein Online Kolmogorov-Smirnov (SWOKS) algorithm is introduced to enable autonomous, online task detection in deep reinforcement learning (DRL) by combining optimal transport theory with statistical hypothesis testing. This method allows agents to learn and manage multiple policies for sequential tasks effectively, particularly addressing changes in environment dynamics or reward functions.

Key insights

• Leveraging Sliced Wasserstein Distance on latent action-reward spaces coupled with the Kolmogorov-Smirnov test provides a statistically sound and autonomous method for online context detection in LRL.
• The approach effectively identifies task changes even when they are subtle, like shifts in reward or transition functions, and can differentiate between known and novel tasks to enable appropriate policy switching.
• Tunable parameters (e.g., the KS adjustment "beta") allow for managing the trade-off between sensitivity to task changes and the rate of false positives, enhancing practical applicability across diverse domains.

Summary by alphaXiv.