Reproduce “Enhancing Interpretability in Deep Reinforcement Learning through Semantic Clustering” on OpenResearch

About this paper

Liang Zhang, Justin Lieffers, Adarsh Pyarelal

Researchers at the University of Arizona developed a method integrating dimensionality reduction and online clustering into Deep Reinforcement Learning (DRL) to enhance model interpretability by revealing the intrinsic semantic organization of learned feature spaces. This approach generates stable, well-separated state clusters that correspond to human-understandable agent "skills" and enables dynamic policy analysis, with human evaluators giving mean Likert scores above 4 for interpretability.

Key insights

• Deep Reinforcement Learning models intrinsically develop semantic clustering capabilities, organizing policy-relevant information dynamically through agent-environment interactions, even without explicit supervision.
• The Semantic Clustering Module (SCM) effectively sharpens and stabilizes these naturally occurring semantic clusters, providing a more distinct and robust representation for analysis.
• Integrating interpretable VQ codes into the state features allows for a latent-conditioned policy, offering a foundation for hierarchical learning and providing human-understandable "skills."

Summary by alphaXiv.