Reproduce “Iterative Bounding MDPs: Learning Interpretable Policies via Non-Interpretable Methods” on OpenResearch

About this paper

Nicholay Topin, Stephanie Milani, Fei Fang, Manuela Veloso

Researchers from Carnegie Mellon University introduce Iterative Bounding MDPs (IBMDPs) and the CUSTARD framework, enabling modern Deep Reinforcement Learning algorithms to directly learn interpretable decision tree policies. This method yields high-performing and more concise decision trees compared to post-hoc approximation techniques, particularly in complex environments or under depth constraints.

Key insights

• By formulating decision tree learning as a meta-MDP, it becomes possible to directly train inherently interpretable policies using the powerful function approximators of modern DRL, bridging a critical gap in XAI.
• The IBMDP framework, with a small penalty for information-gathering actions, intrinsically regularizes the learning process, leading to the discovery of shallower and more concise decision trees.
• The proposed modifications to RL algorithms, particularly the dual discount factors and the omniscient Q-function, enable efficient reward propagation and stable learning within the unique hierarchical structure of IBMDPs.

Summary by alphaXiv.