Reproduce “Efficient Hypergradient Descent for Inverse Reinforcement Learning” on OpenResearch

About this paper

Nikita Sevriukov, Anna Barabanova, Uliana Gagarina, Karina Ivanova, Sofiia Kasaeva, Ilya Levin, Marina Sheshukova

Researchers from HSE University developed an efficient hypergradient descent method for Maximum-Likelihood Inverse Reinforcement Learning, proving that the inner objective's Hessian is proportional to the discounted trajectory Fisher information matrix. Their approach utilizes a Spectral Compensation Frequent Directions (SCFD) sketching technique to approximate the inverse-Fisher-vector product, achieving significant memory and runtime efficiency while maintaining competitive performance in reward inference.

Key insights

• The Hessian of the inner objective in bilevel ML-IRL is proportional to the discounted trajectory Fisher information matrix, allowing for Fisher-based approximations to address the computational bottleneck.
• Matrix sketching with SCFD provides a memory-efficient and scalable way to compute the inverse-Fisher-vector product, enabling implicit hypergradient descent for high-dimensional policies.
• Appropriate damping in the Fisher system is crucial for stability, with sketching allowing for weaker damping while maintaining performance and improved stability over explicit Fisher construction.

Summary by alphaXiv.