Reproduce “Reinforcement Learning, Bit by Bit” on OpenResearch

About this paper

Xiuyuan Lu, Benjamin Van Roy, Vikranth Dwaracherla, Morteza Ibrahimi, Ian Osband, Zheng Wen

This work from DeepMind develops an information-theoretic framework for data-efficient reinforcement learning, introducing Information-Directed Sampling (IDS) and Epistemic Neural Networks (ENNs) to guide intelligent information acquisition and retention. The research establishes a general regret bound that decomposes total regret, demonstrating how richer representations like General Value Functions and strategic learning targets can accelerate learning in data-scarce environments.

Key insights

• Data efficiency in RL can be fundamentally improved by adopting an 'information-first' approach, systematically quantifying and managing uncertainty, akin to breakthroughs in communication theory.
• Using richer environment proxies, such as General Value Functions (GVFs), significantly improves data efficiency by allowing agents to retain more diverse and useful information from observations beyond just immediate rewards.
• Information-Directed Sampling (IDS) enables 'deep exploration' by actively seeking out actions that are highly informative, even if not immediately optimal, leading to exponentially better performance than purely optimistic exploration methods in certain environments.
• Total regret can be elegantly decomposed into the total environment-relevant information about the learning target, the cost per bit of acquired information (sum of information ratios), and the inherent suboptimality of the target policy itself, providing clear design principles for data-efficient agents.

Summary by alphaXiv.