Reproduce “Recurrent networks, hidden states and beliefs in partially observable environments” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Gaspard Lambrechts, Adrien Bolland, Damien Ernst
Gaspard Lambrechts, Adrien Bolland, and Damien Ernst empirically demonstrate that recurrent neural networks (RNNs) trained for Q-learning in partially observable environments develop hidden states that encode the relevant belief state, measured by mutual information (MI). Their work shows this implicit belief representation correlates positively with agent performance and indicates that RNNs learn to filter out irrelevant information.
Key insights
- Recurrent Neural Networks (RNNs) implicitly develop hidden states that act as belief filters, encoding the posterior probability distribution over the true state.
- The ability of an RNN to form a rich representation of the belief, as quantified by mutual information (MI), is directly correlated with the agent's performance (expected return).
- RNNs learn to selectively filter information, increasing MI with relevant state variables while decreasing MI with irrelevant state variables during training.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.