Reproduce “MICo: Improved representations via sampling-based state similarity for Markov decision processes” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Pablo Samuel Castro, Tyler Kastner, Prakash Panangaden, Mark Rowland
MICo, a new behavioral distance for Markov decision processes, facilitates learning improved representations for deep reinforcement learning agents by addressing computational complexity and bias in prior state similarity metrics. It consistently improved performance across various DRL agents on challenging benchmarks like Atari and DM-Control.
Key insights
- The MICo distance offers significantly reduced computational complexity, scaling as $O(|X|^4)$ compared to $O(|X|^5|A|)$ for bisimulation, by avoiding optimal transport computations and action maximization.
- MICo provides an on-policy bound for value function differences ($|V^\pi(x) - V^\pi(y)| \le U^\pi(x, y)$), making the learned similarity directly relevant to the agent's current operating policy.
- Characterizes MICo as a 'diffuse metric,' where non-zero self-distances ($U^\pi(x, x) > 0$) can measure the inherent stochasticity or 'dispersion' of a state's future trajectories under a given policy.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.