Reproduce “Streaming Reinforcement Learning under Partial Observability with Real-Time Recurrent Learning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Noah Farr, Aryaman Reddi, Carlo D'Eramo, Jan Peters
The paper extends streaming reinforcement learning to partial observability by placing a recurrent trace unit between observations and a streaming output head. Its diagonal recurrence enables exact real-time recurrent learning with linear time and memory in parameter count, while an eligibility trace accumulates temporal credit. On MemoryChain, it stays near maximum through 48 steps, but trails batched PPO in most continuous-control regimes.
Key insights
- Exact RTRL, rather than solely the diagonal recurrent architecture of RTUs, is critical for enabling effective long-term credit assignment in streaming reinforcement learning settings.
- RTRL's forward-mode sensitivity traces naturally integrate with streaming algorithm eligibility traces, facilitating untruncated credit propagation through both recurrent states and time.
- RTRL sensitivity matrices exhibit staleness under continuous parameter updates in streaming environments, with the error scaling linearly with the step size (O(η)), and this staleness can be reduced by a first-order Taylor correction.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.