Reproduce “Filtering Learning Histories Enhances In-Context Reinforcement Learning” on OpenResearch

About this paper

Weiqin Chen, Xinjie Zhang, Dharmashankar Subramanian, Santiago Paternain

This paper introduces Learning History Filtering (LHF), a dataset preprocessing method designed to improve In-Context Reinforcement Learning (ICRL) by mitigating the inheritance of suboptimal behaviors from pretraining data. LHF consistently enhanced the performance of state-of-the-art ICRL algorithms across various environments, with average gains ranging from 8.8% to 46.2% depending on the algorithm and task, and showed particularly strong improvements in noisy data conditions.

Key insights

• The performance of In-Context Reinforcement Learning (ICRL) is highly sensitive to the quality of its pretraining dataset, as suboptimal behaviors in source learning histories are inherited by the in-context policy.
• Applying principles similar to Weighted Empirical Risk Minimization (WERM) to ICRL by probabilistically sampling learning histories based on their quality (improvement and stability metrics) can enhance learning.
• A data-centric preprocessing approach, rather than solely algorithmic or architectural improvements, offers a general and effective way to mitigate inherited suboptimality in ICRL.

Summary by alphaXiv.