Reproduce “Test-time Offline Reinforcement Learning on Goal-related Experience” on OpenResearch

About this paper

Marco Bagatella, Mert Albaba, Jonas Hübotter, Georg Martius, Andreas Krause

Goal-Conditioned Test-Time Training (GC-TTT) was introduced for offline reinforcement learning, dynamically fine-tuning pre-trained policies during evaluation by selecting relevant and optimal experience from the offline dataset. This approach consistently boosted the success rates of various backbone algorithms, such as improving GC-BC from 0.05 to 0.78 on a pointmaze expert task, and demonstrated more efficient performance scaling than increasing model size alone.

Key insights

• Locally specializing globally pre-trained GCRL policies at test-time can significantly overcome their suboptimality for specific evaluation goals.
• Effective test-time adaptation hinges on a dual data selection criterion that considers both the temporal relevance of experience to the current state and its optimality with respect to the target goal.
• Dynamically adapting a moderately sized policy at test-time offers better performance scaling and compute efficiency compared to simply increasing the size of a static pre-trained model.

Summary by alphaXiv.