Reproduce “Towards Long-Lived Robots: Continual Learning VLA Models via Reinforcement Fine-Tuning” on OpenResearch

About this paper

Yuan Liu, Haoran Li, Shuai Tian, Yuxing Qin, Yuhui Chen, Yupeng Zheng, Yongzhen Huang, Dongbin Zhao

Researchers from the Chinese Academy of Sciences (CASIA) developed LifeLong-RFT, a reinforcement fine-tuning strategy for Vision-Language-Action (VLA) models, to enable continual learning without online environmental interaction. The method improved average success rates by 3.5% to 8.7% across various simulated and real-world multi-task benchmarks, and demonstrated superior knowledge retention and faster adaptation in continual learning scenarios.

Key insights

• Fine-tuning VLA models using reinforcement learning with internal, process-based rewards can mitigate catastrophic forgetting and data dependence, eliminating the need for environmental interaction.
• A multi-dimensional reward signal that evaluates quantized action consistency, continuous trajectory alignment, and structural format compliance provides effective guidance for learning robust robot policies.
• On-policy reinforcement learning, even at a 'chunking-level' for discrete actions, can enable efficient adaptation to new tasks and better knowledge preservation in continual learning settings.

Summary by alphaXiv.