Reproduce “ForgeVLA: Federated Vision-Language-Action Learning without Language Annotations” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yuhao Zhou, Yunpeng Zhu, Yang Zhou, Jindi Lyu, Jian Lan, Zhangyuan Wang, Dan Si, Thomas Seidl and 2 more
ForgeVLA turns distributed vision–action logs into VLA training examples by selecting instructions from a predefined closed set, training locally, and sharing model updates rather than raw logs. A classifier recovers instructions, while task-representation anchors and contrastive learning address feature collapse. On LIBERO-Goal, it achieves 55.2% success versus 28.8% for FedAvg, without formal privacy guarantees.
Key insights
- A "vision-language feature collapse" occurs in heterogeneous federated VLA learning, where task-specific multimodal representations lose their distinctiveness, degrading performance.
- Conventional non-i.i.d. mitigation strategies like FedProx can degrade performance in federated VLA settings rather than improve it, highlighting the need for VLA-specific solutions.
- Pre-trained VLMs can be effectively fine-tuned to create embodied instruction classifiers that reliably annotate unannotated vision-action data on-device, even with limited initial resources.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.