Reproduce “HMVLM: Human Motion-Vision-Lanuage Model via MoE LoRA” on OpenResearch

About this paper

Lei Hu, Yongjing Ye, Shihong Xia

A Human Motion-Vision-Language Model (HMVLM) is proposed, utilizing a Mixture of Expert Low-Rank Adaptation (MoE LoRA) strategy and body-part-specific tokenization to integrate 3D human motion into foundation language models. This framework significantly mitigates catastrophic forgetting during multimodal fine-tuning and provides enhanced spatial understanding of human poses, enabling a unified approach for motion generation, perception, and understanding.

Key insights

• The MoE LoRA strategy, particularly with its "zero expert" and dedicated gating loss, successfully prevents catastrophic forgetting in foundation models during multimodal adaptation.
• Discretizing human motion based on body parts significantly enhances spatial resolution and semantic fidelity of tokens, improving performance on both pose estimation and motion generation.
• It is feasible to create a unified instruction-tuned model capable of diverse human motion-vision-language tasks, balancing general linguistic abilities with specialized multimodal performance.

Summary by alphaXiv.