Reproduce “Pose-RFT: Enhancing MLLMs for 3D Pose Generation via Hybrid Action Reinforcement Fine-Tuning” on OpenResearch

About this paper

Bao Li, Xiaomei Zhang, Miao Xu, Zhaoxin Fan, Xiangyu Zhu, Zhen Lei

Pose-RFT introduces a reinforcement fine-tuning framework for Multimodal Large Language Models (MLLMs), optimizing 3D human pose generation from images and text using a novel hybrid action reinforcement learning algorithm, HyGRPO. The framework achieved improved Mean Per Joint Position Error (MPJPE) on 3DPW and Human3.6M datasets and higher Recall@K scores on PoseScript-H2, outperforming previous MLLM-based methods.

Key insights

• Jointly optimizing discrete (text) and continuous (3D pose) outputs using a hybrid reinforcement learning approach effectively enhances MLLMs' ability to generate robust and semantically aligned 3D human poses.
• Verifiable, task-specific reward functions are critical for guiding policy optimization, particularly the semantic alignment reward which significantly improves text-to-pose generation beyond reconstruction-focused metrics.
• Incorporating a pose-aware encoder, specifically a pretrained Vision Transformer, provides crucial fine-grained pose-relevant features, leading to notable improvements in image-to-pose spatial accuracy.

Summary by alphaXiv.