Reproduce “AToM: Aligning Text-to-Motion Model at Event-Level with GPT-4Vision Reward” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Haonan Han, Xiangzuo Wu, Huan Liao, Zunnan Xu, Zhongyuan Hu, Ronghui Li, Yachao Zhang, Xiu Li
AToM uses GPT-4V to score rendered motions against event-level instructions, then fine-tunes MotionGPT with preference optimization. It targets missing actions, incorrect event order, and repetition counts. On the reported General test row, it has lower FID and higher Top-1 retrieval than InstructMotion, with similar MM-Dist; evidence is limited to these measures and the tested MotionGPT setup.
Key insights
- Advanced vision-language models like GPT-4Vision can effectively generate high-quality, fine-grained, event-level alignment rewards for complex generative tasks.
- Reinforcement learning with AI feedback (RLAIF) can significantly enhance the faithfulness of generated motions to detailed textual instructions, overcoming human annotation limitations.
- Explicitly designing datasets and reward mechanisms to target specific motion attributes (integrity, temporal, frequency) improves model performance on these challenging aspects.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.