Reproduce “SeMoCo: A Semantic-First Motion Codec for Motion Language Modeling” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Tianlv Huang, Hetian Guo, Ziyi Cai, Song Wang, Yanping Zhang, Zipei Fan, Xuan Song, Guangming Wu and 1 more

SeMoCo introduces a semantic-first motion codec that separates behaviorally meaningful tokens from residual kinematic detail, then generates them through separate temporal and within-packet prediction axes. Trained with a unified 909,913-pair, 1,006-hour Ω-MotionVerse corpus, it improves motion reconstruction and retrieval-based text alignment, while exposing a trade-off between semantic correspondence and physical smoothness.

Key insights

  • Separating semantic supervision from the residual chain improves language-relevant motion representation: in the reported ablation, overall TMR-SOMA FID changes from 0.226 to 0.186 and R@1 from 0.319 to 0.484 when semantic distillation is added to the split-branch codec.
  • Semantic alignment comes with a reconstruction trade-off: MPJPE-77 rises from 13.70 mm to 15.93 mm with semantic supervision, although the parallel design incurs a smaller penalty than supervising the first code in a single RVQ chain.
  • Larger model capacity does not automatically improve forecasting, and retrieval quality does not guarantee physical realism; SeMoCo’s generated motions show higher jerk and foot-skating than some comparison systems.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.