Reproduce “MonSTeR: a Unified Model for Motion, Scene, Text Retrieval” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Luca Collorone, Matteo Gioia, Massimiliano Pappa, Paolo Leoni, Giovanni Ficarra, Or Litany, Indro Spinelli, Fabio Galasso
MonSTeR introduces a unified model for motion, scene, and text retrieval that models higher-order interactions between these modalities in a single latent space. This system achieves superior performance in complex cross-modal retrieval tasks and effectively serves as an evaluator for human-scene interaction generative models.
Key insights
- Modeling higher-order interactions (e.g., aligning a scene-text pair with motion) beyond simple pairwise relationships is crucial for fully capturing the complex interdependencies within multimodal data.
- A unified latent space that explicitly incorporates unimodal, cross-modal, and higher-order alignment terms enables robust and flexible retrieval across various combinations of text, motion, and scene.
- Such a comprehensive multimodal model can reliably function as an evaluator for generative Human Scene Interaction (HSI) models, capable of assessing motion plausibility, scene compliance, and alignment with human preferences.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.