Reproduce “PLUME: Latent Reasoning Based Universal Multimodal Embedding” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Chenwei He, Xiangzhao Hao, Tianyu Yang, Yuxiang Ma, Yuheng Jia, Lingxiang Wu, Chaoyang Zhao, Haiyun Guo and 1 more
Researchers from Southeast University and Chinese Academy of Sciences developed PLUME, a framework that integrates multi-step latent reasoning into universal multimodal embedding by replacing explicit textual rationales with compact hidden-state computations. This approach enhanced retrieval performance across diverse modalities while achieving a substantial reduction in inference latency.
Key insights
- Multi-step reasoning in Universal Multimodal Embedding can be internalized into a compact, continuous latent rollout within the MLLM's hidden space, preserving rich multimodal information more effectively than discrete textual rationales.
- Adaptive latent computation, guided by a semantic anchor, allows the model to dynamically allocate reasoning resources and follow distinct trajectories for various multimodal inputs, optimizing performance across diverse tasks.
- A progressive curriculum that gradually absorbs explicit reasoning steps into latent computation is critical for stabilizing the training process and successfully transferring complex reasoning capabilities without direct explicit supervision.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.