Reproduce “Elysium: Exploring Object-level Perception in Videos via MLLM” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Han Wang, Yanjie Wang, Yongjie Ye, Yuxiang Nie, Can Huang
Elysium, developed by Bytedance Inc., introduces an end-to-end trainable Multi-modal Large Language Model (MLLM) for object-level video perception. The model achieves state-of-the-art results in image grounding and zero-shot VideoQA, while demonstrating superior performance over baselines on novel Referring Single Object Tracking (AUC 87.5) and Video Referring Expression Generation (Meteor 46.7) tasks, facilitated by a new million-scale dataset and a token compression network.
Key insights
- Intelligent visual token compression via the T-Selector effectively reduces the computational burden for LLMs, enabling efficient processing of longer video sequences while retaining salient information.
- A unified, end-to-end MLLM architecture can robustly handle diverse object-level video tasks without relying on external, specialized expert models.
- Explicitly incorporating temporal awareness and training on large-scale, high-quality trajectory data is crucial for endowing MLLMs with the ability to bridge language understanding and dynamic object tracking in videos.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.