Reproduce “ChronusOmni: Improving Time Awareness of Omni Large Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yijing Chen, Yihan Wu, Kaisi Guan, Yuchen Ren, Yuyue Wang, Ruihua Song, Liyun Ru
ChronusOmni introduces a framework enhancing temporal awareness in omni large language models for both explicit and implicit audiovisual temporal grounding. It employs a novel temporal interleaved tokenization, a two-stage training strategy, and the new ChronusAV dataset, achieving state-of-the-art performance across six distinct audiovisual temporal grounding subtasks.
Key insights
- Explicitly encoding time as text tokens and interleaving them with visual and audio tokens enables fine-grained, interpretable temporal alignment within LLMs.
- A phased training approach, progressing from broad supervised alignment to reinforcement learning for fine-tuned precision, is effective in instilling robust temporal awareness.
- A dedicated, comprehensive dataset with disentangled and precise annotations for both explicit and implicit cross-modal temporal relations is critical for advancing the field of audiovisual temporal grounding.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.