Reproduce “Unhackable Temporal Rewarding for Scalable Video MLLMs” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
En Yu, Kangheng Lin, Liang Zhao, Yana Wei, Zining Zhu, Haoran Wei, Jianjian Sun, Zheng Ge and 3 more
Researchers identified "temporal hacking" as a phenomenon causing an "anti-scaling law" in video MLLMs, where models focus on individual frames instead of the entire temporal sequence. Their Unhackable Temporal Rewarding (UTR) framework, incorporating spatiotemporal attribute extraction and bidirectional querying, enabled Video-UTR to achieve improved temporal understanding across various video-language tasks.
Key insights
- Temporal hacking in video MLLMs is re-framed as a form of "reward hacking" where models optimize for a proxy reward (video-text consistency) misaligned with comprehensive video-language alignment.
- Temporal Perplexity (TPL) is introduced as a novel metric to quantify temporal hacking, measuring the difference in perplexity between full video sequences and single keyframes.
- The Unhackable Temporal Rewarding (UTR) framework is designed to encourage models to process and integrate information across all frames by promoting high frame information density and high inter-frame information dynamics.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.