Reproduce “Seeing the Arrow of Time in Large Multimodal Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zihui Xue, Mi Luo, Kristen Grauman
Researchers from The University of Texas at Austin introduced ArrowRL, a reinforcement learning strategy to imbue Large Multimodal Models (LMMs) with "Arrow of Time" (AoT) awareness, allowing them to discern temporal directionality in videos. The method, alongside a new benchmark called AoTBench, significantly improves LMMs' ability to distinguish between forward and reversed video sequences and reason about event causality, achieving up to a +21.0% gain on challenging temporal tasks.
Key insights
- State-of-the-art LMMs demonstrate a "universal failure mode" in fundamental AoT perception tasks, performing at near-chance levels when asked to classify video direction or generate temporally distinct descriptions.
- A "Temporal Divergence Score (TDS)" can effectively quantify an LMM's sensitivity to temporal directionality within existing benchmarks, identifying questions that genuinely require AoT understanding.
- Reinforcement learning, specifically with a reverse reward mechanism, is an effective strategy to train LMMs for AoT awareness, leading to improved performance without degrading general video understanding.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.