Reproduce “Beyond Scene Shortcuts: Action-Centric Video-Text Alignment for Fine-Grained Action Understanding” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Zhiliang Zhu, Jiazheng Huang, Dawei Zhao

The authors introduce a training framework that reduces reliance on surrounding scenes when describing video actions and rewards attention to caption-selected visual evidence. It combines Action-Environment Dissociation with an attention consistency reward during reinforcement learning. At 2 frames per second, the system improves MotionBench accuracy and, in a 400-video human evaluation, lowers unsupported-action captions while increasing action recall; these results do not establish that hallucinations are eliminated.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.