Reproduce “Audio-Visual Scene-Aware Dialog” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Huda Alamri, Vincent Cartillier, Abhishek Das, Jue Wang, Anoop Cherian, Irfan Essa, Dhruv Batra, Tim K. Marks and 4 more
The authors introduce audio-visual scene-aware dialog (AVSD), a task and benchmark for answering follow-up questions using video, audio, the current question, and earlier dialogue. On 11,816 Charades videos, baseline models rank 100 candidate answers in retrieval evaluation. Adding video to question and dialogue history raises MRR from 50.40 to 53.41; adding audio lowers MRR to 53.03, though mean rank improves slightly.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.