Reproduce “Attention-Based Multimodal Fusion for Video Description” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Chiori Hori, Takaaki Hori, Teng-Yok Lee, Kazuhiro Sumi, John R. Hershey, Tim K. Marks

The authors introduce a video captioning method that adjusts how appearance and motion features contribute as each word is generated. It uses the decoder’s current context to weight the streams. On YouTube2Text, it scores 0.507 versus 0.498 on BLEU-4 and 0.699 versus 0.665 on CIDEr, but lower on four other metrics; optimizers differ, so the comparison does not isolate adaptive weighting.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.