Reproduce “Vista-LLaMA: Reducing Hallucination in Video Language Models via Equal Distance to Visual Tokens” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Fan Ma, Xiaojie Jin, Heng Wang, Yuchen Xian, Jiashi Feng, Yi Yang

VISTA-LLAMA, developed by researchers from Zhejiang University and ByteDance Inc., introduces a framework that reduces hallucination in video-language models by ensuring visual tokens maintain an equal influence on generated text and by explicitly modeling temporal relationships. The approach achieves state-of-the-art accuracy on NExT-QA (60.7%), MSVD-QA (65.3%), and MSRVTT-QA (60.5%) in zero-shot open-ended VideoQA tasks.

Key insights

  • Selectively disabling positional embeddings for visual-text interactions can effectively counter the diminishing influence of visual cues, reducing hallucination in long text generation.
  • Explicitly modeling temporal dependencies within the visual projection process through a sequential approach can improve video understanding and enable more efficient processing of longer videos.
  • Pre-training the visual projector on image-text pairs, as with a Q-Former initialized from BLIP-2, is crucial for effectively mapping visual features into the language embedding space.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.