Reproduce “LLaMA-VID: An Image is Worth 2 Tokens in Large Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yanwei Li, Chengyao Wang, Jiaya Jia
LLaMA-VID enables Large Language Models to efficiently understand and reason over hour-long videos by representing each image or frame with only two visual tokens. This dual-token approach achieves leading performance on both video- and image-based benchmarks while significantly reducing computational overhead.
Key insights
- Representing visual information with a dual-token strategy—a context token for overall information guided by instruction and an adaptive content token for fine-grained details—drastically reduces the number of visual tokens needed per frame while preserving critical information.
- Instruction-guided context tokens, generated through an attention mechanism interacting with user queries, significantly enhance performance by focusing on relevant visual cues.
- A multi-stage training approach, starting with modality alignment and progressing to instruction tuning and long video-specific tuning, effectively enables models to understand and reason over hour-long video content.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.