Reproduce “Images are Worth Variable Length of Representations” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Lingjun Mao, Rodolfo Corona, Xin Liang, Wenhao Yan, Zineng Tang
Researchers from UC San Diego, UC Berkeley, and the University of Washington developed DOVE, a dynamic vision encoder that generates variable-length image representations by adaptively determining token sequence length based on content complexity. This approach yields improved reconstruction quality and classification accuracy with reduced token counts, and its query-conditioned variant, Q-DOVE, enhances performance in vision-language tasks while maintaining efficiency.
Key insights
- Dynamically adjusting the length of image representations based on content complexity can significantly improve both computational efficiency and the semantic richness of the generated features.
- The use of an EOS token within an autoregressive transformer, guided by a dynamic reconstruction loss threshold, effectively enables adaptive token sequence termination in a single forward pass.
- Conditioning the dynamic tokenization process on external queries allows for task-driven resource allocation, concentrating representational effort on relevant image areas for enhanced multimodal understanding.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.