Reproduce “Enhancing Multi-Image Understanding through Delimiter Token Scaling” on OpenResearch

About this paper

Minyoung Lee, Yeji Park, Dongjun Hwang, Yejin Kim, Seong Joon Oh, Junsuk Choe

The method strengthens the ability of Large Vision-Language Models to process multiple input images by scaling delimiter tokens, effectively reducing cross-image information leakage. This approach yields consistent performance gains across diverse multi-modal benchmarks and generalizes to multi-document tasks, achieving these improvements with no additional computational cost.

Key insights

• Image delimiter tokens exhibit a unique, localized strong attention pattern, predominantly receiving attention from tokens within their corresponding image, unlike global sink tokens.
• Delimiter tokens act as image-specific "tags," strengthening intra-image interactions by introducing a common additive term in the attention output for all tokens within that image.
• Scaling delimiter token hidden states enhances their attractive power, redirecting attention predominantly to the correct image's context and reducing attention leakage to other images.
• The approach achieves substantial performance improvements without incurring additional computational overhead for training or inference.

Summary by alphaXiv.