Reproduce “VL-CLIP: Enhancing Multimodal Recommendations via Visual Grounding and LLM-Augmented CLIP Embeddings” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Ramin Giahi, Kehui Yao, Sriram Kollipara, Kai Zhao, Vahid Mirjalili, Jianpeng Xu, Topojoy Biswas, Evren Korpeoglu and 1 more
The VL-CLIP framework from Walmart Global Tech enhances multimodal recommendations by integrating visual grounding and LLM-augmented CLIP embeddings to capture fine-grained product details in e-commerce. It delivered an 18.6% increase in Click-Through Rate, 15.5% in Add-to-Cart, and 4.0% in Gross Merchandise Value during online A/B tests.
Key insights
- Explicitly refining image inputs using visual grounding to focus on product regions significantly improves fine-grained visual understanding for distinguishing similar items.
- Leveraging Large Language Models for iterative, self-correcting textual query synthesis mitigates ambiguity and enriches product descriptions for better semantic alignment.
- Domain-specific adaptation and contrastive fine-tuning of foundation models like CLIP, guided by refined multimodal inputs, are essential for achieving high performance in complex domains like e-commerce.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.