Reproduce “NaiLIA: Multimodal Nail Design Retrieval Based on Dense Intent Descriptions and Palette Queries” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Kanon Amemiya, Daichi Yashima, Kei Katsumata, Takumi Komatsu, Ryosuke Korekata, Seitaro Otsuki, Komei Sugiura
Keio University researchers developed NaiLIA, a multimodal retrieval system capable of accurately searching for nail designs using detailed intent descriptions and continuous color palette queries. NaiLIA achieved superior performance, outperforming state-of-the-art baselines by up to 8.9 Recall@1 points on the new NAIL-STAR benchmark and 9.4 points on Marqo Fashion200K.
Key insights
- Deconstructing dense user intent into structured components using LLMs (e.g., multi-layered design descriptions and normalized noun phrases) significantly improves the model's understanding of complex queries.
- Fusing multiple visual feature types, including those derived from MLLM-generated descriptions of image design (img2txt intent structure representation), is crucial for capturing abstract visual concepts and intricate design relationships.
- Incorporating MLLM-estimated confidence scores for "unlabeled positive" samples into a relaxed contrastive loss function enhances learning effectiveness by accounting for semantic similarities beyond explicit labels.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.