Reproduce “Decoding Style: Efficient Fine-Tuning of LLMs for Image-Guided Outfit Recommendation with Preference” on OpenResearch

About this paper

Najmeh Forouzandehmehr, Nima Farrokhsiar, Ramin Giahi, Evren Korpeoglu, Kannan Achan

A framework efficiently fine-tunes Large Language Models (LLMs) for image-guided outfit recommendations by combining Multimodal LLMs for visual understanding with Parameter-Efficient Fine-Tuning and Direct Preference Optimization. The approach achieved an AUC of 81.03% for outfit compatibility prediction and 61% accuracy for fill-in-the-blank outfit completion on the Polyvore dataset, substantially improving over PEFT-only baselines.

Key insights

• Combining MLLM image captioning with a two-stage LLM fine-tuning (PEFT + DPO) effectively enables LLMs to understand and reason about complex visual fashion attributes.
• Direct Preference Optimization (DPO) is crucial for aligning LLM outputs with subjective human fashion sensibilities, leading to significantly more accurate and human-preferred recommendations.
• Parameter-Efficient Fine-Tuning makes the adaptation of large LLMs for specialized, multimodal tasks practical and cost-effective for industrial applications.

Summary by alphaXiv.