Reproduce “A General Framework for Multimodal LLM-Based Multimedia Understanding in Large-Scale Recommendation Systems” on OpenResearch

About this paper

Yiming Zhu, Xu Liu, Ziyun Xu, Zheng Wu, Joena Zhang, Sirius Chen, Chenheli Hua, Silvester Yao and 4 more

Researchers at Meta Platforms developed a general framework to integrate Multimodal Large Language Models (MM-LLMs) into large-scale recommendation systems, enhancing multimedia content understanding to generate high-fidelity semantic features. This approach improved offline AUC by up to 0.16% and reduced Normalized Entropy by up to 0.07%, while also yielding a +0.02% increase in online user engagement metrics without adding latency.

Key insights

• Multimodal Large Language Models can extract fine-grained, latent semantic details from multimedia content that complement traditional visual features, enabling more nuanced user preference modeling.
• It is possible to effectively translate the generative outputs of MM-LLMs into structured features compatible with existing, latency-constrained industrial recommender systems without incurring additional computational overhead.
• Integrating MM-LLM-derived semantic features demonstrably improves the accuracy of recommender systems in offline benchmarks and increases real-world user engagement metrics.

Summary by alphaXiv.