Reproduce “Deciding When to Rely on Visual Information: Gated Multimodal Fusion in Sequential Recommendation” on OpenResearch

About this paper

Natalija Glisovic, Danica Kragic, Martin Tegner

The VisGate framework introduces a context-aware adaptive gating mechanism for sequential recommender systems, dynamically deciding when to leverage visual information for individual items based on user interaction history. It achieves superior recommendation performance, particularly on sparse datasets, by learning complementary visual representations and providing insights into the item- and context-dependent utility of visual data.

Key insights

• Visual utility is a dynamic, item- and context-dependent variable, which varies based on the item's characteristics and the user's evolving interaction history.
• Learning and preserving complementary representations between modalities, rather than aligning them into a shared space, is crucial for effective multimodal fusion in sequential recommendation.
• The adaptive gating mechanism functions as an interpretable diagnostic tool, revealing semantically meaningful correlations between gate decisions and visual distinctiveness, and quantifying modality contributions.

Summary by alphaXiv.