Reproduce “BidirLM: From Text to Omnimodal Bidirectional Encoders by Adapting and Composing Causal LLMs” on OpenResearch

About this paper

Nicolas Boizard, Théo Deschamps-Berger, Hippolyte Gisserot-Boukhlef, Céline Hudelot, Pierre Colombo

The BIDIRLM framework transforms causal decoder LLMs into omnimodal bidirectional encoders, demonstrating that a sequence of Masked Next-Token Prediction and contrastive learning, combined with weight merging, efficiently adapts and composes specialized causal models. This approach establishes new performance frontiers across diverse text, image, and audio representation tasks, exemplified by the creation of BidirLM-Omni-2.5B which surpasses larger multimodal models while being computationally efficient.

Key insights

• A prior MNTP adaptation phase is crucial for effectively leveraging bidirectional attention and unlocking robust performance on fine-tuning tasks when adapting causal LLMs into encoders.
• MNTP and InfoNCE contrastive training are complementary; a sequential application (MNTP followed by contrastive training) yields strong performance across both fine-tuning and generic embedding evaluations.
• Linear weight merging with the original base model and the inclusion of lightweight multi-domain data mixtures are effective, data-independent strategies for mitigating catastrophic forgetting during scaled adaptation.

Summary by alphaXiv.