Reproduce “Separating Representation from Reconstruction Enables Scalable Text Encoders” on OpenResearch

About this paper

Megi Dervishi, Mathurin Videau, Yann LeCun

CrossBERT trains a text encoder to form representations while a smaller predictor reconstructs masked tokens through cross-attention. Complementary masking supplies two views, and the predictor bears the direct reconstruction task. In frozen-backbone MTEB evaluation after one epoch of adapter training on MS-MARCO, CrossBERT averages 53.6 across seven categories, versus BERT’s 45.7; this is not zero-shot evaluation.

Key insights

• Standard BERT's 'flat' architecture leads to representations over-specialized for token reconstruction, causing frozen representation quality to degrade severely with increased scale.
• Decoupling representation learning from reconstruction in a bipartite architecture like CrossBERT enables monotonic scaling of high-quality, versatile frozen embeddings for text.
• High masking ratios and complementary masking not only boost training efficiency (throughput and sample efficiency) but also contribute to learning more robust and generalizable representations.

Summary by alphaXiv.