Reproduce “The Factorization Curse: Which Tokens You Predict Underlie the Reversal Curse and More” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Ouail Kitouni, Niklas Nolte, Diane Bouchacourt, Adina Williams, Mike Rabbat, Mark Ibrahim

Researchers from FAIR (Meta) and MIT identified the "factorization curse" as the core reason large language models struggle with retrieving information in a different order than trained, stemming from the standard next-token prediction objective. They demonstrate that Uniform-Rate Masked Language Modeling (MLM-U), which uses variable masking rates, consistently improves backward-direction retrieval accuracy and leads to more structured knowledge representations in LLMs.

Key insights

  • The "factorization curse," rooted in the next-token prediction objective, is a fundamental limitation causing LLMs to struggle with retrieving information in varying orders.
  • Uniform-Rate Masked Language Modeling (MLM-U) trains models on a wide spectrum of masking patterns, allowing for a more factorization-agnostic representation of joint probability distributions.
  • Addressing the factorization curse through objectives like MLM-U can also improve performance on higher-order cognitive tasks such as planning.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.