Reproduce “Language Models As Semantic Indexers” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Bowen Jin, Hansi Zeng, Guoyin Wang, Xiusi Chen, Tianxin Wei, Ruirui Li, Zhengyang Wang, Zheng Li and 5 more

A self-supervised generative language model framework, LMINDEXER, learns sequential discrete semantic IDs for documents that capture hierarchical meaning. This method enhances performance across recommendation, product search, and document retrieval tasks while demonstrating improved generalization to unseen data.

Key insights

  • A shallow reconstructor, in conjunction with a reconstruction objective, forces the semantic indexer to encode robust, high-quality information into the discrete IDs.
  • Progressive training, by gradually reducing context hints, successfully fosters a hierarchical, coarse-to-fine-grained semantic representation within sequential IDs.
  • Specific strategies, including reconstructor warm-up and K-means initialization for codebooks, effectively mitigate training instability issues like reconstructor and posterior collapse.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.