Reproduce “Autoregressive Ranking: Bridging the Gap Between Dual and Cross Encoders” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Benjamin Rozonoyer, Chong You, Michael Boratko, Himanshu Jain, Nilesh Gupta, Srinadh Bhojanapalli, Andrew McCallum, Felix Yu

Autoregressive Ranking (ARR) is presented as a generative alternative to dual- and cross-encoder ranking: a causal language model generates document identifiers and can represent orderings that fixed-dimensional vector geometry cannot. The paper proves a capacity separation under idealized assumptions and introduces SToICaL, a rank-aware loss that improves multi-document retrieval, reaching roughly 99.8 nDCG on WordNet and increasing ESCI recall@5 from 27.64% to 51.07%.

Key insights

  • A dual encoder in an n-dimensional Euclidean space cannot represent all rankings of k documents when n is below ln(k!)/(2 ln k), implying an embedding-dimension requirement that grows approximately linearly with corpus size.
  • Under an infinite-capacity hidden-state assumption, ARR can express arbitrary rankings over arbitrarily many multi-token docIDs when the augmented valid-token embedding matrix has full rank.
  • Rank-aware supervision improves deeper retrieval beyond top-1 training, but trie marginalization can trade top-rank precision for broader ranked-set recall and remains sensitive to docID design and teacher-forcing mismatch.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.