Reproduce “Time-Reversal Provides Unsupervised Feedback to LLMs” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Yerram Varun, Rahul Madhavan, Sravanti Addepalli, Arun Suggala, Karthikeyan Shanmugam, Prateek Jain

Google DeepMind researchers introduce Time Reversed Language Models (TRLMs) that generate unsupervised feedback by operating in a response query direction. This novel approach leads to up to a 5% improvement in AlpacaEval reranking win rates, significantly enhances citation attribution by 44.15% in Gecko cosine similarity, improves document retrieval NDCG@10 by 44.2 points on NF-Corpus, and reduces jailbreak attack false negative rates by over 70% with negligible false positive impact.

Key insights

  • Pre-training language models in a fully reversed token order (TRLM-Ba) yields distinct performance advantages for reverse operations compared to merely prompting forward-trained models.
  • TRLMs can provide valuable unsupervised feedback for LLM outputs, offering an alternative to expensive human-supervised alignment methods.
  • Using TRLM scores for alignment, as demonstrated by a stylized theoretical model, can induce a distribution shift that may help mitigate hallucination by restricting answers to valid ground truth.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.