Reproduce “Small Reward Models via Backward Inference” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yike Wang, Faeze Brahman, Shangbin Feng, Teng Xiao, Hannaneh Hajishirzi, Yulia Tsvetkov
Researchers at the University of Washington and Allen Institute for Artificial Intelligence developed FLIP, a backward inference method that enables small language models to effectively serve as reward models by inferring original instructions from responses. This approach consistently achieved higher accuracy than LLM-as-a-Judge baselines by an average of 79.6% in intrinsic evaluations and improved downstream task performance.
Key insights
- Small language models (SLMs) exhibit a 'validation–generation gap,' where they struggle with direct judgment but retain relatively strong generative inference capabilities.
- Reframing reward modeling as a generative task of instruction reconstruction effectively leverages these generative strengths of SLMs for quality assessment.
- The word-level F1 score serves as an effective and computationally lightweight similarity metric for comparing inferred and original instructions, suitable for FLIP's approach.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.