Reproduce “AnyJev Technical Report” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jiamu Zhang, Tianze Yang, Yucheng Shi, Evan Chen, Zixiang Nie, Kelly Wan, Liangjie Hong, Ninghao Liu and 1 more
AnyJev treats typed model decisions as a readout problem: it freezes an instruction-tuned language model, reads next-token probabilities for option labels, and averages log-probabilities across cyclic option rotations to reduce position bias. It estimates a label prior from unlabelled inputs. Rotation averaging reduced order-flip rates and increased accuracy for all 11 models on two 20-option tasks; however, JevBench’s public subset, with 2–6 options, showed no measurable accuracy change.
Key insights
- Across 11 models, rotation averaging reduced mean order-flip rates on two 20-option tasks, indicating that position bias is especially relevant when there are many choices.
- Accuracy gains were not universal: the 20-option tasks improved, while JevBench showed no measurable overall accuracy change and label-prior correction sometimes hurt performance.
- The stopping rule is designed to reproduce the full-rotation readout, not to guarantee correctness; some reported serving agreement estimates also used the same data to select and assess the threshold.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.