Reproduce “CoAct: Co-Active LLM Preference Learning with Human-AI Synergy” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Ruiyao Xu, Mihir Parmar, Tiankai Yang, Zhengyu Hu, Yue Zhao, Kaize Ding
CoAct combines self-labeling and oracle review to build preference data for reasoning tasks. It samples multiple responses, uses consistency to form pairs, sends uncertain and selected high-consistency cases for oracle checking, and uses verified examples to generate instructions. After four iterations, it improves accuracy by 13.25 points on GSM8K, 8.19 on MATH, and 13.16 on WebInstruct, evaluated on objective-answer tasks.
Key insights
- LLM self-consistency scores strongly correlate with test accuracy and can reliably proxy response quality, strengthening with model performance.
- Strategically applying human oracle feedback to both uncertain and potentially self-consistently incorrect AI-generated preferences is crucial for effective learning.
- Oracle-guided instruction augmentation effectively diversifies training data within the model's solvable range, enhancing generalization.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.