Reproduce “LongDPO: Unlock Better Long-form Generation Abilities for LLMs via Critique-augmented Stepwise Information” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Bowen Ping, Jiali Zeng, Fandong Meng, Shuo Wang, Jie Zhou, Shanghang Zhang
The LongDPO framework enhances large language models' ability to generate high-quality, long-form text by employing critique-augmented stepwise information and a fine-grained preference learning objective. It significantly improves output quality, consistency, and instruction adherence on long-form benchmarks while maintaining general task performance, such as an 8% improvement in length score for Llama-based models on outputs exceeding 4,000 words.
Key insights
- Process supervision, which provides granular feedback at each generation step, is more effective than outcome-level supervision for enhancing long-form text generation quality and consistency.
- Utilizing a dynamic global memory pool during generation can effectively combat factual inconsistencies and hallucinations in long outputs, improving overall reliability.
- External critiques from a powerful model are crucial for refining suboptimal chosen candidates in preference data, leading to more stable and effective learning compared to self-critique mechanisms.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.