Reproduce “DEPO: Dual-Efficiency Preference Optimization for LLM Agents” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Sirui Chen, Mengshi Zhao, Lei Xu, Yuying Zhao, Beier Zhu, Hanwang Zhang, Shengjie Zhao, Chaochao Lu
Researchers from Tongji University, Nanyang Technological University, and the Shanghai AI Lab developed DEPO (Dual-Efficiency Preference Optimization), a method that extends KTO to enhance the practical efficiency of LLM agents. This approach reduces token usage by up to 60.9% and interaction steps by up to 26.9% across various tasks, while preserving or improving task performance and demonstrating strong generalizability to out-of-domain reasoning problems.
Key insights
- Jointly optimizing for both step-level (succinctness) and trajectory-level (strategic action) efficiency is crucial for developing practical LLM agents, as focusing on only one aspect can compromise overall performance or other efficiency gains.
- An efficiency-aware bonus can be effectively integrated into preference optimization frameworks like KTO using offline data, enabling stable and sample-efficient training without the complexities of reward model training or online sampling.
- Rephrasing the 'Thought' components of desirable trajectories to be more concise, combined with the efficiency bonus, directly contributes to significant reductions in average tokens per step while maintaining or improving task performance.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.