Reproduce “DSMentor: Enhancing Data Science Agents with Curriculum Learning and Online Knowledge Accumulation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
He Wang, Alexander Hanbo Li, Yiqun Hu, Sheng Zhang, Hideo Kobayashi, Jiani Zhang, Henry Zhu, Chung-Wei Hang and 1 more
DSMentor, an inference-time optimization framework from Carnegie Mellon University and AWS AI, enhances Large Language Model agents' performance in data science tasks by integrating curriculum learning and online knowledge accumulation. The framework achieved an 8.8% improvement on the DSEval-Exercise benchmark and an 8.8% boost in pass rate for causal reasoning problems on QRData compared to baselines.
Key insights
- Integrating curriculum learning, particularly an easy-to-hard progression, and online long-term memory significantly improves LLM agent performance over vanilla methods and random task ordering.
- The framework's ability to accumulate and utilize knowledge from prior examples is especially effective for more challenging data science benchmarks and complex causal reasoning tasks.
- Including previously incorrectly answered questions in the long-term memory can further enhance the pass rate, particularly for datasets with fewer problem sets.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.