Reproduce “Programming with Data: Test-Driven Data Engineering for Self-Improving LLMs from Raw Corpora” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Chenkai Pan, Xinglong Xu, Yuhang Xu, Yujun Wu, Siyuan Li, Jintao Chen, Conghui He, Jingxuan Wei and 1 more
The "Programming with Data" paradigm provides a structured, closed-loop approach for engineering specialized knowledge into large language models by directly analogizing LLM data engineering to test-driven software development. This framework enables diagnostic-driven data repair based on a three-level knowledge structure, leading to average performance gains of up to +32.67 points on a 16-discipline benchmark and superior sample efficiency compared to baseline methods.
Key insights
- The relationship between LLM training data and model behavior is demonstrably traceable and systematically repairable when mediated by a shared, structured knowledge specification.
- A top-down extraction methodology (L3 then L2 then L1) ensures that all extracted knowledge is connected and testable, eliminating 'orphan' knowledge nodes.
- Diagnostic-driven data repair, which targets specific concept gaps or reasoning deficits identified by benchmark failures, is significantly more sample-efficient and effective than indiscriminate data scaling.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.