Reproduce “When Incentives Backfire, Data Stops Being Human” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Sebastin Santy, Prasanta Bhattacharya, Manoel Horta Ribeiro, Kelsey Allen, Sewoong Oh
The paper critically analyzes the diminishing quality of human-generated data for AI, attributing it to the proliferation of large language models and flawed, extrinsic incentive structures in data collection. It advocates for a new data collection paradigm centered on intrinsic human motivation and game-like environments to ensure a sustainable supply of high-quality, authentic data.
Key insights
- Over-reliance on extrinsic financial incentives can diminish intrinsic motivation and lead to contributors 'gaming' the system, illustrating principles like the Overjustification Effect and Goodhart's Law in data collection.
- The persistent trade-off between data quantity and quality observed in current systems is primarily a consequence of flawed design choices (e.g., task fragmentation) rather than an inherent limitation of human data.
- Designing systems for a 'middle resolution of control'—offering sufficient structure while preserving contributor autonomy—is essential for fostering sustained, high-quality human contributions, as evidenced by intrinsically motivated online communities.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.