Reproduce “Daunce: Data Attribution through Uncertainty Estimation” on OpenResearch

About this paper

Xingyuan Pan, Chenlu Ye, Joseph Melkonian, Jiaqi W. Ma, Tong Zhang

DAUNCE, developed by a team from the University of Illinois Urbana-Champaign and Womp Labs, introduces a method for scalable and accurate training data attribution, capable of operating on proprietary, API-based models like OpenAI's GPT series. It estimates data influence through uncertainty by leveraging the covariance of per-example losses across perturbed model versions, consistently outperforming existing scalable methods on computer vision and large language model tasks and demonstrating the first empirical data attribution on black-box LLMs.

Key insights

• An analytical connection between influence functions and uncertainty estimation allows for approximating influence through the covariance of per-example losses across perturbed model versions, bypassing explicit Hessian computation.
• Using correlation instead of covariance for influence scores can lead to better attribution accuracy by normalizing gradient magnitudes and reducing outlier effects.
• Data attribution for black-box models is feasible by creating perturbed models through API-based fine-tuning and calculating losses from API-provided probabilities, even without internal gradient access.

Summary by alphaXiv.