Reproduce “Better Hessians Matter: Studying the Impact of Curvature Approximations in Influence Functions” on OpenResearch

About this paper

Steve Hong, Runa Eschenhagen, Bruno Mlodozeniec, Richard Turner

The authors test whether curvature approximations improve influence-function data attributions for handwritten-digit predictions without retraining. They compare Hessian-based methods in tanh MLPs and evaluate rankings against retraining. Attribution fidelity tracks curvature fidelity, though approximation steps have regime-dependent effects; the experiments are limited to a small Digits task and do not establish the ordering for larger models or other architectures.

Key insights

• A consistent inverse relationship exists between lower curvature approximation error and higher influence function fidelity (LDS), empirically confirming that better Hessian approximations improve attribution.
• The within-block Kronecker factorisation (Block-GGN to K-FAC/EK-FAC) is identified as the largest contributor to total approximation error, largely due to spectral mis-scaling and eigenbasis mismatch.
• Influence function fidelity's sensitivity to approximation errors varies with context: GGN approximation errors are critical early in training, while block-diagonal errors become more impactful in deeper networks.

Summary by alphaXiv.