Reproduce “Visualizing high-dimensional loss landscapes with Hessian directions” on OpenResearch

About this paper

Lucas Böttcher, Gregory Wheeler

Researchers demonstrate that widely used random projections of high-dimensional loss landscapes often misrepresent saddle points as apparent minima, proposing an alternative visualization method utilizing dominant Hessian directions that accurately reveals the true local geometry and can identify paths to lower loss values. This approach efficiently computes crucial curvature information, showing that optimizers frequently converge to saddle points rather than true minima.

Key insights

• Random projections frequently misrepresent saddle points in high-dimensional loss landscapes, often portraying them as apparent minima, maxima, or flat regions due to the averaging of positive and negative curvatures.
• The mean curvature in the original high-dimensional space dictates whether saddle points, on average, appear as minima (positive mean curvature) or maxima (negative mean curvature) in random lower-dimensional projections.
• Dominant Hessian directions provide a more accurate and faithful representation of the true local geometry of loss landscapes, effectively revealing saddle structures and potential paths to lower loss values that random projections obscure.

Summary by alphaXiv.