Reproduce “Avoiding pathologies in very deep networks” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
David Duvenaud, Oren Rippel, Ryan P. Adams, Zoubin Ghahramani
The authors use deep Gaussian-process priors to analyze how compositions affect representations and whether giving every layer the original input can alleviate lost directions. In a 5-dimensional prior, standard compositions shift from a more balanced Jacobian singular-value spectrum at 2 layers toward one dominant direction at 6; input-connected priors retain more similarly sized singular values, but this is not evidence of improved accuracy in trained models.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.