Reproduce “Analyzing Populations of Neural Networks via Dynamical Model Embedding” on OpenResearch

About this paper

Jordan Cotler, Kai Sheng Tai, Felipe Hernández, Blake Elias, David Sussillo

DYNAMO organizes trained networks as runnable models indexed by learned vectors. A shared meta-model matches each base model’s outputs and hidden-state dynamics; users can adjust a vector with additional labeled data while keeping the meta-model fixed. On IMDB, this reached 86.4% test accuracy versus 80.3% for the best base model, but base models must share an input structure.

Key insights

• The model embedding spaces generated by DYNAMO enable intuitive clustering of neural networks based on shared computational characteristics, such as training data size, architecture, task, and data augmentation policies, reflecting a more semantically meaningful grouping than pairwise comparison techniques.
• Test accuracy varies smoothly within the low-dimensional embedding space, facilitating meaningful interpolation between existing base models and enabling the discovery of optimal configurations through extrapolation beyond the convex hull of trained models.
• For recurrent networks, DYNAMO preserves and continuously interpolates intrinsic dynamics like line attractors across the embedding space, offering a framework to analyze how topological features of learned dynamics correlate with task performance and clustering.

Summary by alphaXiv.