Seminars

Home Centers & Programs AI and Natural Sciences Seminars

TITLE: The surprising usefulness of layerwise linear models in describing dynamical phenomena of DNNs (neural collapse, emergence, scaling laws, grokking)

ABSTRACT: Linear neural networks and their variants are overlooked due to their lack of non-linear activations and limited expressivity. Yet, their dynamics are non-linear and can explain various DNN phenomena, previously considered a complicated artifact of the dataset or architecture. We introduce the dynamical feedback principle — how layers mutually govern and amplify each other’s evolution — as an intuitive description of gradient descent under layerwise structures. Building on this principle, we review how simple, solvable layerwise-linear models capture a range of seemingly unconnected phenomena, revealing the intuition beneath technical results. Biography: Yoonsoo Nam is a PhD student in the Theoretical Physics department at the University of Oxford, working in the Ard Louis group. His research focuses on understanding the inductive biases of neural networks through solvable and intuitive models, often extending linear neural networks and linear models. Prior to his PhD, he worked at NAVER on restaurant recommendation systems and neural architecture search. He completed Part III of the Mathematical Tripos at Cambridge and studied Physics at Oxford.