[GS_C_MS] Learning at Criticality: A Landau Perspective on In-Context Learning and Regression
ABSTRACT
Why can prediction error become singular precisely when a learning model acquires enough capacity to fit its training data? I will address this question through our Landau theory of linear in-context learning, in which a pretrained network solves new regression tasks from examples in its prompt without updating its parameters. The ratio of pretraining samples to learnable parameters plays the role of temperature, ridge regularization acts as a conjugate field, and the order parameter has a geometric interpretation in terms of unconstrained directions in parameter space. The divergent contribution to prediction error is governed by a susceptibility associated with the vanishing curvature of the Landau potential. This correspondence identifies the double-descent singularity as a critical phenomenon driven by quenched fluctuations and provides a framework for understanding its phase diagram and critical scaling.
I will then describe ongoing work extending this perspective to linear, random-feature, and kernel regression. The aim is to reinterpret established theories through Landau equations of state and to understand their prediction-error singularities in terms of diverging response functions. This approach connects the geometry of weakly constrained parameter directions, the amplification of noise, and the softening of an effective potential. It also distinguishes a shared critical structure from model-dependent behavior arising from the spectrum of the data.
Finally, I will briefly discuss a complementary replica formulation of linear in-context learning. This formulation recovers the Landau susceptibility within a quenched-disorder calculation and extends the analysis to prediction covariances and stability, while clarifying the distinction between the Landau potential and the replica free energy.
