Training Dynamics of In-Context Learning in Linear Attention

ICML 2025 Spotlight: an exact theory of how linear-attention transformers acquire in-context learning, characterizing abrupt versus saddle-to-saddle learning regimes. Clean UCL (Zhang, Singh, Latham, Saxe; Gatsby Unit + Sainsbury Wellcome Centre) — a theory anchor of the Gatsby Unit's program.