Textbook of AI

Abstract. Establishes that in the infinite-width limit, training a neural network with gradient descent and small learning rate is equivalent to kernel regression with the Neural Tangent Kernel (NTK), a kernel determined by the network architecture at initialisation. The NTK remains constant during training in this limit, so dynamics become linear and analytically tractable. The paper provided the first solid theoretical handle on the optimisation and generalisation of overparameterised networks and seeded a substantial follow-up literature on lazy training, feature learning and the gap between NTK predictions and finite-width network behaviour.