Textbook of AI

Abstract. Shows that the L2 penalty in Adam interacts badly with adaptive scaling, and proposes AdamW, applying weight decay directly to parameters after the adaptive update, which has become the standard optimiser for transformer training.