Textbook of AI

Abstract. Introduces the Switch Transformer, a mixture-of-experts model that routes each token to a single expert. The paper demonstrates that MoE models with over a trillion parameters can be trained efficiently, decoupling total parameters from per-token compute.