Textbook of AI

Abstract. Introduces the Gaussian Error Linear Unit, a smooth activation that weights its input by the CDF of a standard normal. GELU has become the default activation in transformer architectures including BERT and GPT.