Textbook of AI

In June 2017 a paper from Google Brain and Google Research carried a title that, in retrospect, reads like a manifesto: Attention Is All You Need Vaswani, 2017. The eight authors proposed an architecture they called the Transformer, a sequence model with no recurrence and no convolution, built entirely from a single primitive: attention. Within five years almost every state-of-the-art result in natural language processing, computer vision, speech, biology, and reinforcement learning was being produced by Transformers or close descendants. The Transformer is now the architectural substrate of GPT-4, Claude, Gemini, LLaMA, BERT, ViT, AlphaFold, Whisper, Sora, and the broader generation of foundation models that defines contemporary AI.

This chapter is the centrepiece of the textbook. We start where the field stood in 2014: an encoder–decoder recurrent network with a fixed-size bottleneck. We derive attention as the response to that bottleneck, develop scaled dot-product attention from variance arguments, generalise to multi-head attention, and assemble the full Transformer block including positional encodings, residuals, normalisation, and feed-forward layers. We work through a numerical example by hand. We then implement an entire decoder-only Transformer in roughly two hundred lines of PyTorch and train it on a toy task. From that foundation we tour the families that have grown out of the original paper (BERT, GPT, T5, ViT, multimodal models, Mixture-of-Experts). We close with the engineering questions that dominate today: the quadratic-complexity wall, FlashAttention, linear-attention variants such as Mamba, and the inference economics of prefill, decode, and the KV cache.

By the end of this chapter you should be able to draw the Transformer block from memory, explain every line, write the code, and reason about why it scales the way it does.

A note on style. This chapter is denser than the others. The mathematics of attention is genuinely subtle, and the engineering details of modern Transformers (RoPE, GQA, FlashAttention, KV caching, MoE routing) each deserve their own paragraph. We will not skimp on any of them. If a section feels heavy, read the pseudocode and worked examples first; the prose will then settle into place.

In this chapter

• 13.1 From RNN attention to self-attention
• 13.2 Scaled dot-product attention
• 13.3 Multi-head attention
• 13.4 Self-, cross-, and encoder–decoder attention
• 13.5 Positional encoding
• 13.6 The Transformer block
• 13.7 Encoder, decoder, encoder–decoder
• 13.8 Causal masking and autoregressive prediction
• 13.9 Parameter count and FLOPs
• 13.10 Implementing a Transformer from scratch
• 13.11 BERT: masked LM pretraining and fine-tuning
• 13.12 GPT: autoregressive LM and in-context learning
• 13.13 The quadratic wall: attention complexity
• 13.14 FlashAttention: IO-aware exact attention
• 13.15 Linear attention: Linformer, Performer, RetNet, RWKV, Mamba
• 13.16 Vision Transformer (ViT)
• 13.17 Multimodal Transformers: CLIP, Flamingo, GPT-4V, Gemini
• 13.18 Sparse attention and mixture of experts
• 13.19 Inference economics: KV cache, prefill, decode, PagedAttention
• 13.20 The Transformer's place in AI history
• Key principles
• Summary
• Exercises
• Selected solutions