3.15 Where to go next
Calculus is one of the three legs of the mathematical tripod that supports modern AI. We have given the chain rule a workout it will not get out of: every backward pass in this book is an instance of §3.5; every gradient-descent step is an instance of §3.9; every weight update of every model is, ultimately, the algorithm of §3.7 dressed in the notation of §3.12.
The next chapter, probability, gives us the language to talk about uncertainty, datasets, and noisy gradients, and to read off what a loss is in the first place. Chapter 5 stitches probability to calculus to give us statistics. Chapter 10 marries calculus to optimisation to give us the engine room of training. By Chapter 9, neural networks, we will be writing down full architectures, and by Chapter 10, training and optimisation, we will be putting together everything in this chapter to drive transformer-scale models.
For deeper reading on the calculus foundations:
- Spivak, Calculus. The classic rigorous undergraduate text on single-variable calculus, especially good on limits and Taylor's theorem.
- Hubbard & Hubbard, Vector Calculus, Linear Algebra, and Differential Forms. The cleanest unified presentation of the multivariable material in §3.4 and §3.5.
- Boyd & Vandenberghe, Convex Optimization, chapters 2 and 9. Convexity, Hessians, and Newton's method, with proofs of convergence.
- Griewank & Walther, Evaluating Derivatives. The standard reference on automatic differentiation, including reverse and forward modes, checkpointing, and source transformation.
- Baydin, Pearlmutter, Radul & Siskind (2018), "Automatic differentiation in machine learning: a survey". A more readable modern survey of AD as it appears in deep-learning frameworks.
- Karpathy's micrograd and makemore repositories. A 200-line autograd plus minimal training loops, very close in spirit to §3.7.
The exercises that follow are a rite of passage. Work as many as you can; the derivations in Chapter 9 onward will assume the muscle memory that they build.