Exercises
(Perceptron geometry.) Plot the four points $(0,0), (0,1), (1,0), (1,1)$ in the plane and indicate the decision boundary $w_1 x_1 + w_2 x_2 + b = 0$ for the OR function with $w_1 = w_2 = 1$, $b = -0.5$. Verify that all four points are correctly classified.
(Convergence theorem.) Suppose $R = 1$ and $\gamma = 0.01$. What is the maximum number of mistakes the perceptron can make? Suppose the data is rescaled so that $R = 100$ but $\gamma$ doubles. How does the bound change?
(XOR by hand.) Construct a two-layer perceptron with hard-threshold activations that computes XOR. Specify all weights and biases.
(XOR with sigmoids.) Repeat exercise 3 using sigmoid activations. Why does a hard-threshold solution not directly translate?
(Forward pass.) For the network of Section 9.3, compute the forward pass on $\mathbf{x} = (0, 1)^\top$.
(Vanishing sigmoids.) Consider a 20-layer network with sigmoid activations. Approximate the typical magnitude of $\sigma'(z)$ at saturation and bound the gradient at the input layer in terms of the gradient at the output. Why is this a problem?
(ReLU derivative at zero.) What is $\mathrm{ReLU}'(0)$? Discuss why this discontinuity is harmless in practice.
(Softmax invariance.) Show that $\mathrm{softmax}(\mathbf{z} + c \mathbf{1}) = \mathrm{softmax}(\mathbf{z})$ for any scalar $c$. Why does this matter for numerical stability?
(Cross-entropy gradient.) Verify by direct calculation that for $\mathbf{p} = \mathrm{softmax}(\mathbf{z})$ and one-hot $\mathbf{y}$, $\partial \mathcal{L}_{\text{CE}} / \partial z_i = p_i - y_i$.
(Sigmoid + BCE gradient.) Show that for $p = \sigma(z)$ and binary cross-entropy, $\partial \mathcal{L}_{\text{BCE}} / \partial z = p - y$.
(Backprop by hand.) Repeat the worked example of Section 9.7 with $\mathbf{x} = (0, 1)^\top$ and $y = 0$. Compute every gradient, then a single SGD step with $\eta = 0.1$.
(Universal approximation construction.) Sketch a one-hidden-layer ReLU network with two hidden units that approximates $f(x) = |x|$ on $[-1, 1]$ exactly. Generalise to $K$ hidden units approximating any continuous piecewise linear function.
(Depth efficiency.) Construct a function on $[0, 1]$ that requires $\Omega(2^L)$ width at depth 1 but is computable by a depth-$L$ ReLU network of width 2. (Hint: use the triangle wave by composing reflections.)
(Xavier derivation.) Derive Glorot's variance bound by demanding that the variance of pre-activations and gradients be preserved at initialisation. State your assumptions.
(He derivation.) Repeat for ReLU. Why does the variance double?
(Dying ReLU.) Construct a small example (specific weights, biases, and data) in which a ReLU unit has zero output and zero gradient on every training point.
(Batch norm Jacobian.) Verify the batch-norm backward formula given in Section 9.13 by differentiating through $\hat{z}_b = (z_b - \mu_B) / \sqrt{\sigma_B^2 + \varepsilon}$.
(Layer norm vs batch norm.) Explain why layer norm is preferred for small batch sizes and for sequence models with variable-length inputs.
(Dropout as ensemble.) For a network with $K$ hidden units and dropout rate $p$, how many distinct subnetworks can dropout sample?
(L2 and weight decay.) Prove that for plain SGD, adding $\lambda \|\mathbf{W}\|^2 / 2$ to the loss is exactly equivalent to a weight-decay rule $\mathbf{W} \leftarrow (1-\eta\lambda)\mathbf{W} - \eta\mathbf{g}$. Show that for Adam they are not equivalent.
(Gradient clipping.) Implement gradient clipping by global norm in NumPy. Test it on the MNIST MLP.
(Cosine schedule.) Plot a cosine annealing schedule with warmup over 10% of training. Why is warmup needed for Transformer-scale learning rates?
(Mini-batch noise.) Explain why SGD with batch size 32 often generalises better than full-batch GD. Cite at least two mechanisms.
(Numerical gradient check.) Use finite differences to verify the gradient of a small MLP. Why is this only practical for small networks?
(MNIST baseline.) Train the from-scratch MLP of Section 9.16 with three different learning rates. Report training and test accuracy. Plot the curves.
(MNIST PyTorch.) Train the PyTorch MLP of Section 9.17 with and without dropout. Compare test accuracy.
(Activation comparison.) Train an MLP with sigmoid, tanh, ReLU, and GELU on MNIST. Report the wall-clock time to reach 95% test accuracy for each.
(Initialisation sweep.) Train a 10-layer MLP on MNIST with all-zero, $\mathcal{N}(0, 1)$, Xavier, and He initialisation. Explain the results.
(Autograd extension.) Extend the autograd engine of Section 9.15 with
__pow__,exp, andlog. Verify that gradients agree with finite differences.(Transformer activation.) Replace ReLU with GELU in the PyTorch MLP. Does it help on MNIST? Why might the answer differ on a Transformer language model?
(Label smoothing.) Implement label smoothing with $\varepsilon = 0.1$. Train on MNIST and report the calibration improvement (expected calibration error).
(Mixup.) Implement mixup with $\alpha = 0.2$ on MNIST. Report training and test accuracy with and without mixup.
(Symmetric initialisation.) Prove that if every weight in a layer is initialised to the same value (and all biases are zero), then every unit in that layer computes the same function for all inputs throughout training. Conclude that breaking symmetry requires randomness.
(Backprop computational cost.) Show that for an MLP with sizes $(d_0, d_1, \ldots, d_L)$, the backward pass costs the same number of FLOPs as the forward pass to within a factor of two. Identify the precise factor.
(AdamW vs Adam.) Show analytically that for a parameter on which the running second moment $\hat{v}_t$ is large, Adam with L2 regularisation applies less effective weight decay than AdamW with the same nominal $\lambda$. Construct a synthetic example where this matters.
(Saturating ReLU on negative-mean input.) Suppose a hidden unit's pre-activation has distribution $\mathcal{N}(\mu, 1)$ with $\mu = -3$. What is the probability that the unit is active on a random input? What does this imply for He initialisation when input distributions are not zero-mean?