Textbook of AI

Despite its name, Logistic Regression is a classification algorithm. It models the probability that input $\mathbf{x}$ belongs to the positive class by passing a linear combination of features through the sigmoid function:

$$P(y = 1 \mid \mathbf{x}) = \sigma(\mathbf{w}^T \mathbf{x} + b) = \frac{1}{1 + e^{-(\mathbf{w}^T \mathbf{x} + b)}}$$

The sigmoid maps the real line to $(0, 1)$, giving a principled probabilistic interpretation. The decision boundary—where the predicted probability equals 0.5—is a hyperplane, making logistic regression a linear classifier.

Parameters are estimated by maximising the log-likelihood, equivalent to minimising binary cross-entropy. The loss is convex, so gradient-based optimisation converges to the global minimum. Logistic regression produces calibrated probability estimates, valuable in medical diagnosis, credit scoring, and any application where the cost of errors varies. Coefficients admit a natural interpretation as log-odds ratios: a unit increase in feature $x_j$ multiplies the odds by $e^{w_j}$.

Extension to $K > 2$ classes uses the softmax function, yielding multinomial logistic regression: $P(y = k \mid \mathbf{x}) = \exp(\mathbf{w}_k^T \mathbf{x}) / \sum_j \exp(\mathbf{w}_j^T \mathbf{x})$. The softmax reappears throughout deep learning as the standard output layer for multi-class classification, making logistic regression the direct ancestor of the classification head on every modern neural network.