Textbook of AI

Cross-Validation is the standard technique for estimating generalisation performance and selecting among competing models when data are limited. The central idea is to partition the data into complementary subsets, train on some and evaluate on the rest, and repeat this process to obtain a robust estimate of out-of-sample performance. Cross-validation mitigates the instability of a single train-test split and makes efficient use of limited data.

The most common variant is k-fold cross-validation: the dataset is partitioned into $k$ equally-sized folds, and in each of $k$ iterations one fold is held out for validation while the remaining $k-1$ serve for training. Common choices are $k = 5$ or $k = 10$. Leave-one-out cross-validation uses $k = n$, which is nearly unbiased but computationally expensive and high-variance. Stratified k-fold preserves class proportions within each fold, important for imbalanced classification.

Cross-validation has two distinct purposes that are sometimes conflated: model selection (choosing the best model from candidates) and performance estimation (reporting generalisation error of the final model). If the same cross-validation is used for both, the reported estimate is optimistically biased. Nested cross-validation addresses this by embedding an inner loop for model selection within an outer loop for performance estimation. For time-series data, standard k-fold violates temporal order; time-series cross-validation with expanding windows is the appropriate substitute.