Textbook of AI

ImageNet is a hand-labelled image dataset of over 14 million images across 22,000 categories, organised by the WordNet hierarchy. The version most often referred to as "ImageNet" in benchmarking is the ImageNet Large-Scale Visual Recognition Challenge (ILSVRC) 2012 subset: 1,281,167 training images across 1,000 leaf categories, plus 50,000 validation and 100,000 test images. ILSVRC ran annually from 2010 to 2017, shifting the field from feature-engineering plus SVMs to end-to-end deep learning.

The benchmark is scored on top-1 error (the model's single highest-probability class must match the ground truth) and top-5 error (the gold class must appear in the top five predictions). Top-5 was the headline ILSVRC metric until the field switched to top-1 around 2017.

Performance trajectory, the trajectory that defined modern AI:

• 2010 (first ILSVRC): hand-crafted SIFT + Fisher vectors + linear SVM, top-5 error 28.2%.
• 2011: 25.8% (more SIFT/HOG variants).
• 2012: AlexNet (Krizhevsky, Sutskever, Hinton), first CNN, two GPUs, ReLU, dropout, top-5 error 15.3% vs runner-up 26.2%. The 10.9-point gap is widely regarded as the moment the deep-learning era began.
• 2013: ZFNet 11.7%.
• 2014: GoogLeNet 6.7%, VGG 7.3%.
• 2015: ResNet-152 (He et al.), first to exceed estimated human performance (~5.1% top-5), 3.57%.
• 2016: ensemble systems 2.99%.
• 2017: SENet 2.25% (final ILSVRC).
• 2020+: top-1 error becomes the headline; Vision Transformer (ViT-G) reaches 88.6% top-1 (11.4% top-1 error), EfficientNet-V2-XL 87.3%, and self-supervised methods like DINOv2 push frozen-feature linear-probe top-1 above 86%.
• 2024+: top-1 saturating around 91% with frontier vision-language and self-supervised systems; remaining errors are largely concentrated in genuinely ambiguous or mislabelled images (an estimated 5% of the validation set).

Known issues. Decades of papers have documented label noise, class imbalance, geographic bias, and culturally skewed categories ("groom"/"bride" examples are disproportionately Western). The WordNet hierarchy is notoriously inconsistent. Recent work (Northcutt et al., 2021) found that ~6% of validation labels are wrong, capping achievable accuracy.

Modern relevance. ImageNet remains the canonical pretraining and evaluation benchmark for vision backbones, the foundational pretraining target for ViT, ConvNeXt, ResNet, EfficientNet, and the de-facto ancestor of every modern multimodal model. It is the benchmark whose ascent catalysed the deep-learning revolution, without 2012 AlexNet, modern AI would look very different.

Reference: Russakovsky et al., "ImageNet Large Scale Visual Recognition Challenge", IJCV 2015.