Textbook of AI

The NIPS 2006 workshop on deep learning, formally titled NIPS 2006 Workshop on Deep Learning and Unsupervised Feature Learning , was the venue at which the work of Geoffrey Hinton's group at Toronto, Yoshua Bengio's group at Montréal, Yann LeCun's group at NYU and a handful of allied researchers was presented to the wider machine-learning community. The workshop is widely cited as the moment the deep-learning revival became visible beyond a small specialist audience, and it marks the conventional starting point for the period later branded "deep learning".

Context

Through the late 1990s and early 2000s, neural networks had been deeply unfashionable in mainstream machine learning. Support vector machines, kernel methods, and probabilistic graphical models dominated NIPS and ICML proceedings, while neural-network papers were often desk-rejected or relegated to poster sessions. Hinton, in particular, has often described the 2000s as a period of "neural-network winter" comparable to the AI winter of the late 1980s.

The turn began in 2006 with two landmark papers from Hinton's group:

• Hinton, Osindero and Teh, A Fast Learning Algorithm for Deep Belief Nets (Neural Computation, 2006), which introduced layerwise unsupervised pre-training of stacked restricted Boltzmann machines as a way to initialise deep networks.
• Hinton and Salakhutdinov, Reducing the Dimensionality of Data with Neural Networks (Science, July 2006), which used the same approach to train a deep autoencoder that outperformed PCA on document retrieval.

Bengio, Lamblin, Popovici and Larochelle's NIPS 2006 paper Greedy Layer-Wise Training of Deep Networks extended the technique to autoencoder-based pre-training.

What was presented

The workshop itself, organised by Hinton, LeCun, Bengio and Marc'Aurelio Ranzato, brought together presentations on:

• Restricted Boltzmann machines (RBMs) and contrastive divergence
• Deep belief networks and their pre-training
• The Hinton–Salakhutdinov dimensionality-reduction result
• Stacked denoising autoencoders
• Convolutional networks for object recognition
• Early speech-recognition results from Mohamed and Hinton

Mainstream NIPS attendees were sceptical at first, the methods were computationally expensive, the results modest by today's standards, and the theoretical foundations contested. By NIPS 2009, however, deep-learning papers were a substantial minority of the conference; by 2012, after AlexNet's ImageNet victory and the Microsoft–Google–IBM speech-recognition breakthroughs, they were dominant.

Continuation

The workshop continued annually as NIPS expanded, eventually being absorbed into the broader machine-learning programme as deep learning became the conference's mainstream. NIPS itself was renamed NeurIPS in 2018, partly to address the "NIPS" acronym's offensive double meaning , and is now the world's largest machine-learning conference, with attendance and submission counts in the tens of thousands.

The 2006 workshop is now a recognised inflection point in the field's history: the moment a small, persistent research community's results became impossible for the wider field to ignore.