Textbook of AI

Hebbian learning is the principle, set out by the Canadian psychologist Donald Hebb in The Organization of Behavior (1949), that the synaptic connection between two neurons strengthens when they fire in close temporal coincidence. Hebb's actual formulation, "When an axon of cell A is near enough to excite a cell B and repeatedly or persistently takes part in firing it, some growth process or metabolic change takes place in one or both cells such that A's efficiency, as one of the cells firing B, is increased", has been compressed into the slogan "cells that fire together, wire together" (a phrasing actually due to Carla Shatz, 1992).

Mathematically the simplest Hebbian rule updates the weight $w_{ij}$ from neuron $i$ to neuron $j$ by

$$\Delta w_{ij} \propto x_i \cdot x_j,$$

the product of pre- and post-synaptic activations. The pure Hebb rule is unstable, weights grow without bound, so practical Hebbian algorithms add normalisation, decay, or competition.

Variants

• Oja's rule (Erkki Oja, 1982) constrains weights to unit norm and turns out to extract the principal component of the input distribution: $\Delta w = \eta \, y (x - y w)$ where $y = w^\top x$.
• Sanger's rule (Generalised Hebbian Algorithm, 1989) extends Oja to multiple components, sequentially extracting the top-$k$ principal components.
• Bienenstock–Cooper–Munro (BCM) (1982) introduces a sliding modification threshold $\theta_M$, making weight changes positive above the threshold and negative below it; this captures the experimental observation that synapses can both potentiate and depress.
• Spike-timing-dependent plasticity (STDP) refines Hebbian learning to depend on the precise relative timing of pre- and post-synaptic spikes, pre-before-post within ~20 ms strengthens; post-before-pre weakens.
• Anti-Hebbian rules ($\Delta w \propto -x_i x_j$) underlie decorrelation, as in Földiák's network and the lateral connections of self-organising maps.

In machine learning

Hebbian-style updates underlie the Hopfield network's weight matrix $W = \sum_p \xi^p (\xi^p)^\top$ (an outer-product Hebbian sum over stored patterns), the Kohonen self-organising map, the Boltzmann machine's wake–sleep updates (positive Hebbian phase under the data, negative anti-Hebbian phase under the model), and a wide class of unsupervised contrastive objectives in modern deep learning. In modern self-supervised contrastive learning, the InfoNCE objective can be read as a Hebbian-versus-anti-Hebbian balance, pull positive pairs together (Hebbian), push negative pairs apart (anti-Hebbian).

Biological vindication

Hebb's prediction was experimentally vindicated when Tim Bliss and Terje Lømo demonstrated long-term potentiation (LTP) in the rabbit hippocampus in 1973, repeated high-frequency stimulation of perforant-path fibres produced a long-lasting enhancement of the synaptic response in dentate granule cells. LTP is now the dominant cellular model of learning and memory in the mammalian brain, and its molecular machinery (NMDA receptor coincidence detection, calcium-dependent CaMKII activation, AMPA receptor insertion) provides the biological substrate for Hebbian-style coincidence detection.