Textbook of AI

1952–2021, Physicist, computational neuroscientist

Naftali Tishby was an Israeli physicist and computational neuroscientist whose 1999 paper with Pereira and Bialek The Information Bottleneck Method introduced a deeply influential information-theoretic framework for representation learning. The information bottleneck seeks a representation T of input X that is maximally informative about a target Y while being maximally compressed: max I(T; Y) − β I(T; X).

Tishby's later work argued that deep learning can be understood as an information-bottleneck process in which networks during training first fit the training labels (high I(T; Y)) and then compress representations (low I(T; X)) through stochastic gradient noise. The proposal was vigorously debated; subsequent empirical work has produced mixed results, but the framework remains one of the more substantive attempts at a principled theory of deep learning. He held a chair at the Hebrew University of Jerusalem.