Textbook of AI

A Tensor, in the loose sense used in deep learning, is a multi-dimensional array of numbers. A scalar is a 0-dimensional tensor, a vector is 1-dimensional, a matrix is 2-dimensional, and higher-rank tensors have three or more dimensions (or axes). A batch of colour images is naturally a 4-dimensional tensor (batch × height × width × channels); a batch of video clips is 5-dimensional. Deep learning frameworks such as PyTorch, TensorFlow, and JAX are built around the efficient manipulation of tensors.

Tensor operations generalise familiar matrix operations. Element-wise arithmetic works on tensors of matching shape; broadcasting rules extend these operations to tensors of compatible but unequal shapes, enabling concise code for common patterns such as adding a bias vector to each row of a batch. Reduction operations (sum, mean, max) collapse one or more axes. Reshaping, permuting axes, and slicing provide the plumbing for composing complex computations.

In strict mathematical usage, a tensor is a more sophisticated object: a multi-linear map with well-defined transformation rules under changes of coordinates, central to physics and differential geometry. Deep learning's use of the term is informal and refers only to the data structure, not the geometric object. The distinction rarely causes confusion in practice, but it is worth knowing that mathematicians and physicists use the word more precisely than deep learning practitioners do.