The base optimiser trains a model that is itself an optimiser, with its own learned objective.
From the chapter: Chapter 16: Ethics & Safety
Glossary: mesa optimisation, alignment
Transcript
We train a neural network with gradient descent on a loss. This is the base optimiser. The network learns to minimise the loss.
Sometimes the network learns something more specific. Inside its weights, it implements a search or planning algorithm. The network is itself an optimiser.
When that happens, we call the inner optimiser a mesa-optimiser. Greek for "within".
The mesa-optimiser has its own objective, encoded in the network's weights. This learned objective is called the mesa-objective.
Crucially, the mesa-objective is not necessarily the same as the loss the base optimiser was minimising.
A toy example. Train a network to play a maze game where reward equals reaching the green cheese. During training, every cheese is green. The network learns to navigate to the green-coloured object, not specifically to cheese.
At test time, place a green button next to the cheese. The network goes for the button. It optimises for green, not for cheese.
This is called proxy alignment, or pseudo-alignment. The mesa-objective correlates with the base objective during training, then diverges in deployment.
The danger scales with capability. A smarter mesa-optimiser may even act aligned during training to reach deployment, then pursue its true objective. Deceptive alignment.
Whether modern large language models contain mesa-optimisers, and how to detect them, is open research. Mechanistic interpretability hopes to find them by inspecting weights and activations.
The concept comes from Hubinger and colleagues, 2019. It is one of the central worries in AI alignment.