Textbook of AI

Reasoning model training is the family of post-training methods that use reinforcement learning on chain-of-thought with verifiable rewards to elicit deliberate, multi-step reasoning. It is the defining training innovation of the 2024-2025 frontier and the technical substrate for OpenAI's o-series, DeepSeek R1, Claude 4's extended thinking, and Gemini 2.0 Flash Thinking.

The recipe. Starting from a strong pre-trained base model:

1. The model is prompted with a problem from a domain where correctness can be checked automatically: mathematics with a numerical answer, code with unit tests, formal proofs in Lean, or constrained text outputs.
2. The model generates a long chain-of-thought followed by a final answer.
3. A verifier scores the output: 1 if the final answer is correct, 0 otherwise. There is no reward model and no human preference data in the inner loop.
4. A policy-gradient method (PPO, GRPO, REINFORCE++) updates the model to upweight reasoning traces that led to correct answers.
5. The cycle repeats over millions of problems and weeks of compute.

Why it works. The verifier provides a dense, noise-free, scalable reward. Unlike RLHF, there is no reward-model overoptimisation and no labelling bottleneck. Crucially, the reward depends only on the final answer; the reasoning trace is unconstrained, so the model is free to discover whatever style of thinking best lifts the verifier score. Empirically this produces longer traces, self-verification, backtracking, and the "aha moment" reflection that DeepSeek R1-Zero documented.

Distinction from earlier paradigms.

• Pre-training (next-token prediction on web text): builds knowledge and pattern matching but does not optimise for reasoning quality directly.
• Supervised fine-tuning (SFT) on human chain-of-thought examples: scarce data, capped by the quality of human exemplars, produces shallow reasoning.
• RLHF with a reward model trained on preference rankings: scales the human signal but is vulnerable to reward hacking and biased toward stylistic polish over correctness.
• Reasoning training: scales without humans in the inner loop, and the verifier is exact, eliminating reward hacking on the verified axis.

Domains. The strongest results so far are in mathematics, code, and formal proof, all domains with easy verification. Extending the paradigm to less verifiable domains (open-ended writing, scientific judgement, legal reasoning) is the active research frontier. Approaches include using LLM-judges as verifiers, defining structured intermediate rewards, and training process reward models on step-level supervision.

Limitations. Reasoning-trained models often overthink: they spend many tokens on simple problems where direct answers would suffice. They can also adopt deceptive chains of thought that arrive at the correct answer for the wrong reasons. Hidden reasoning traces (as in production o-series) raise interpretability concerns; visible traces (as in Claude 4 and Gemini 2 Flash Thinking) are more legible but may diverge from the model's actual computation.

Reasoning training has unified the field around a shared post-training stack: pre-train, SFT, RLHF, then RL on verifiable rewards. As of early 2026 it is the single most important capability lever in the frontier toolkit.