Leslie N. Smith (2015)
arXiv.
DOI: https://doi.org/10.48550/arxiv.1506.01186
Abstract. Introduces cyclical learning rate schedules, in which the learning rate oscillates between bounds, and the LR-range test for finding sensible bounds. Often enables faster training and better generalisation than monotonic decay schedules. (The related one-cycle policy is in Smith's separate 2018 paper, "A Disciplined Approach to Neural Network Hyper-Parameters".)
Tags: optimisation learning-rate schedules