Learning Rate
Last Updated: July 29, 2026 | By Mihail Sebastian | AI Dictionary
The hyperparameter that sets how far an optimizer moves a model's weights on each update: too large and training diverges, too small and it crawls.
What is Learning Rate?
The learning rate is the hyperparameter that sets how far an optimizer moves a model’s weights at each training step. It is widely considered the single most consequential knob in training: get it wrong and nothing else matters.
In the gradient descent update \( \theta_{t+1} = \theta_t - \eta \nabla L(\theta_t) \), the learning rate is \( \eta \). The gradient supplies the direction; the learning rate decides the stride.
Choosing a Learning Rate
Set it too high and each step overshoots the minimum. The loss oscillates or explodes to infinity, and the model never converges. Set it too low and training crawls: the loss inches downward, the compute bill grows, and the optimizer risks stalling in a flat region or a shallow local minimum.
The workable range depends on the architecture, the batch size, and the optimizer, so practitioners find it empirically. A standard approach is to try values on a logarithmic scale (0.1, 0.01, 0.001, and so on) and watch the first few epochs of loss.
Two refinements are near-universal. Learning rate schedules change \( \eta \) over the course of training, warming up from a small value and decaying toward the end. Adaptive optimizers such as Adam go further and give every parameter its own learning rate, adjusted from its gradient history.
Example of Learning Rate
Suppose you train the same small neural network three times, changing only the learning rate. At 1.0, the loss shoots upward within a handful of steps and turns into NaN: each update leaps clear across the valley to a worse position on the far slope.
At 0.0000001, the loss curve after an hour is almost flat. The updates are correct in direction but microscopically small.
At 0.001, the loss drops steeply, then levels off near its minimum. Same data, same architecture, same optimizer: the stride length alone separated failure from success.
Related AI terms: Gradient Descent · Adam Optimizer · Hyperparameter · Hyperparameter Tuning · Stochastic Gradient Descent
Did you like the Learning Rate gist?
Learn about 250+ need-to-know artificial intelligence terms in the AI Dictionary.
Mihail Sebastian — Writes about AI governance, regulation, and the technology behind them. Placeholder bio — replace with a real credential line. About