Adam Optimizer
Last Updated: July 29, 2026 | By Mihail Sebastian | AI Dictionary
An optimization algorithm that gives every neural network parameter its own adaptive learning rate, combining momentum with per-parameter scaling.
What is Adam Optimizer?
Adam (Adaptive Moment Estimation) is an algorithm for updating a neural network’s weights during training. Plain gradient descent applies one learning rate to every parameter; Adam gives each parameter its own and adjusts it as training progresses.
It borrows from two earlier optimization algorithms: AdaGrad, which adapts the learning rate per parameter, and RMSProp, which keeps a running average of squared gradients. The combination made Adam the default optimizer for training deep networks.
How Adam Optimizer Works
For every parameter, Adam tracks two moving averages of the gradient:
- First moment (the mean): the average direction the gradient has been pointing. This is the momentum part: it smooths out noisy updates so the parameter doesn’t zigzag.
- Second moment (the variance): how much the gradient has been fluctuating. Adam uses this to shrink the step size for parameters with unstable gradients and grow it for stable ones.
Both averages start at zero, which biases them toward zero in the first steps of training. Adam corrects for this bias so early updates aren’t artificially small. It then updates each parameter using the smoothed gradient, scaled by that parameter’s adapted learning rate.
Update formula:
- \( m_t = \beta_1 \cdot m_{t-1} + (1 - \beta_1) \cdot g_t \)
- \( v_t = \beta_2 \cdot v_{t-1} + (1 - \beta_2) \cdot g_t^2 \)
- \( \hat{m_t} = \frac{m_t}{1 - \beta_1^t} \) (bias-corrected)
- \( \hat{v_t} = \frac{v_t}{1 - \beta_2^t} \) (bias-corrected)
- Update: \( \theta_t = \theta_{t-1} - \frac{\eta \cdot \hat{m_t}}{\sqrt{\hat{v_t}} + \epsilon} \)
where:
- \( g_t \) is the gradient,
- \( \beta_1 \) and \( \beta_2 \) are decay rates for the first and second moments,
- \( \eta \) is the learning rate,
- \( \epsilon \) is a small constant to prevent division by zero.
Why Adam Is Popular
Adam usually works without much tuning. The per-parameter learning rates absorb differences in gradient scale, so the defaults from the original paper – learning rate 0.001, \( \beta_1 = 0.9 \), \( \beta_2 = 0.999 \) – train a wide range of networks reasonably well. It also tends to make faster progress early in training than plain SGD, which matters when a single run takes hours or days.
Example of Adam Optimizer
Consider training a convolutional neural network for image recognition. Filters in early layers receive strong, frequent gradients while parameters deep in the network receive weak, sparse ones. With a single global learning rate, one group trains well and the other barely moves.
Adam gives each parameter a step size matched to its own gradient history, so both groups make progress in the same run. That’s one reason it’s the first optimizer most practitioners reach for.
FAQ
Is Adam always better than SGD?
No. Adam converges faster with less tuning, but well-tuned SGD with momentum sometimes generalizes better, and some vision models are still trained with it for that reason. Adam is the safer default; SGD is worth trying when you can afford the tuning budget.
What is the difference between Adam and AdamW?
AdamW changes how weight decay is applied: it decouples the decay from the gradient update instead of folding it into the gradient as L2 regularization. With Adam’s adaptive learning rates the two are not equivalent, and the decoupled version regularizes more predictably. AdamW is the standard choice for training transformers.
Related AI terms: Optimization · Learning Rate · Gradient Descent · Neural Network · Fine-tuning
Did you like the Adam Optimizer gist?
Learn about 250+ need-to-know artificial intelligence terms in the AI Dictionary.
Mihail Sebastian — Writes about AI governance, regulation, and the technology behind them. Placeholder bio — replace with a real credential line. About